te - unicef · web viewin 2011 external testing of students in primary and secondary education was...

EVALUATION OF EARLY LITERACY AND NUMERACY PROGRAMMEIN THE FORMER YUGOSLAV REPUBLIC OF MACEDONIA

TERMS OF REFERENCEEVALUATION OF EARLY LITERACY AND EARLY NUMERACY

PROGRAMMEIN THE FORMER YUGOSLAV REPUBLIC OF MACEDONIA 2009 – 2015

CONTENTSABBREVIATIONS.....................................................................................................................................2

I. BACKGROUND...........................................................................................................................3

II. THE PROGRAMME.....................................................................................................................6

III. RATIONALE..............................................................................................................................11

IV. OBJECTIVES OF THE EVALUATION............................................................................................12

V. SCOPE AND LIMITATIONS........................................................................................................12

VI. EVALUATION FRAMEWORK.....................................................................................................13

VII. METHODOLOGY.......................................................................................................................14

VIII. WORK PLAN AND EVALUATION MANAGEMENT......................................................................16

IX. TEAM COMPOSITION, ROLES AND RESPONSIBILITIES..............................................................18

X. PRODUCTS TO BE DELIVERED and STRUCTURE OF EVALUATION REPORT...............................20

XI. ETHICAL STANDARDS AND SAFEGUARDS.................................................................................21

XII. QUALITY ASSURANCE...............................................................................................................21

XIII. RESOURCES AND PAYMENT SCHEDULE...................................................................................21

XIV. REMARKS AND RESERVATIONS................................................................................................22

Annex 1- Logical Framework........................................................................................................23

Annex 2: UNICEF Core Roles........................................................................................................25

Annex 3: Dissemination Plan.......................................................................................................26

Annex 4: Sources of Information available from UNICEF.............................................................27

1


ABBREVIATIONS

AFT American Federation of Teachers

BDE Bureau for Development of Education

CFS Child Friendly School

CRC Convention on the Rights of the Child

CO Country office

ET Evaluation Team

MCEC Macedonian Civic Education Center

FGD Focus Group Discussion

PIRLS Progress in International Reading Literacy Study

PISA Programme for International Student Assessment

RLTs Regional Learning Teams

RSE Research Studies and Evaluation

SoPs Standard Operating Procedures

SSO State Statistical Office

SEI State Education Inspectorate

TIMSS Trends in International Mathematics and Science Study

TOR Terms of Reference

2


I. BACKGROUND The education system in the Former Yugoslav Republic of Macedonia is normed by a set of legislative and policy documents adopted by either the Parliament or the Government. Article 44 of the Constitution of the Republic of Macedonia stipulates that everyone has the right to education; education is accessible to everyone under equal conditions; primary and secondary education is compulsory and free of charge in public educational establishments. Article 48 states that minorities have the right to education in their own language, both in primary and secondary schools.

The Ministry of Education and Science (MoES) is accountable for the overall educational system from primary to tertiary education. MoES is in charge of the establishment and operation of educational institutions and their accreditation, approval and adoption of curricula, publication of textbooks, and endorsing appointments in educational institutions. Other key institutions responsible for education are: Bureau for Development of Education (BDE), State Education Inspectorate (SEI), National Examination Centre (NEC), Pedagogical Service (PS) and the Directorate for Development and Promotion of Education in the Languages of the Communities.

The national education budget for 2011 amounted to approximately 4.6 per cent of GDP, which is well below the OECD average of 5.7 per cent. Although responsibility for provision of primary and secondary education services have been devolved to the local level, education remains mainly funded from the central budget. Expenditures on education constitute almost half of all government transfers to municipalities. However, about 80 per cent of the local budget is allocated for employees’ salaries, leaving little or almost no resources available for other essential areas like training or service development.

Based on the last census conducted in 2002, the national literacy rate is 96 per cent 1. This indicates that 4 per cent of the population aged 15 and above is illiterate. The illiteracy rate among ethnic Macedonians was the lowest (2%), while the highest was among Roma population (21%). In 2002, 33 per cent of the Roma population had failed to complete primary education and of the remaining 67 per cent, 93 per cent had completed only primary education. Overall, there are no substantial differences in the literacy rate among the geographical regions as well as between urban and rural areas in the country. The literacy level of students is strongly associated with education of mothers 2.

Primary education is organized in three cycles of three years each. While curricula development, textbooks and teacher training remain as responsibilities of the central government, municipalities still have responsibility for building and maintenance of schools, selection of teachers and staff for employment, provision of textbooks, materials and support services. In September 2008, primary education was transformed into a nine-year education (three cycles of three years each) programme with children being enrolled at the age of six instead of seven. According to official data, there are 360 elementary schools in the country in 2015 and 105 secondary schools. Based on State Statistical Office (SSO) data at the end of school year 2014/15 there were 188,361 (91,397 females) students in the regular primary schools [out of which 120,716 (58,887 females) students are taught in Macedonian language, 61,608 (29,589 females) in Albanian language, 5,566 (2,709 females) in Turkish language and 471 (222 females) in Serbian language]. The World Bank data on “Gross enrollment ratio in primary schools” and SSO’s data on total number of children enrolled in primary school show a decline between 2002 and 2012, and then they stabilize between 2012-2014.1 More recent data on literacy is not available. According to World Bank Data, in the period between 2005 and 2013, youth literacy rates (for males and females age 15-24) remains at a relatively high level (99% and 98% respectively). Source: World Development Indicators (http://data.worldbank.org/)2 UNICEF Republic of Macedonia, Multiple Indicator Cluster Survey - 2005-2006 (pg.49)

3


Secondary education has been compulsory since 2008. General secondary education covers period of four years, whereas vocational secondary education lasts from two to four years, depending on the specific programme. Students enrol in secondary education on the basis of their interest and performance in primary education. At the end of the school year 2014/15, the total of 81,788 (39,533 females) students were enrolled in the secondary schools [out of which 54,858 (26,767 females) were taught in Macedonian language of instruction, 23,308 (11,024 females) in Albanian language, 1574 (761 females) in Turkish and 2,048 (981 females) in English language of instruction)]. According to official data, approximately 60 per cent of secondary school students in the country attended vocational programmes.

In accordance with the new Law for Teachers adopted in 2015, teachers are entitled to 60 hours of compulsory professional development over a period of three years. Teacher licencing is also introduced, but the implementation is pending.

BDE is responsible for curriculum development, research in education, supporting the education process in schools, monitoring teachers’ performance and organizing teacher training. The capacities and the budget for this type of support are limited in terms of number of staff and funds allocated by MoES. Such activities are, therefore, to a large extent supported by donors. For several professional development initiatives and trainings implemented in the last decade there is limited data about their effectiveness. There is no system of measuring the effectiveness of trainings, or linking them with the results from the national, external or international assessments (TIMSS, PIRLS and PISA).

The State Education Inspectorate (SEI) reports to the National Inspection Council3 and is responsible for surveillance on the quality of education process and its effectiveness through conducting external (integral) evaluation of the work of the primary and secondary schools, as well as surveillance on the application of laws, other regulations and general legal acts that regulate the field of education. The National School Performance Quality Indicators are the basis for evaluation of schools. The overall performance of schools is evaluated in the following areas: curricula, pupils’ attainment, learning and teaching, support to pupils, school climate, resources, and leadership and management. Through these evaluations SEI identifies strengths and weaknesses and provides recommendations for improvement to schools, and monitors their progress through control evaluations. In 2009, the individual assessment of teachers and school support staff (pedagogue, psychologists, sociologist, social worker, special educator and librarian) was introduced as part of the integral evaluation.

In 2006, education was announced as one of the four Government priorities. Education was seen as key to increasing the quality of human capital. Education reform launched by the Government included also curricular reform in 2008 (shift towards goal-oriented curricula, new subjects focussing on skills and outcomes – Life skills based education) and incorporation of Child-Friendly School Principles4.

Assessment of students’ performance in grades one to three is descriptive. Usually almost all students advanced to the next grade. Even though assessment standards were developed, there is grades inflation in the upper cycles of primary education and students are sometimes permitted from one grade to the next despite of failing to meeting the performance criteria. Although this may be done with the child’s well-being in mind, the practice allows teachers to abdicate their responsibility for students acquiring academic competencies.

3 SEI was a body within the Ministry of Education and Science until 2013.4 One of the key principles of the CFS approach is effectiveness or what children learn – focus on higher results and relevant knowledge.

4


In 2011 external testing of students in primary and secondary education was introduced with a main purpose to assess teacher’s objectivity in student assessment. At the end of each school year, students from grades 4-9 in primary schools and all grades in secondary schools are tested in two randomly selected subjects. The results from the external testing are then compared with the final marks that the student has received from the teachers of those two subjects at the end of the school year. In the legislation, it is stipulated that 20% of all teachers with the highest discrepancies between the results from the external testing and final students’ assessment marks will be financially punished for the period of one year, and 20% of teachers with lowest discrepancies in marks will be financially awarded. Due to teacher strikes the enforcement of this article is postponed. Furthermore, the legislation outlines that the employment of teachers who are punished two years in a row must be terminated. The external testing is heavily criticized by the expert community and the public.

The country does not participate regularly in the international comparative studies such as TIMSS5, PIRLS6 and PISA7. However, when the degree to which students aged 15 can read and comprehend was measured through PISA of the OECD in 2000, Macedonia ranked 38 out of 41 countries. The PISA results indicated that only 3.5 per cent of pupils who had completed or were about to complete primary education had not achieved the first of five levels of proficiency. Furthermore, in 2006 in PIRLS the country was ranked 33rd out of 40 participating countries – only 2% of 4th grade students reached the “advanced” benchmark, 15% “high” benchmark and 40% “intermediate” benchmark; in 20038 in TIMSS (the mathematics study) the country was ranked 29 out of 38 countries -70% of eight grade students reached the minimum level. These results were taken as serious indicator of poor quality of teaching in the country and illustrated a need for systematic changes in order to raise the level of the population’s actual literacy. In all studies including National assessments in numeracy and literacy in lower and upper primary, female students reached better results, but the differences were not significant.

Relevant Governmental and Sectorial Policies

The legal framework is comprehensive and reflects efforts made to harmonize it with the European Directives, recommendations and indicators.9 The overall reference framework for activities in the field of education in the former Yugoslav Republic of Macedonia is the National Programme for the Development of Education (2005-2015). Other policy documents in the field of education include: A Concept for Nine-year Primary Education (2007); Curricula documents for primary education (2009); School Quality Performance Indicators (2009); The Steps toward Integrated Education in the Republic of Macedonia (2009); Strategy for Vocational Education and Training in the context of lifelong learning (2013-2020); Strategy for Reducing Violence in Schools (2012-2015); Strategy for Roma People in the Republic of Macedonia (2014-2020); National Action Plan for Children Rights (2012-2015); National Strategy for Prevention and Protection of Domestic Violence (2012-2015), etc.

5 In TIMSS -Trends in International Mathematics and Science Study for forth and eight grade students the country participated in 1999, 2003 and 2011.6 In PIRLS - Progress in International Reading Literacy Study, nine and ten years old students, the country participated in 2001 and 20167 PISA - Programme for International Student Assessment, of 15 old students, the country participated in 2000 and in 2015 (for the later the results are not yet ready).8 In 2011 in TIMSS mathematics study the country was ranked 29rd out of 42 participating countries – only 3% of 8th grade students reached the “advanced” benchmark, 12% “high” benchmark and 35% “intermediate” benchmark. The results were significant lower than TIMSS 1999 and TIMSS 2003.9 European Union (2013): Teacher Education and Training in the Western Balkans, Report on: the Former Yugoslav Republic of Macedonia

5


Currently, the Ministry of Education and Science with support from IPA is drafting a new Strategy for Education (2016-2020) which is expected to be adopted by the end of 2016.

II. THE PROGRAMME The Early Literacy and Numeracy Programme was developed in response to repeated low student achievement in international and national assessment tests, the reactions of secondary schools on the low level of numeracy and literacy of students entering secondary schools, as well as, the weak mechanisms in place to support learners lagging behind or not performing to the best of their abilities. The system and teachers in particular had generally low expectations for students and little attention was paid to those not mastering the language of instruction (such as Roma children), or having gaps in their mathematics or language knowledge.

The programme was conceptualized as a teacher education programme aiming to improve students’ outcomes in early literacy and early numeracy by developing teachers’ understanding of what constitutes quality instruction in numeracy and literacy and by equipping them with the necessary skills for effective teaching and learning.

The objectives of the programme were to:

- Establish effective and sustainable approaches to teacher professional development;- Deepen teachers’ understanding of quality teaching in numeracy and literacy;- Provide teachers with opportunities to acquire knowledge and skills for implementation of

quality mathematics instruction and quality literacy instruction;- Maximise on-the-job support to encourage teachers to change their classroom practices;- Improve students’ outcomes in early numeracy and literacy to provide a good basis for

further learning in all subjects, and better results in future national and international assessments.

Strategies and activities

The main strategy of the Literacy and Numeracy programme is focused on building capacities of teachers and stimulating changes in the system by changing teachers’ practice (as a longer term objective). The programme used a cascade professional development model, by which all teachers in the country were trained by master trainers.

The strategy was pursued through the following activities:

Preparation phase (2008-2010): focused on gaining a better understanding of influencing factors.1. Review of national curricula in early literacy and mathematics and review of teaching

practice; 2. Identifying international best practice of teacher education programmes in two subjects:

mathematics and language. The „Thinking Mathematics in Early Grades“– developed by the American Federation of Teachers (AFT) was selected as the most appropriate10; and for

10 The programme is used in Minnesota, USA and has helped teachers to get deeper knowledge and understanding about mathematics, and enabled change in teacher instruction resulting in improved student outcomes. The AFT had given permission for use of this programme in Macedonian context. The first version was adapted for use by Judy Rohde, Miske Witts&Associates and Alice Gill, the author. The later adaptations were made by the national trainers, BDE advisors and trained teachers.

6


language, the teacher education programme was developed by Nancy Clair, Jan Westrick, Miske Witts & Associates11.

3. Development of teacher manuals- teacher manuals for both mathematics and literacy were developed and distributed to all teachers during the training;

4. Baseline studies-based on the principles and content of the new numeracy and literacy teacher education programmes were conducted to measure the initial situation of both early grade teachers knowledge and student outcomes at the end of grade 312;

Implementation phase (2010- 2015): teacher training, and developing a model of teacher professional development and support.

1. Training of national trainers and BDE advisors – The cascade training model was based on the recruitment of 90 national trainers and 30 BDE advisors for numeracy and for literacy. The National Trainers were recruited through a public call to ensure that the best teachers were selected nation-wide in a transparent manner, based on clear criteria. National trainers received three extensive 5-day trainings alongside BDE advisors, which contributed to building up a shared understanding and a common language between advisors and trainers to address the gaps in literacy and numeracy instruction.

2. Dissemination of teacher training (cascade model) in all primary schools: National trainers first trained all early grade teachers (Grade 1 to 3) within their school and within one neighbouring school in both numeracy and literacy. They then trained four school trainers (two in literacy, two in numeracy) in the remaining schools. All school trainers were responsible for training the remaining teachers in their respective schools. Teachers were not trained in literacy and numeracy in the same year to avoid overburden. While initially only teachers from grades 1 to 3 were targeted, teachers from grades 4 and 5 were included later in the trainings13. The teachers received total of 40 hours training for each programme. The trained teachers were also provided with continuous support and feedback on the quality of applied new concepts from the national trainers, the school trainers and BDE advisors.

3. Teacher Certification - An innovative teacher certification model was established to motivate and incentivise participants to implement the programme. Apart from the certificates given at the end of the training, additional certificates were issued upon evidence of concrete application of the new techniques (new literacy & numeracy instruction strategies, mentoring other teachers, etc.). Certificates were awarded to: national trainers, school trainers, regular teachers, RLTs’ mentor teachers14.

4. Regional Learning Teams (RLTs) were set up in 2013 as a professional networking mechanism to offer teachers an opportunity to deepen their professional competences in the area in which they were trained and to receive extra support as they change their practice. Through RLTs, teachers, school trainers, national trainers, BDE advisors and academics exchange experiences support learning across schools and review, assess and share the best practices and materials. Based on the fidelity tools, areas for improvement are identified and discussed drawing on the RLTs’ expertise and external expertise where

11 Similarly, later versions were adapted by national trainers and trained teachers.12 http://www.unicef.org/tfyrmacedonia/UNICEF_Math_Report_2009_-ANG_za_na_web_so_korici.pdfhttp://www.unicef.org/tfyrmacedonia/JAZICNA_PISMENOST_Lan_Report_2010_ANG_za_na_web.pdf13 Low primary teachers are teaching all subjects to one class of the same students from grade 1 until grade 5. In whole day stay classes there are two teachers per class, one in the morning and one in the afternoon. 14 These certificates are awarded more points for the teacher career advancement.

7

http://www.unicef.org/tfyrmacedonia/UNICEF_Math_Report_2009_-ANG_za_na_web_so_korici.pdf

http://www.unicef.org/tfyrmacedonia/JAZICNA_PISMENOST_Lan_Report_2010_ANG_za_na_web.pdf


necessary. RLTs also provide an opportunity for academics to get involved in action research on pedagogical innovation.

5. Continuous monitoring and support - the international consultants developed a set of monitoring tools, and trained BDE advisors on how use them and how to provide tailored support to teachers. In addition, BDE advisors supported the national trainers on usage of the monitoring tools.

6. Measuring progress – after three years of implementation, the teachers’ knowledge and attitudes related to the programs were measured, as well as the achievements of students in numeracy and literacy at the end of Grade 3 in 15 pilot schools, compared with equivalent sample of 15 control schools15 (the control schools were included in 2014 for numeracy and in 2015 for literacy program).

7. Linking the programmes with pre-service teacher training – In the interest of sustainability, academics were also trained in the new literacy and numeracy teaching approaches in order to support their integration in university courses.

ResultsWith UNICEF support, the following results were achieved in Literacy and Numeracy programme:

Results for teachers

As part of the program, a total of 6,593 (4,807 females) lower primary educators (teachers and other teaching and support staff) were trained in both programs; 5,156 educators in Macedonian language, 1,346 in Albanian language and 91 in Turkish language.

Mid-term surveys and analysis of BDE monitoring reports show that teachers have improved their knowledge, skills and attitudes towards literacy and numeracy teaching. In addition the analysis indicated that:

- teachers’ confidence has increased in terms of implementing new pedagogical approaches, creating new materials and being open to share practices and receive critical feedback. Reports show that teachers have implemented the new approach in respectively a third and three quarters of their mathematics and literacy instructional time;

- teachers have developed reflective strategies and are more ready to accept change; - classrooms have become more conducive learning environments (display of students’ work,

teachers’ aid and literacy and numeracy corners); - teachers are gradually transferring their newly acquired pedagogical skills to other subjects; - teachers’ portfolios are of better quality, demonstrating a diversity of approaches and the

development of innovative materials and activities.

Results for students

Comparison between baseline and mid-term surveys demonstrated an improvement in students’ learning outcomes at the end of grade 3 in schools that participated in the programme.

In numeracy:Learning outcomes were improved in project schools, with no equity gap between students of Macedonian and Albanian language of instruction;

15 http://www.unicef.org/tfyrmacedonia/UNICEF_Math_Report_2012_ANG_za_na_nweb_so_korici.pdfhttp://www.unicef.org/tfyrmacedonia/JAZICNA_PISMENOST_Lan_Report_2013_ANG_za_na_web.pdf

8

http://www.unicef.org/tfyrmacedonia/UNICEF_Math_Report_2012_ANG_za_na_nweb_so_korici.pdf



There is a positive statistically significant difference in the improvement of learning outcomes between project schools and control schools.

In literacy:Better learning outcomes in project schools compared with control schools after programme implementation, particularly in reading;Students tested in Macedonian language are more successful than those in Albanian language, but the equity gap in learning narrowed after programme implementation, with students learning in Albanian experiencing greater gains.

BDE reports identify improved student engagement in the classroom, improved learning strategies and reasoning.

Results for schools

The programme has greatly contributed to increasing teachers’ cooperation within and across schools. Schools are better equipped to set up and sustain mentoring schemes and continuous professional development practices. This is a positive first step towards transforming Macedonian schools into professional learning communities although more work is needed in order to sustain and transform these initial results.

There is indicative evidence of School Development Plans addressing better learning assessment and students’ achievement. Some have made provision for literacy or numeracy focal points in the school, some have embedded action research on students’ performance and others have committed to monitor student achievement more closely.

Theory of Change

The programme is following a simple theory of change based on the assumption that when teachers see improvement in their students’ learning following a change in their teaching practice, only then does attitudinal change occur. This means that it is not the professional development alone that changes teachers’ attitudes and beliefs, but the experience of successful implementation of the newly gained knowledge and skills and their impact on students’ learning.

The programme strategies and activities are based on the model of teacher change according to which teachers define their success in terms of their students’ learning.

According to the model, the three major goals of professional development programmes are: change in the classroom practices of teachers, change in their attitudes and beliefs, and change in the

9


learning outcomes of students. The relationship among these outcomes is detailed and highly complex. Significant change in teachers’ attitudes and beliefs occurs primarily after they gain evidence of improvements in student learning. These improvements typically result from changes teachers have made in their classroom practices - a new instructional approach, the use of new materials or curricula, or simply a modification in teaching procedures or classroom format. The crucial point is that it is not the professional development per se, but the experience of successful implementation that changes teachers’ attitudes and beliefs. They believe it works because they have seen that it works, and that experience shapes their attitudes and beliefs.

The mechanisms to realize such theory follow the three principles deriving from this model:

1. Recognize that Change is a Gradual and Difficult Process for TeachersThe requirements of the new programme can add to teachers’ workload and anxiety that come with the need to change. No new programme is implemented uniformly, and teaching and learning are influenced by a range of situational and contextual factors. Therefore, there is a need for balance between the programme at national level and specific adaptation. Close collaboration between the MoES, institutions (programme developers) and teachers is of high importance. The new role of the BDE provides the opportunity for teachers to discuss concerns and relief anxiety related to programme implementation. Additional mechanisms are the regional learning teams which through the continuous process of school support ensure adaptions due to situational and contextual variables.

2. Ensure that Teachers Receive Regular Feedback on Student Learning Progress. If the use of new practices is to be sustained, the individuals involved need to receive regular feedback on the effects of their efforts. New practices are likely to be abandoned when evidence of their positive effects is lacking. The BDE support and the support by the regional learning teams based on the use of the fidelity tool have a specific focus on gathering evidence and providing feedback on results, both students’ result and results of teacher effectiveness. When teachers gain this evidence and see that a new programme or innovation works well in their classrooms, change in their attitudes and beliefs occurs.

3. Provide Continued Follow-Up, Support and PressureThe quality of the initial training is not the only crucial aspect to change. Since, according to the model, change occurs following implementation and evidence of improved student learning, continued follow-up, support, and pressure is even more crucial. Teachers support helps with those teachers whose motivation for change is not great and it provides the encouragement. BDE and the regional learning teams play this role too.

Stakeholder’s analysis (Roles/contributions to the Project and Interest in the Evaluation)

A range of stakeholders have 1) interest in and influence on this programme and 2) interest in the evaluation. The following stakeholders are considered as key: Ministry of Education and Science, Bureau for Development of Education, teachers in primary schools, Macedonian Civic Education Center and UNICEF Country office.

Below is the description of the key stakeholders involvement and interest in the evaluation. Teachers in the former Yugoslav Republic of Macedonia, in particular early grade teachers are

the main focus of this evaluation.

10


o Interest and involvement in the evaluation: Principal source of information on the relevance, effectiveness and sustainability of activities, obtained through Focus Group Discussions (FGDs) and Survey/Questionnaires on programme performance.

Implementing Partners: Ministry of Education and Science and the Bureau for Development of Education and Macedonian Civic Education Centre. BDE was the leading organization, whereas, MCEC was hired by UNICEF to provide logistical and expert support, specifically for: 1) conducting baseline studies as part of both numeracy and literacy components of the programme; 2) dissemination of teacher training in schools and certification of national trainers and teachers; and 3) providing support for the functioning of the regional learning (mentoring) teams. o Interest and involvement in the evaluation: The interest and stake of MoES and BDE in the

evaluation relates to UNICEF’s commitments to support access to quality education and holding UNICEF to account for the resources and interventions provided. MCEC interests in the evaluation relate mainly to their strategic and operational relationships with UNICEF, and substantive findings to improve any future education programmes. Information will be obtained in the form of Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs).

UNICEF Country Office provided international expertise. Miske Witt & Associates was hired to give support for 1) review of curricula in mathematics and language for early grades and review of teaching practice; 2) identifying international best practices in teacher education for potential replication in the country; and 3) training of national trainers, training BDE advisors to support dissemination and implementation in schools. o Interest and involvement in the evaluation: The CO will be involved in the evaluation

through development of the ToR and selection of the evaluation team (ET); provision of documents, reports, information and data to ET; interviewed as key informants; and will comment on the Inception and Evaluation Reports. Direct support to ET in country including administrative and logistic support for the evaluation missions. Introduce ET to key stakeholders. Participate in debriefings and provide feedback on preliminary findings and conclusions.

III. RATIONALE

Purpose The UNICEF country office will conduct an evaluation that is primarily summative with a purpose of assessing the results of UNICEF support in raising the quality of teaching and students achievements in early literacy and numeracy programme. The evaluation is undertaken at the end of UNICEF Country program (2010 -2015) signed with the Ministry of Education and Science.

Expected Users and Intended UseKey audiences for the evaluation are Ministry of Education and Science, Bureau for Development of Education, State Education Inspectorate, Civil Society Organizations and UNICEF. The evaluation findings will serve to:

Ministry of Education and Science developing action plans for implementation of the Education Strategy 2016-2020; evaluation of the programmes results and making evidence based decisions;

11


implementing the new system for teacher professional development;

Bureau for Development of Education monitoring and supporting teachers improving the language and mathematics curricula identifying and sharing high quality teaching practices continues professional development of teachers evaluation of the BDE management of the programmes and applying lessons learnt in

implementing innovative projects

State Education Inspectorate evaluating management of innovative approaches on a school level

UNICEF assessing the achievement of programme objectives assessing the capacities of the implementing partners using lessons learnt from these programme in implementation of similar programmes

The plan for dissemination of findings and recommendations from the evaluation is outlined in Annex 3.

IV. OBJECTIVES OF THE EVALUATION

The main objectives of this evaluation are to: Assess the relevance, efficiency, effectiveness, sustainability, coherence and, to the extent

possible, impact of the project interventions; Identify any unintended outcomes; Identify and document lessons learnt in relation to strategies and interventions used to

address certain critical bottlenecks; and Provide recommendations and guide the strategic planning for improved programmes for

Roma early inclusion and education as part of the joint partnership programmes with respective Governments.

V. SCOPE AND LIMITATIONSThe evaluation will cover the implementation and achievements in early literacy and early numeracy programme in the former Yugoslav Republic of Macedonia in primary education implemented in all schools in period 2009-2015. The evaluation will need to focus on three key areas of the programmes: 1) capacity building of all involved in the program; 2) management and support provided by the responsible institutions; and 3) implementation of the programmes in schools. The programmes need to be evaluated on the different levels starting from the policy development, policy implementation and its practical application. Considering that the teaching process in Macedonia is conducted in different languages of instructions, the evaluation will need to cover schools in Macedonian and Albanian language. It will also need to cover urban and rural schools in three main regions of the country (central, east and west). Since, the program was implemented in all primary schools in the country, the evaluation team will need to select representative sample of schools to evaluate the program. The evaluation team will need to propose sample type and minimal sample size that guaranty reliable information.

12


One of the limitations to the evaluation of the impact of the programmes will be the lack of system for national assessments and lack of participation in the international assessments (TIMSS, PIRLS) that can provide large scale and reliable data on students’ achievements. Existing external testing in grades four and five (based only on multiple choice items drown from publicly available item pool) that force memorizing, also can limit programs impact. During the program implementation new national curricula for mathematics (so called Cambridge curricula) were developed and implemented. Interference of the two programs approaches and trainings should be considered. Furthermore, assessing the aspect of support provided by the BDE, it might be affected by the high turnover of staff in the last couple of years. Shifts of the school directors during the program implementation also can influence schools management support to the programmes implementation.

VI. EVALUATION FRAMEWORK The proposed set of evaluation questions aligned to the evaluation criteria has been developed by UNICEF. They are listed below. This list can be refined by the evaluation team to be hired for this evaluation. They have been selected as a) the standard international criteria for development evaluation, as reflected in UNEG standards, to which UNICEF adheres to, and b) appropriately geared to the Purpose and Objectives of the evaluation, as set out above. Equity is a cross cutting principle which will be analysed as part of each of the criteria, as elaborated below.

1) Relevance Has the programme been aligned with government and BDE priorities/policies/reform

agendas? To what extent did the programme respond to the state priorities on improving early literacy

and early numeracy teacher’s skills and knowledge to improve the student’s achievements? Has the programme (content and delivery) been aligned with the Ministry of Education

prescribed curricula? How relevant have been the capacity building activities? Were the needs of the teachers addressed? To what extent the data collecting and monitoring activities developed with UNICEF support

are relevant for strengthening or/end upgrading the program activities and BDE capacities?

Impact: To what extent did the results contribute to raising the quality of teaching in early literacy

and early numeracy? To what extent did the programmes contribute to increasing students’ numeracy and

literacy? To what extent did results contribute to students’ attitudes and learning of the mathematics

and reading and writing? To what extent did the programmes influence the school support to teaching and learning

mathematics and reading and writing? To what extent did the Numeracy program contribute to strengthening the implementation

of the new national mathematics curricula (Cambridge curricula)?

2) Effectiveness: Have the planned results been achieved (quantitative and qualitative)? To what extent the programme contributed to creating or improving the curricula and/or

13


teachers pre-service and/or in-service preparation? To what extent the programme contributed to strengthening monitoring capacities of BDE? To what extent the programme contributed to strengthening the teachers’ capacities for

quality numeracy and literacy teaching? How effective was the support/mentoring provided to teachers? What management and

monitoring tools have been used and what tools could have been used? Are there any unintended results of the interventions?

3) Efficiency: How cost effective was the teacher training and teacher support? Could the same results have been achieved by using different strategies (or set of strategies)

with less resources? How well the implementation of activities has been managed? Were the management and

monitoring tools well developed and appropriately used, and what tools could have been used?

Did the Early Literacy and Numeracy programmes ensure coordination with other relevant programme interventions?

4) Sustainability: To what extent do the state institutions involved in the project have the capacities to use the

results and lessons learnt in improving further the literacy and numeracy teaching? Did the programme promote ownership over different programme activities? Did the

relevant partners own the results of the programme? Did the programmes contribute towards establishing innovative approaches towards

professional development of teachers?

Evaluation questions will be further refined and additional questions will be incorporated by the evaluation team as required during the inspection phase.

VII. METHODOLOGYThe methodology for evaluation will be proposed by the evaluator/s. It will need to respond to the set objectives and questions of the evaluation and to take into consideration defined programme output indicators (Annex 1). UNICEF and key stakeholders (MoES and BDE) will provide input in finalizing the methodology in order to secure high quality standards of the evaluation.

The programme has already established methodology and processes for monitoring and evaluation which could be a starting point for conducting the evaluation. In this context, the evaluators in the phase of desk review should consider: baselines, progress measurements and annual reports, and other available reliable data and materials collected through monitoring and evaluation activities (database of trained teachers; programme reports; school monitoring reports; surveys, etc.). Disaggregated data needs to be considered - e.g. disaggregated data by gender and ethnicity in teachers’ database, disaggregated data on students’ achievement by language of instruction. During the inception phase, the evaluation team is expected to agree with UNICEF on alternative approaches, including additional collection of data.

The evaluation methodology should demonstrate impartiality and lack of bias by relying on a cross-section of information sources (e.g. stakeholder groups, including beneficiaries, etc.) and aiming to

14


use if possible a mixed methodological approach to ensure triangulation of information through a variety of means. The participation of key stakeholders will be promoted, as much as possible, in all phases of the evaluation, including the planning, inception, fact-finding, reporting phases as well as the management response phase when determining the specific use the findings and recommendations, follow-up decisions, and dissemination strategies as relevant.

Generally, the evaluation will have to use a non-experimental design.

The strategy based on Human Rights Based Approach, including equity and gender equality and mainstreaming, will be used to assess the intervention areas as a general criterion, rather than as a stand-alone strategy.

Inception Phase: At inception stage, the evaluator(s) are expected to conduct a thorough review and analysis of the wide array of secondary data available in order to identify information gaps and other challenges and discuss solutions to address these. The methodology and the sample will be discussed and elaborated during the Inception Phase, when the site selection criteria and evaluation questions will be refined. At this point, the evaluator(s) should also develop a more precise evaluation work plan. The Evaluation Matrix, including the evaluation criteria and associated questions, indicators and prescribed data gathering tools and methods, will be developed by the evaluation team leader.

Desk Review: Prior to the start of field work desk review and analysis of existing project related documents as provided by UNICEF have to be finalized, existing international and national policies/priorities appropriate and connected with early literacy and numeracy programme in Macedonia have to be analysed, questionnaires focus group protocols (and possibly quick on-line survey) with list of questions/indicators against set objectives have to be developed. Refined evaluation questions and adjusted data collection, sampling (list of stakeholders to be interviewed and locations to be visited), and analysis methods, and operational work plan have to be finalized.

In-country data collection: Through field visits and meetings the data have to be collected: From key stakeholders (BDE advisors, master trainers, university professors that teach

mathematics and language teaching methodology), including government representatives using structured interviews;

From schools and teachers (the scope will depend on time and availability of these) using the structured observations of project activities within the project framework (and/or interviews);

From project partners using structured interviews.

Data analysis and report writing: Methods for ensuring validity and reliability at analysis stage have to include:

Triangulation – to confirm and corroborate results reached by different methods Complementarity - to explain and understand findings obtained by one method by applying

a second. E.g. explaining and understanding the distinctions around the design of particular strategies and interventions

Interrogation - where diverging results emerge from the application of different methods – these will need to be interrogated to either reconcile, or explain, the differences apparent.

Following the completion of the fact-finding and analysis phase, a draft and final report (in English) should be submitted.

15


Sources of information: Existing sources of information are available in Annex 4

UNEG Norms and Standards: The evaluation will apply the UNEG norms and standards, including evaluation criteria of relevance, efficiency, effectiveness, impact, and sustainability, and the UNEG ethical guideline, including UNICEF Procedure for Ethical Standards in Research, Evaluation, and Data Collection and Analysis (effective as of 1st April 2015), in order to ensure quality of evaluation process. Moreover, the evaluation will mainstream gender and human rights considerations, using gender-sensitive, child-sensitive and human rights-based lenses throughout.

VIII. WORK PLAN AND EVALUATION MANAGEMENT Evaluation PhasesThe evaluation process will consist of three phases: 1) Inception Phase, including desk review and drafting of inception report; 2) Data collection phase, including an appropriate mix of data collection methods, as indicated above; and 3) Analysis and reporting phase, in which following the completion of the fact-finding and analysis phase, a draft and final report (in English) should be submitted. The process will be guided by the schedule presented below.

Phases and deliverablesThe implementation timeframe of the evaluation: September 2016 – April 2017The main tasks and deliverables for the evaluation team as part of this assignment are presented in the table below:

Activity Deliverable Working days Who INCEPTION PHASE1. Preliminary desk review. Discussion

with UNICEF Country Office Memo on the result of a desk review

SEPTEMBER 20162 working days per consultant

International and National Consultant

2. Drafting of Inception Report that must include: refined evaluation questions and adjusted data collection, sampling (list of stakeholders to be interviewed and locations to be visited), and analysis methods, and operational work plan, including potential limitations of the evaluation according to data availability and reliability.

Inception Report SEPTEMBER 20165 working days for international consultant and 1 working days for national consultant to provide feedback

International Consultant and national consultant (from residence)

3. Revision based on the UNICEF feedback /peer review, according to the quality assurance procedures as per RSE SoPs.

Final Inception Report

OCTOBER 2016Within 2 weeks

1 working day

UNICEF/Peer ReviewInternational Consultant (from residence)

IMPLEMENTATION, DATA COLLECTION PHASE4. Interviews, FGD, meetings with

different stakeholders, lessons Data matrix, as provided within the

OCTOBER 20165 working days for

National and International

16


observation etc.

In country mission of the international consultant

Inception Report national consultant5 working days for International consultant

Consultants (in country mission)

ANALYSIS AND REPORTING PHASE5. Conduct analysis based on data

received/collected;Draft notes/report on the preliminary findings

NOVEMBER 20166 working days international cons.2 working days national consultant

National and International Consultant (from residence)

6. Develop the draft report Draft report DECEMBER 20168 working days for international consul.2 working days national consultant

International Consultant (from residence)

7. Feedback session with UNICEF, according to the quality assurance procedures as per RSE SoPs.

JANUARY 2017Within 2-3 weeks

1 working day

UNICEF/ peer review; National and International Consultant (from residence)

8. Submission of final report, including an extended executive summary; and a power point presentation of main findings and recommendations

Final Report, and PowerPoint presentation of main findings and recommendations (generic and country specific)

FEBRUARY 20175 working days

International Consultant(from residence)

9. Presentation of findings PowerPoint presentation findings and recommendations specific for each country

MARCH 20171 working day per consultant

National Consultant (presentation) International consultant (skype session)

10. Dissemination of the report Posting the report on the UNICEF website

APRIL 2017 UNICEF Education Team

Total number of days 14 working days for the national consultant; 33 working days for the international consultant.

The above mentioned tasks to be performed are intended as a guide and should not be viewed as an inflexible specification as it may be modified following discussion with the selected consultant or firm/institution.

The reports will be delivered to UNICEF Education Team, based on the calendar in the table above. All reports should be submitted electronically in English language.

17


18


IX. TEAM COMPOSITION, ROLES AND RESPONSIBILITIESTeam Composition, Roles and Responsibilities

Lead Responsibilities and Components

Evaluation team leader/International Consultant

Coordinating evaluation processes and overseeing the tasks of the evaluation team, in communication with the UNICEF Education team.

Providing daily guidance and manages the national consultant. Maintaining timeframe of the evaluation, identifying any potential

issues, bringing these to the attention of the relevant stakeholders, and resolving challenges at the earliest opportunity.

Desk review of the core documents. Participating in the field work with the national consultant. Developing the Inception Report, including stakeholder identification

and sampling; finalising the evaluation method and tools and ensuring their validity, reliability and quality, and incorporating relevant comments from UNICEF.

Quality assurance of data collection, storage, analysis and triangulation and all other inputs provided by the evaluation team members.

Safe guarding standards of ethics, data protection, and independence. Drafting an evaluation report in line with UNICEF and UNEG standards,

and finalising the report and presentation on the basis of comments received.

Evaluation Team Member/National Consultant

Pre-mission review of all core documents. Participating in skype discussions with International Consultant and

UNICEF Education team. Conducting and systematically and rigorously documenting interviews,

data collection, findings, and providing inputs to the evaluation team leader needed for drafting the evaluation report.

Safe guarding standards of ethics, data protection, and independence. Present evaluation findings and recommendations (respective

country).

UNICEF Country Office

UNICEF Education team will fully support the evaluation through: Drafting evaluation TOR. Recruitment and management of International and National

Consultants/or Consulting firm/institution. Organize logistical arrangements during the in-country mission of the

International Consultant. Provide support in organizing evaluation meetings, focus group

discussions in line with the schedule of activities. The provision of necessary documentation. Overseeing the overall evaluation schedule. Submission of final draft report to the Representative for approval. Uploading evaluation report on Evaluation Research Database,

developing the management response and tracking implementation of recommendations.

Ensuring wide dissemination and follow up of the recommendations

19


Lead Responsibilities and Components

from the evaluation. Mainstreaming of the key recommendations into the next

programmes/interventions as appropriate.

Ministry of Education and Science and Bureau for Development of Education

Participate in the evaluation as key source of information. Review evaluation report and recommendations. Ensuring wide dissemination of the evaluation findings and

recommendations Developing plan for implementation of the recommendations.

Evaluation Team: Required qualifications and areas of expertise

Individuals, group of individuals as well as consultancy firms/institutions having the requisite skills/ experience are eligible to apply. The evaluation team should be consisted of international consultant/s and one national consultant.

The competencies required from the international consultant/s are the following: Advanced degree in Social Sciences or related fields; Extensive experience, knowledge and expertise in M&E, in conducting research and different

kinds of surveys, assessments, and evaluations; Proven extensive experience in quantitative and qualitative data collection and analysis; Excellent analytical and report writing skills; Experience in monitoring and evaluation of education or other social programmes; Previous experience of working with UNICEF in evaluations in the Central and Eastern Europe/

Commonwealth of Independent States (CEE/CIS) region is an asset; Excellent writing and oral skills in English and ability to synthetize complex information; and Proven ability to work in multi-ethnic/multi-language environment required. Expertize in early child education, gender equity and human rights is an asset.

The competencies required for the national consultant are the following: University degree in Social Sciences or related fields; Professional experience in conducting research and different kinds of surveys, assessments, and

evaluations especially in education; Extensive knowledge of the national education context and familiarity with the concept of Child

Friendly School; Familiarity with early literacy and numeracy teaching and learning. Proven experience in quantitative and qualitative data collection and analysis; Excellent analytical and report writing skills; Proven ability to work in multi-ethnic/multi-language environment required; Previous experience of working with UNICEF will be considered an asset. Mastery of English and Macedonian language; Albanian language is an asset.

The evaluation team is responsible to ensure that the process is in line with the United Nations Evaluation Group (UNEG) Ethical Guidelines (http://www.unevaluation.org/document/detail/102) and UNICEF Procedure for Ethical Standards in Research, Evaluation, Data Collection and Analysis

20

http://www.unevaluation.org/document/detail/102


(http://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF). The evaluator(s) should be sensitive to beliefs, manners and customs and act with integrity and honesty in their relationships with all stakeholders. Furthermore, they should protect the anonymity and confidentiality of individual information. All participants should be informed of the context and purpose of the evaluation, as well as of the confidentiality of the information shared.

X. PRODUCTS TO BE DELIVERED AND STRUCTURE OF EVALUATION REPORT

The report should be provided in electronic version in English, in the required UNICEF format. All objects/graphics in the evaluation report must be editable to allow for eventual translation and/or reformatting. Presentations may use PowerPoint or other formats, as appropriate. The Evaluation Report should include an extended executive summary, description of sampling and evaluation methodology used, data collection instruments, types of data analysis, assessment of methodology (including limitations), findings, conclusions, recommendations, and lessons learned. It should also contain Annexes, including: Terms of Reference, data collection tools, developed list of indicators and questionnaires and other relevant information as described below.

The deliverables will include:a) The Inception Reportb) The Evaluation Report.

Following is the proposed structure for the inception and evaluation report.

Structure of the Inception Report Response to the TOR Evaluation Framework Methodology Potential limitations of the evaluation according to data availability and reliability

Structure of the Evaluation Report (Tentative) Title Page Table of content List of Acronyms Executive Summary Object of the Evaluation Acknowledgements Evaluation Purpose, Objectives and Scope Evaluation Methodology Findings Conclusions and Lessons Learned Recommendations Case Studies Annexes

The structure of the final report will be further discussed with the Evaluation Team (during the Inception Phase).

The Evaluation Report should comply with UNICEF Evaluation Report Standards as outlined at http://www.unicef.org /evaldatabase/files/UNICEF_Eval_Report_Standards.pdf . Its quality will be

21

http://www.unicef.org/evaldatabase/files/UNICEF_Eval_Report_Standards.pdf

http://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF

http://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF


assessed through UNICEF Global Evaluation Reports Oversight System on the basis of these standards.

XI. ETHICAL STANDARDS AND SAFEGUARDSThe UNEG Guidance on Human Rights and Gender Equality in Evaluation i will be fully applied throughout. In addition to investigating and analysing attention to human rights through the evaluation questions, attention will be paid to human rights dimensions during identification of stakeholders and their interests, during selection of interviewees and in the selection of methods, timeframe etc. The evaluation will be conducted in full coherence with the UNEG Ethical Guidelines and Code of Conduct.

Key features to be applied are: Respecting gender and human rights principles throughout the Evaluation process,

including: the protection of confidentiality; the protection of rights; the protection of dignity and welfare of people; and ensuring informed consent. Feedback will be provided to participants wherever possible, and data validation will take place at all levels with participant consent, including with UNICEF.

Maximizing the degree of participation of stakeholders in the Evaluation itself wherever feasible and a commitment to using participatory approaches in field studies in particular.

Ensuring that the Evaluation matrix integrates the Convention on the Elimination of all forms of Discrimination against Women (CEDAW), Convention on the Rights of the child (CRC) and human rights commitments.

Disaggregating data by gender, geography, and social groups where feasible. Ensuring that outputs use human-rights and gender-sensitive language. Special care taken in

relation to any contact that the evaluation team has with children.

XII. QUALITY ASSURANCEThe Evaluation Team members takes the primary responsibility for the Quality Assurance process, ensuring a rigorous process of data collection, analysis and synthesis to minimise errors. The process of on-going triangulation and verification, will help ensure this, as will the proposed validation session. No evaluation team member will have any potential conflict of interest with the evaluation object (e.g. will not be involved in the development or implementation of the programmes, will not be employed in the state institutions that have a role in the programmes implementation etc.) or UNICEF.

XIII. RESOURCES AND PAYMENT SCHEDULE The budget for the evaluation will be subject to financial offers agreed with the selected candidates, based on ‘best value for money’ selection process.

Payment shall be made as follows: 40% will be paid upon submission of the Inception report 60% will be paid upon submission of the final report

The UNICEF CO will provide interprets during the in-county visits. The evaluators will be provided with office space, vehicles for site visits and official meetings, logistical support for meeting and visa procedures. Laptops or computers will not be provided.

22


XIV. REMARKS AND RESERVATIONS UNICEF reserves the right to withhold all or a portion of payment if performance is unsatisfactory, if work/deliverables are incomplete, not delivered or for failure to meet deadlines.

All material developed will remain the copyright of UNICEF. Evaluators are responsible for their performance and products. UNICEF reserves the copyrights and the products cannot be published or disseminated without prior permission of UNICEF.

Candidates interested in the consultancy should submit a proposal with approximate methodological proposal, time line, and resume of the evaluators who will take part in evaluation process. Narrative proposals should be submitted to [email protected] . The financial part of the proposal should be sent in a separate document to the following email address [email protected] . In the subject of the email messages please state: Proposal for Early Numeracy and Literacy Programme Evaluation – Confidential

23

mailto:[email protected]

mailto:[email protected]


Annex 1- Logical Framework

Logical Framework

Country Programme Development Outcome: By end of 2015, as a result of social sector reforms, the Government addresses targeted disparities in the quality of and access to basic social services for vulnerable children and familiesIndicator: The ratio of boys and girls in institutional care to children in alternative family or community-based Baseline:2.6/1 Country Programme Development Output: By of 2015, the Ministry of Education and Science and municipalities ensure that the national education system is inclusive (regardless of ethnicity, ability, and socio-economic background) and effective, in line with CFS standardsIndicator: % of 1-3 grade teachers that are applying new teaching and learning methods in line with CFS standards, including inclusive education.Indicator: Availability of revised by-laws and standards based on the social model of inclusive education Indicator: Availability of funding formula for allocating resources per student Early Literacy and Numeracy Programme Output: By end of 2015, the Ministry of Education and Science ensures improved quality basic education for all students including marginalized groupsIndicator: Raised achievements in literacy and numeracy of the lower primary studentsIndicator: Improved capacities of the officials in the educational institutions as a part of the system for quality assurance

EARLY LITERACY AND NUMERACY PROGRAMME Interventions and brief description Indicators Status /

timeframeImplement

ed atImplemented by Means of verification

Preparatory phase - An in-depth analysis of the early grades curricula and teacher instruction in

mathematics and language;- Identification of international best practices in teacher education related to

early numeracy and literacy instruction;- Development of new teacher training programmes based on the latest research

for teacher professional development;- Teacher manuals for both numeracy and literacy were developed and

distributed to all trained teachers;- Training and certification of national trainers and BDE advisors;- Baseline studies conducted to measure the initial situation of both early grade

- Teacher training programme identified

- Training materials developed and available

- Baseline studies conducted

2009-2010 National level

- Miske Witt and Associates Inc.

- Working groups (BDE advisors and selected teachers)

- MWAI reports - Teacher manuals- Numeracy baseline

report 2009- Literacy baseline 2010

24


teachers’ knowledge and student outcomes at the end of grade. Total estimated budget – 190,000 USD Pilot

schoolsImplementation phase- Roll out of the training to all early grade teachers in the country through a

cascade training process complemented with continuous support provided to teachers at school level;

- Rigorous certification of teacher competencies led by the BDE; - Development of tools and training of all national trainers and BDE advisors on

use to ensure quality of implementation;- Establishment of Regional Learning teams as a professional networking

mechanism to offer teachers an opportunity to receive additional support as they change their practice;

- Conducting progress studies to measure progress after 3 years of implementation in literacy and numeracy.

Total estimated budget – 950,000 USD

- At least 95% of all early grade teachers trained in the new programmes

- At least 30% of all early grade teachers certified in application of the two programmes

- Mechanisms in place to ensure sustainability of interventions

- Raised students results in numeracy and literacy

2010-2015 National level

- Miske Witt and Associates Inc.

- Bureau for Education Development

- NGO- Macedonian Civic Education Center

- National trainers

- Numeracy progress assessment report 2012

- Literacy progress assessment report 2013

- Literacy effectiveness of training in Macedonia European Education Macedonia Article

- MCEC reports - Improving learning

outcomes in literacy and numeracy (Case study)

- Literacy effectiveness of training in Macedonia, European Education Macedonia, Article

- Teacher Education Programme on Early Numeracy and Literacy in FYR Macedonia Monitoring note 1 and Monitoring note 2.

25


Annex 2: UNICEF Core Roles 16

UNICEF plays a key role supporting changes in the system conducive of reduction of equity gaps and child rights violations. There is consensus that the following Core Roles are indispensable for a sustainable UNICEF engagement and its universal presence in support of results and the realization of the rights of children everywhere:

The ‘Voice’ for children and adolescents -- advocating and communicating around key national policies, social issues, mind-sets and attitudes;

Monitoring and evaluation – assisting independent assessments of the functioning of the Child Rights guarantee systems, the progressive realisation of child rights and the reduction in equity gaps in child well-being;

Policy advice and technical assistance – through well-designed UNICEF positions (based on local, regional, international best practices) on key issues, supporting the development of the normative frameworks related to specific national legislation, policy or programme as well as private sector standards that can improve equity;

Leveraging resources from the public and private sectors – accompanying and redirecting reforms, including those supported by the EU, IFIs, bilaterals and national/multi-national corporations;

Facilitating national dialogue towards child friendly social norms – bringing together government, private sector and civil society, as well as convening divergent forces to enhance public debate, participation and action around equity and child rights;

Enabling knowledge exchange – fostering horizontal cooperation and exchange of experience among countries and regions on ‘what works’ for enhancing child well-being and equity.

modeling/piloting demonstrating how system could meaningfully evolve to reduce equity gaps and children’s rights violations

16 Regional Knowledge and Leadership Agenda Generic Theory of Change underlying UNICEF’s approach in CEE/CIS Region, January 2013

26


Annex 3: Dissemination Plan

Planned Dissemination Activities

Aim Focal Point

Timing

Strategic discussion partners and implementers on draft evaluation findings, lessons learnt and next steps.

Validate all findings of the evaluation, and prepare final comments (if any), select priority action and transform them in the recommendations for future action. In the same time it should be form of capacity building of partners in

UNICEF Education team

February 2017

Present the overall findings and recommendations in the final evaluation report to the key education stakeholders

Introduce the key education stakeholders (MOES, BDE, SEI, MCEC, primary schools) to the overall findings and recommendations of the evaluation report and agree on future actions.


March 2017

Evaluation report published in UNICEF website and social media for wider dissemination.

Inform child UNICEF partners and public at large on the results of the evaluation of the project


April 2017

27


Annex 4: Sources of Information available from UNICEF1. Child Friendly Schools: A Situation Analysis

http://www.unicef.org/tfyrmacedonia/CFS_BASELINE_STUDY_FINAL_ENG(4).pdf2. Child Friendly Schools: Case Study

http://www.unicef.org/tfyrmacedonia/MK_CFS_CaseStudy(3).pdf3. Thinking mathematics in lower grades - Report of the baseline study


4. Thinking mathematics in lower grades - Research of the achievements at the end of the first cycle of the realization of the project http://www.unicef.org/tfyrmacedonia/UNICEF_Math_Report_2012_ANG_za_na_nweb_so_korici.pdf

5. Language literacy in the early grades - report of the baseline study http://www.unicef.org/tfyrmacedonia/JAZICNA_PISMENOST_Lan_Report_2010_ANG_za_na_web.pdf

6. Language literacy in the early grades - Research of the achievements at the end of the first cycle of the realization of the project http://www.unicef.org/tfyrmacedonia/JAZICNA_PISMENOST_Lan_Report_2013_ANG_za_na_web.pdf

7. UNICEF (Former Yugoslav Republic of Macedonia) Teacher Education Programme on Early Numeracy and Literacy http://www.oecd.org/edu/ceri/menus/ceri/UNICEF.MCD.SystemNote.pdf

8. Teacher Education Programme on Early Numeracy and Literacy in the former Yugoslav Republic of Macedonia Monitoring note 1 http://www.oecd.org/edu/ceri/UNICEF.MCD.MonitoringNote1.pdf

9. Teacher Education Programme on Early Numeracy and Literacy in the Former Yugoslav Republic of Macedonia Monitoring Note 2 http://www.oecd.org/edu/ceri/Unicef-Former-Yugoslav-Republic-of-Macedonia-Monitoring-Note-2.pdf

10. Thinking mathematics – teachers training manual http://umd.gov.mk/wp-content/uploads/2013/10/Matematika-so-razmisluvanje.pdf

11. Numeracy additional materials http://umd.gov.mk/wp-content/uploads/2013/12/Priracnik-Matematika%20so%20razmisluvanje%20vo%20pocetnite%20oddelenija-2009.pdf

12. Language literacy – teachers training manual http://umd.gov.mk/wp-content/uploads/2013/10/Prirachnik-za-jazichna-pismenost.pdf

13. Literacy additional materials http://umd.gov.mk/wp-content/uploads/2013/10/Da-staneme-vistinski-chitateli.pdf

14. Teachers certification guidelines and criteria http://umd.gov.mk/wp-content/uploads/2014/01/Kriteriumi-i-nasoki-za-sertifikacija-so-formulari.pdf

15. Action Research in Primary Education http://www.unicef.org/tfyrmacedonia/Akciono_istrazuvanje_vo_vospitno-obrazovnata_praktika-MK.pdf

16. Formative Assessment for Lower Primary Education http://www.unicef.org/tfyrmacedonia/Formativno_ocenuvanje_vo_oddelenska_nastava-_MK.pdf

17. National curricula for primary education, http://bro.gov.mk/?q=osnovno-obrazovanie . 18. Adapted curricula for mathematics and science by Cambridge International Examination

Centre http://bro.gov.mk/index.php?q=mk/node/163 .

28

http://bro.gov.mk/index.php?q=mk/node/163

http://bro.gov.mk/?q=osnovno-obrazovanie

http://www.unicef.org/tfyrmacedonia/Formativno_ocenuvanje_vo_oddelenska_nastava-_MK.pdf

http://www.unicef.org/tfyrmacedonia/Formativno_ocenuvanje_vo_oddelenska_nastava-_MK.pdf

http://www.unicef.org/tfyrmacedonia/Akciono_istrazuvanje_vo_vospitno-obrazovnata_praktika-MK.pdf

http://www.unicef.org/tfyrmacedonia/Akciono_istrazuvanje_vo_vospitno-obrazovnata_praktika-MK.pdf

http://umd.gov.mk/wp-content/uploads/2014/01/Kriteriumi-i-nasoki-za-sertifikacija-so-formulari.pdf

http://umd.gov.mk/wp-content/uploads/2014/01/Kriteriumi-i-nasoki-za-sertifikacija-so-formulari.pdf

http://umd.gov.mk/wp-content/uploads/2013/10/Da-staneme-vistinski-chitateli.pdf

http://umd.gov.mk/wp-content/uploads/2013/10/Da-staneme-vistinski-chitateli.pdf

http://umd.gov.mk/wp-content/uploads/2013/10/Prirachnik-za-jazichna-pismenost.pdf

http://umd.gov.mk/wp-content/uploads/2013/12/Priracnik-Matematika%20so%20razmisluvanje%20vo%20pocetnite%20oddelenija-2009.pdf

http://umd.gov.mk/wp-content/uploads/2013/12/Priracnik-Matematika%20so%20razmisluvanje%20vo%20pocetnite%20oddelenija-2009.pdf

http://umd.gov.mk/wp-content/uploads/2013/10/Matematika-so-razmisluvanje.pdf

http://www.oecd.org/edu/ceri/Unicef-Former-Yugoslav-Republic-of-Macedonia-Monitoring-Note-2.pdf

http://www.oecd.org/edu/ceri/Unicef-Former-Yugoslav-Republic-of-Macedonia-Monitoring-Note-2.pdf

http://www.oecd.org/edu/ceri/UNICEF.MCD.MonitoringNote1.pdf

http://www.oecd.org/edu/ceri/menus/ceri/UNICEF.MCD.SystemNote.pdf









http://www.unicef.org/tfyrmacedonia/MK_CFS_CaseStudy(3).pdf

http://www.unicef.org/tfyrmacedonia/CFS_BASELINE_STUDY_FINAL_ENG(4).pdf

i UNEG (2011) Integrating Human Rights and Gender Equality in Evaluation – Towards UNEG Guidance

te - unicef · web viewin 2011 external testing of students in primary and secondary education was...

Documents