multi-country evaluation of regional knowledge and ... · final evaluation report: multi-country...

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas i

Area 4:Inclusion of all Out of School Children in Quality Learning in CEE/CIS

LIST OF ABBREVIATIONS

ANER Adjusted Net Enrolment Rate

CBPR Conducive Behaviour for Participation Ratio

CCT Conditional Cash Transfer

CDP Common Development Plan

CEDAW Convention on the Elimination of all Forms of Discrimination against Women

CEE Central and Eastern Europe

CESCR Committee on Economic, Social and Cultural Rights

CfS Child-Friendly Schools

CIS Commonwealth of Independent States

CO Country Office

COAR Country Office Annual Report

CPAP Country Programme Action Plan

CPD Country Programme Document

CRC Convention on the Rights of the Child

CWD Children with Disabilities

DAC Development Assistance Committee

DHS Demographic and Health Survey

DILS Delivery of Improved Local Services

ECD Early Childhood Development

ECE Early Childhood Education

EFA Education For All

ERI Education Research Initiative

ESOMAR The World Association for Social, Opinion and Market Research

FGD Focus Group Discussion

GMR Global Monitoring Report

HQ Headquarters

HRBA Human Rights-based Approach (to programming)

HRBP Human-Rights-Based Approach to Programming

IFI International Financial Institution

IPA Instrument for Pre-Accession Assistance (EU)

ISCED International Standard Classification of Education

ISSA International Social Security Association

KCF Kosovo (UNSCR 1244) Curriculum Framework

KEC Kosovo (UNSCR 1244) Education Centre

KESP Kosovo (UNSCR 1244) Education Sector Plan

KTA Key Thematic Area

LFS Labour Force Survey

LSBE Life skills-Based Education

M+E Monitoring and Evaluation

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas ii


MCE Multi-Country Evaluation

MCEC The former Yugoslav Republic of Macedonia Civic Education Centre

MDG Millennium Development Goals

MEST Ministry of Education, Science and Technology (Kosovo [UNSCR 1244], the former Yugoslav Republic of Macedonia)

MICS Multiple Indicator Cluster Survey

MODS Mreža organizacija za decu Srbije [Network of Organisations for Children of Serbia]

MoEST Ministry of Education, Science and Technology (Kosovo (UNSCR 1244))

MoESTD Ministry of Education, Science and Technological Development (Serbia)

MoL Ministry of Labour

MoRES Monitoring of Results for Equity Systems

MoSA Ministry of Social Affairs

MoU Memorandum of Understanding

MTR Mid-Term Review

MTSP Medium Term Strategic Plan

NGO Non-Governmental Organisation

NIE Network for Support to Inclusive Education (Serbia)

OOSC Out-Of-School-Children

OSCE Organisation for Security and Cooperation in Europe

OSI Open Society Institute

PA Pedagogical Assistant (Serbia)

PEIS Primary Education Institution Standards (Turkey)

PISA Programme for International Student Assessment

PITR Pupil Initiative Talk Ratio

PSA Physical and Sexual Abuse

PTR Pupil:Teacher Ratio

RAE Roma, Ashkali and Egyptians

RKLA Regional Knowledge and Leadership Agenda

RO Regional Office

RT Round Table

RTC Regional Theory of Change

SITAN Situation Analysis

ToC Theory of Change

ToR Terms of Reference

TTC Teacher Training College

UNCT United Nations Country Team

UNDAF United Nations Development Assistance Framework

UNESCO United Nations Educational, Scientific and Cultural Organisation

UNICEF United Nations Children’s Fund

UNSCR United Nations Security Council Resolution; regarding Kosovo, reference is being made to Resolution 1244 (1999). Adopted by the Security Council at its 4011th meeting, on 10 June 1999.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas iii


GLOSSARY OF SPECIFIC TERMINOLOGY

Dropouts

(or ‘pushouts’)

Children of school age that leave school before completing compulsory

education or who are no longer attending school. There is no standard

definition of ‘no longer attending school’1. The term ‘drop out’ implies

that children who leave school do so of their own volition. ‘Push outs’ is

increasingly used to emphasise that the school is responsible for ensuring

all children’s education and that when children leave school early it is

school and system factors that lead them to do so.

In countries where upper secondary and/or pre-primary education is not

compulsory, children that leave these levels of schooling are not

considered as dropouts by government and thus are not counted as out of

school in this evaluation report.

Early School Leaver, as

defined by

EUROSTAT

Young people aged between 18 and 24 years that have completed

International Standard Classification of Education (ISCED) Level 2

(lower secondary education) or below and are not currently enrolled in

education or training.

Inclusive education, as

defined in the

Salamanca Statement

Inclusive education promotes the “recognition of the need to work towards

‘schools for all’ – institutions which include everybody, celebrate

differences, support learning, and respond to individual needs”

(UNESCO, 1994).

Inclusive Education systems are those that have developed schools based

upon: ‘A child-centred pedagogy capable of successfully educating all

children, including those who have serious disadvantages and disabilities.

The merit of such schools is not only that they are capable of providing

quality education to all children; their establishment is a crucial step in

helping to change discriminatory attitudes, in creating welcoming

communities and in developing an inclusive society. A change in social

perspective is imperative’ (UNESCO, 1994; Salamanca Statement and

Framework for Action).

Inclusive Education efforts incorporate but are not limited to work to

include children with disabilities in mainstream schools.

1 In the MICS household survey, a child is considered to be out of school when he or she is not currently attending

school at the time the survey is taken, even though he or she may be enrolled in school. This is an important point

– that ‘attendance’ rather than ‘enrolment’ should be used, as a child may be enrolled but never attend school. For

this reason, a fixed period of absenteeism from school should be adopted which indicates that a child is no longer

attending school, and is therefore out of school. It is proposed that a child is considered to be no longer attending

school when she or he not attended school for the past 20 school days without a valid reason. It is also important

that school enrolment and attendance should include all types of schools and education institutions, including

public, private and all other institutions that provide organised educational programmes.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas iv


Out of School Children Out of School Children are defined according to five main categories:

Category 1 is the children of pre-primary school age (1 year before

the start of compulsory education) not enrolled in a pre-primary

programme;

Categories 2 and 3 are children of primary and lower secondary

school age, respectively, that are not enrolled, will enter late or

have dropped out of primary or lower secondary school;

Categories 4 and 5 are children that are currently enrolled in

school but are at risk of dropping out.

Primary education Primary education is the first stage of compulsory education. Children

receive primary education between the ages of about five to about eleven.

The number of primary education years may vary with a time span in

between six to nine years.

Standards2 Broad goal statements that define what stakeholders should know and be

able to do across an educational system. “Standards are statements that

specify an expectation for achievement. They may be used as a basis of

comparison in measuring or judging capacity, quality, value, or quantity”

(Kagan & Britto, 2005, p.2).

Student learning

outcome

Concise measurable statement that specifies what students will

know, be able to do or be able to demonstrate when they have

completed/ participated in a programme/course/project or

received a service. Outcomes are usually expressed as knowledge,

skills or attitudes;

A measurable result of a specific, planned educational experience

for students.

Quality education3 Quality education includes:

Learners who are healthy, well-nourished and ready to participate

and learn, and supported in learning by their families and

communities;

Environments that are healthy, safe, protective and gender-

sensitive, and provide adequate resources and facilities;

Content that is reflected in relevant curricula and materials for the

acquisition of basic skills, especially in the areas of literacy,

numeracy and skills for life, and knowledge in such areas as

gender, health, nutrition, HIV/AIDS prevention and peace;

2 Developing CfS Standards, final report 2010. 3 Defining Quality in Education, UNICEF Working Paper Series 2000. This definition allows for an understanding

of education as a complex system embedded in a political, cultural and economic context.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas v


Processes through which trained teachers use child-centred

teaching approaches in well-managed classrooms and schools and

skilful assessment to facilitate learning and reduce disparities;

Outcomes that encompass knowledge, skills and attitudes, and are

linked to national goals for education and positive participation in

society.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas vi


TABLE OF CONTENTS

Executive Summary x

1 INTRODUCTION 1

1.1 Background 1

1.2 Purpose and objectives of the evaluation 2

1.3 Underlying Theory of Change in relation to the evaluation object 3

1.3.1 Regional Theory of Change and reflexions at country levels 3

1.3.2 Mainstreaming the Child-friendly School approach 4

1.3.3 A programmatic shift towards a focus on systemic change 5

1.3.4 Monitoring for equity: the MoRES framework 5

1.3.5 Human Rights-based approach of programming 7

2 OVERALL FRAMEWORK OF THE EVALUATION 8

2.1 Scope and context of the evaluation 8

2.2 Target countries and territories for the evaluation 8

2.3 Strategic principles and ethical considerations 9

2.3.1 Participatory approach 9

2.3.2 School selection for field visits 10

2.3.3 Framework for interviews and focus group discussions 11

2.3.4 Protection of informants 11

2.3.5 Capacity building and transfer of knowledge 12

3 EVALUATION FRAMEWORK AND METHODOLOGY 13

3.1 Extensive document/desk review 13

3.2 Review of national and international education outcome data 14

3.3 Key informant interviews and Focus Group Discussions (FGDs) 15

3.4 Self-assessment by national education and UNICEF CO staff 16

3.5 Qualitative and quantitative data analysis 16

3.5.1 Theoretical background 16

3.5.2 Overview of evaluation questions, indicators and related information 17

3.5.3 Methodological limitations 18

3.5.4 Data reliability 18

3.6 Sociological Sensing 18

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas vii


4 ISSUES HAVING EMERGED DURING THE INCEPTION PHASE 20

4.1 Inception phase up to the completion of the draft inception report 20

4.2 Inception phase up to the completion and approval of the final inception report 20

4.3 Key outcomes from the pilot visit to Kosovo (UNSCR 1244) 20

5 EVALUATION FINDINGS 22

5.1 Changes in the lives of children (impact results) 23

5.1.1 Equitable access to basic education 23

5.1.1.1 Girls and boys 24

5.1.1.2 Gender parity 29

5.1.1.3 Ethnic minorities 31

5.1.1.4 Children with Special Educational Needs (SEN) 33

5.1.1.5 Children from disadvantaged backgrounds 34

5.1.2: Education quality and learning achievements 36

5.1.3 Equality in learning opportunity 40

5.2 Effectiveness of systemic changes for the most vulnerable children 44

5.3 Relevance of UNICEF’s approach 46

5.3.1 Relevance towards the most vulnerable 46

5.3.2: Relevance towards prevailing education sector policies and

international standards 47

5.3.3: Relevance towards the work of national partners 47

5.4 Contribution of UNICEF to system changes through its core roles (efficiency) 49

5.4.1 Policy advice and technical assistance 52

5.4.2 Modelling 55

5.4.3 Facilitating national dialogue towards child-friendly social norms 57

5.4.4 Enabling knowledge exchange 58

5.4.5 Monitoring and evaluation 60

5.4.6 Leveraging resources from the public and private sector 61

5.4.7 Giving children and adolescents a voice 66

5.5 Sustainability of results for children and system changes 66

5.6 Cross-cutting issues of a Human Rights-based approach 69

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas viii


6 KEY CONCLUSIONS AND RECOMMENDATIONS 71

6.1 Summary of the context of the visited countries and territories 71

6.1.1 Armenia 71

6.1.2 Kosovo (UNSCR 1244) 71

6.1.3 The former Yugoslav Republic of Macedonia 71

6.1.4 Serbia 72

6.1.5 Turkey 72

6.2 Summary conclusions as a response to the evaluation questions 72

6.2.1 Impact results 72

6.2.2 Effectiveness 73

6.2.3 Relevance 74

6.2.4 Efficiency 74

6.2.5 Sustainability 75

6.3 Recommendations 75

6.3.1 General recommendations 76

6.3.1.1 Systems change contributions 76

6.3.1.2 UNICEF contributions to system changes 77

6.3.2 Specific recommendations 79

6.3.2.1 Recommendations for the Regional Office 79

6.3.2.2 Recommendations for the Country Offices 79

7 ADDITIONAL LESSONS LEARNT 82

Details on Regional Consultant Team 83

List of persons consulted 84

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas ix


Appendices

Appendix 1: Terms of Reference

Appendix 2: Work Plan

Appendix 3: List of documents consulted

Appendix 4: Evaluation Matrix

(Final version following the validation at the team workshop

during the pilot phase in Kosovo (UNSCR 1244), 09-12

September 2013)

Appendix 4a: Assessment of general answerability of evaluation questions (as

contained in Inception Report)

Appendix 4b: Assessment of achievement of individual indicators contained in

the Evaluation Matrix

Appendix 5: Data collection tools

Evaluation Matrix template for completion after Interviews/

FGDs

Overview of Interview/FGD/Round Table items against

indicators and MoRES Determinants

Interview Sheets INT-1a, INT-1b, INT-2, INT-3

Round Table Discussion Sheet RT-1

Focus Group Discussion Sheets FGD-1, FGD-2, FGD-2

Recording Sheet for Interviews

Recording Sheet for Round Table Discussion

Recording Sheet for Focus Group Discussions

Classroom Observation Sheet CO-1

Self-Assessment Sheets SAS-1 and SAS-2

Appendix 6: Logic model for UNICEF’s Generic Theory of Change

Appendix 7: Programme for the country visits

Appendix 8: Evaluation matrices for the five visited countries and territories

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas x


EXECUTIVE SUMMARY

Background

Although national primary school enrolment rates in the CEE/CIS region are, on average relatively

high, millions of children nevertheless remain without access to equality education. UNESCO estimated

in 2010 that 3.7 million children of primary and lower-secondary school age, 1.6 million children of

pre-primary school age and 12 million adolescents were out of school in CEE/CIS. Children and

adolescents out of school are those from the most socially, culturally and economically marginalised

communities and thus are the hardest to reach.

UNICEF’s vision in CEE/CIS is that every child in the region will access and complete a basic education

of good quality. To support this vision UNICEF has, for the past decade, implemented a ‘system’

approach to programming to contribute to concrete changes at institutional, societal and individual

levels.

Objective, scope and methodology

In 2012, UNICEF decided to commission an independent evaluation of its basic education work in the

region as part of a series of thematic multi-country evaluations framed around the organizations global

and regional strategy.

The overall goal of this Multi-Country Evaluation was: (i) to assess the extent to which UNICEF's

programme interventions contributed to results at impact level for children – in terms of reduction in

the number of children out of school and improved quality of education - learning outcomes; and (ii) to

generate learning on practices, innovations and models that can be shared throughout the region.

The specific objectives of the evaluation were: (i) to document results in terms of changes in children’s

inclusion in school and reduction of equity gaps; (ii) to assess how system changes led to impact results;

and (iii) to document the contribution of UNICEF to these system changes and assess how results

achieved at country level have contributed to UNICEF regional Theory of Change.

The evaluation covered the period 2002-2012 in five countries which had reported significant results in

terms of increased enrolment and retention of Roma, children with disabilities and girls: Armenia;

Kosovo* , the former Yugoslav Republic of Macedonia, Serbia and Turkey.

The evaluation methodology included a desk review covering both literature and statistical data at

national level and five country visits where key informant interviews, focus group discussions, self-

assessments by national education officials and UNICEF staff and classroom observations took place.

The independent evaluation was conducted in 2013 and 2014 by the consulting firm PROMAN S.A.

Equity focus

High national enrolment rates mask significant disparities across groups and sub-national levels.

Children with disabilities, children from linguistic and ethnic minorities, working children, children

from migrant and undocumented families, children living in remote rural areas, children from the

poorest social and economic backgrounds make up a significant part of the out of school and

underachieving population. In some countries gender stereotypes combined with age characteristics

lead to boys being more excluded, and in others girls.

Major equity gaps in both access to and outcomes of education exist between these children and their

peers. The evaluation focused particularly on the reduction of equity gaps over a decade in the five

countries under consideration in terms of the enabling environment, supply of and access to education,

education quality and learning achievements and equality in learning opportunities.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xi


Key Evaluation Findings

Changes in the Lives of Children and Equity (Impact)

Adjusted Net Enrolment rates – primary education (UIS)

Access to Education. As indicated in the graph above adjusted net enrolment rates in primary education

have increased or stabilised in the five countries over the period. Rates increased from 87.5% in 2002

to 93% in 2007 in Armenia and from 91.5% in 2002 to 98.2% in 2010 in the former Yugoslav Republic

of Macedonia. Where the rate is decreasing (Serbia), this can generally be attributed to better data

collection mechanisms.

All visited countries and territories show a Gender Parity Index (GPI) of nearing or even exceeding 1.0

which is a strong indication of girls and boys having equal access to schooling (net intake rate). Trends

over time suggest that gender parity with respect to primary education enrolment has improved over the

past 10 years in Kosovo* and in Turkey, where the Gender Parity Index (GPI) for primary education

increased from 0.92 to 0.99 between 2002 and 2011. Aggregated data masks disparities within groups

however, for example amongst girls whose mother tongue differs from the official language of

instruction. Over the evaluation period, both girls and boys have however increasingly gained equal

opportunities to progress from primary through to lower secondary. In Turkey, the GPI increased from

0.76 in 2002 to 0.92 in 2011 in secondary education.

In Kosovo*, the share of ethnic minority students increased from 2.2% to 4.2% between 2004 and 2012

in primary education. Roma children access primary education but start dropping out in lower secondary

in all visited countries. Significant progress in lower secondary attendance for both boys and girls is

observed, however, for example in Serbia where the attendance rate of the 11-15 year old Roma children

has increased from 74% in 2005 to 89% in 2010.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xii


Roma children vs. all children attendance rate by Age, Serbia (MICS)

Data on children with special educational needs and disability remains scarce and not comparable across

countries. There has been an increase, however, in the number of children with special educational

needs being deinstitutionalised to attend mainstream schools (whether in special or mainstream classes)

in Armenia, Serbia, Turkey and Kosovo*. In Turkey, the number of children with special educational

needs enrolled in regular schools more than doubled between 2009 and 2012 (from 90,000 to 190,000).

The proportion of children with disabilities enrolled in special schools has also decreased over the

period, such as in Armenia where the number of children enrolled in special schools was almost halved

between 2007 and 2012.

Learning. Pupil-Teacher ratios have reduced in the 5 countries, partially due to population decline. PISA

results for Serbia and Turkey show improved learning outcomes for both boys and girls between 2003

and 2012, with girls however performing better than boys in reading and sciences.

System level changes (Outcomes/Effectiveness)

Key system changes have occurred during the evaluation period which have contributed to increased

access and learning for children in the region. This section presents evaluation findings with respect to

these system changes as well as remaining barriers and bottlenecks.

Enabling environment

Social norms. Societal mind set change is a condition to the realisation of education inclusion. Societies

have been making steps towards accepting inclusive education and traditionally excluded groups have

become more empowered in referring to legislation to secure their right to education. Resistance

towards greater inclusion remains, however, both with regard to children with disabilities and minorities

such as Roma, on the part of education officials, teachers and parents. Fears that inclusion might

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xiii


negatively impact the rights of the majority and beliefs that children are safer in single-ethnicity or

special schools are persistent.

The concept of inclusive education inclusion represents a paradigm shift which requires changes in both

mind sets and approaches within families, schools, communities and societies and more or different

types of work on changing societal norms and social and cultural practices around demand for education

and learning.

Legislation, policy and budget. Legislation and policies have increasingly secured and enforced equal

access for boys and girls regardless of ethnicity, disability and socio-economic background in all

countries. Anti-discrimination legislation and inclusion policies have been strengthened throughout the

region since 2002.

Education legislative frameworks have become more inclusive over the period, making provisions for

an individual approach to support inclusion (former Yugoslav Republic of Macedonia, Serbia), anti-

discrimination in education (former Yugoslav Republic of Macedonia, Kosovo*), drop-out prevention

and response (Kosovo*, Serbia, Turkey), deinstitutionalisation (Serbia, Armenia), and violence

prevention (Serbia, Kosovo*). Legislation has also been passed on financial support for children with

disabilities (Kosovo*) and on social protection floors (Armenia).

National policies and strategies have been developed on the education of children with special needs

(Kosovo*), deinstitutionalisation of children in special schools (Serbia, Armenia), integration of Roma

children in education (Kosovo*, Serbia, former Yugoslav Republic of Macedonia), drop-out prevention

(Kosovo*) and catch-up programmes for drop-out children (Turkey).

School enrolment has been facilitated by the removal of administrative hurdles (particularly around

birth certificates and school documentation for returnees). The extension of compulsory education,

often through the introduction of a pre-primary grade or the extension of basic education has contributed

to increased school attendance and completion, particularly for marginalised children.

Child friendly school components have been integrated in education policies across countries, often

through the development or revision of curricula and curriculum frameworks (Kosovo*), of education

quality standards (Turkey, Serbia) and of teacher competencies (former Yugoslav Republic of

Macedonia). Early learning development standards for preschool children have been developed in the

majority of the visited countries to strengthen school preparedness and the link between preschool and

primary education (Armenia, Kosovo*, former Yugoslav Republic of Macedonia).

Although education budgets have increasingly provided for inclusion, education financing does not yet

fully support inclusive education, with few equity funding mechanisms to target the most deprived

schools and inadequate recognition of children with special educational needs in per capita funding

allocations.

Management & coordination mechanisms. The progressive decentralisation of education systems over

the period has resulted in increased responsibilities being transferred to local administrations. In this

context, the role of national agencies including teacher training institutions, inspectorates and

pedagogical institutes or equivalent bodies for inclusion has been redefined and the functions of

monitoring and supervision revised - although overlaps and gaps remain in some instances.

School leadership and management has become more standardised with regard to inclusion, with new

approaches to school self-evaluation (former Yugoslav Republic of Macedonia) and comprehensive

school development planning which address inclusion holistically (former Yugoslav Republic of

Macedonia) and cover drop-out (Serbia). However, local and decentralised authorities and schools do

not all have, the necessary support to reach out to the most vulnerable children and to expand and sustain

inclusive practices, monitoring of at-risk children and equitable approaches to access and learning.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xiv


Education inclusion is hampered by weak education planning systems that are not sufficiently evidence-

based at national and local levels. Although education data collection systems have improved over the

period, data rarely informs transparent decision-making for equity.

Weak cross-sector and vertical and horizontal collaboration also negatively impacts on sound education

planning for the inclusion of all children in quality learning, which requires a holistic and integrated

approach.

Policy implementation monitoring remains weak across all countries and there are no effective quality

assurance systems to monitor education inclusion. Promising steps have, however, been taken in some

countries. In Serbia, instruments for monitoring inclusion in preschool and primary education are being

developed under a comprehensive monitoring framework for inclusive education from classroom to

national level. Drop-out monitoring has also been included in the main responsibilities of the Serbian

National Education Council. In Turkey, the minimum standards for primary education are being used

as benchmarks in all schools, providing a unique opportunity for monitoring inclusiveness and for

evidence-based planning. The development of a database for identifying and monitoring out-of-school

children has also led to concrete measures to support timely enrolment in school and re-integration into

schools.

Supply

There is evidence of increased education provision and of availability of learning and teaching

materials, including in minority languages across countries included in the evaluation. Facilities are

also increasingly becoming accessible for children with disabilities. There is little teacher shortage in

primary education but teacher deployment can hamper equity.

Demand

Indirect education costs remain a significant barrier to attendance and retention in the region for

marginalised groups and children from poor families. Countries have addressed demand-side

constraints in terms of financial access by implementing Cash Transfer programmes (Turkey), free

textbook schemes (Kosovo*), and of social and cultural practices by raising awareness on the

importance of education (Girls’ Education Campaign in Turkey, work with Roma communities, parents

and mediators in Serbia, Kosovo* and the former Yugoslav Republic of Macedonia).

Quality

Curriculum reviews have been conducted and curriculum frameworks developed (Kosovo*). The

introduction of competencies, through life skills education for instance in Armenia and the former

Yugoslav Republic of Macedonia, have contributed to a better balance between knowledge, skills and

attitudes for children.

While teachers have been increasingly trained on inclusive education principles and child-centred

teaching and learning methodologies, particularly through in-service training, practical implementation

varies within countries depending on school leadership and teachers’ personalities and abilities.

Initiatives, such as the introduction of teaching/pedagogical assistants, Individual Education Plans,

catch-up classes and school-community mediators have been critical in several countries across the

period in moving forward the inclusion agenda in schools, particularly for children with special

educational needs, drop-out students or students in and out of school and children from ethnic

minorities.

However, weak competencies of teachers remain a common challenge across countries with regard to

quality education for all children. Primary data collected during the evaluation shows that only 25% of

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xv


teachers’ initiated interaction with students can be regarded as conducive for learning. Classrooms

remain teacher-centred and children have few opportunities actively to participate in the teaching and

learning process. While girls do not seem excluded from the learning process, inclusion of students with

special educational needs and Roma children rely too often on teachers’ individual will and dedication.

UNICEF’s contribution (Efficiency and Effectiveness)

The evaluation period has seen a change in how UNICEF operates, moving from pilot projects to a

system approach focusing on system changes for greater results and sustainability.

UNICEF’s investments in education in the region have been relatively low when compared to those of

other development partners such as the World Bank or the European Union. Nevertheless the evaluation

found that UNICEF has made significant contributions to the above mentioned system changes for

education inclusion, particularly in influencing the enabling environment for education inclusion and

in improving the quality of education for all children. UNICEF has played a catalytic role, efficiently

combining its core roles at regional and national levels.

Policy advice and technical assistance. UNICEF was constrained in its work in the early 2000s by the

lack of supportive legislation with regard to ethnic minorities and marginalised groups. UNICEF’s

efforts to create an enabling environment for the inclusion of all children in school also stemmed from

this initial legislative gap which has taken some time to fill.

The focus on the enabling environment aimed to protect the rights of marginalised children but also to

enable rights holders to claim their rights based on newly established legislation. UNICEF’s role in the

improvement of the legislative and normative framework on inclusive education and anti-discrimination

as well as in the integration of child-friendly school principles in education strategies and education

sector plans in the region has therefore been critical.

UNICEF has established with national stakeholders a relationship based on mutual trust and

professional reliability which has resulted in policy advice being generally reflected in policy changes.

UNICEF's support of the development of laws, by-laws, strategies and action plans, contributed to

ensuring that key issues of inclusion became reflected in the legislative foundations of the visited

countries and territories.

UNICEF, by actively participating in donor coordination in education or in the area of child rights,

contributed to efforts to avoid fragmentation of partners’ contributions with positive results for inclusive

education (Serbia).

UNICEF has provided coherent technical assistance to secure policy changes while also proposing

mechanisms for implementation at local and school levels. This has been the case with child-friendly

school standards (former Yugoslav Republic of Macedonia) and drop-out response (Turkey, Kosovo*).

UNICEF has also been instrumental in designing, promoting and institutionalising training modules

and programmes on active learning, child-centred approaches and inclusive education strategies

(Kosovo*, Armenia, Serbia, former Yugoslav Republic of Macedonia). While many teachers have been

trained and some are applying child-centred active learning methods, classroom practices remain mostly

traditional and teacher-centred. More knowledge, know-how and readiness from teachers, directors and

officials is needed to tackle equity in learning outcomes and participation and achievement of all

children.

Over the period UNICEF has strengthened its inputs in education financing, particularly around the

development of a formula that would reflect better the needs of each individual child (former Yugoslav

Republic of Macedonia).

UNICEF interventions have also contributed to capacity development of local expertise in inclusive

education (Kosovo*, Armenia).

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xvi


Modelling and piloting. Modelling is, next to policy advice and technical assistance, the second best

funded core role for the UNICEF offices participating in this evaluation. A pattern identified in all

visited countries shows how UNICEF has linked innovative pilots to policy development and then to

broader and deeper changes and shifts in education systems and legislation.

The child-friendly schools ethos has been fully integrated into the policy debate and sector plans,

gradually entering school practices. Modelling of the Child-friendly school approach was particularly

successful when coupled with sound capacity building at school level (former Yugoslav Republic of

Macedonia, Armenia). The approach also enabled schools and teacher training institutions to embrace

issues around active learning, multiculturalism, inter-ethnic relations and the reduction of school-based

violence.

Other UNICEF supported models aiming to reduce education inequities that have been successfully

scaled-up include the development of an absenteeism management system for the monitoring of

children at risk of dropping out in Turkey and the institutionalisation of Pedagogical Assistants in

Serbia, co-piloted by UNICEF. Pedagogical Assistants support students, families and teachers and

provide a link between the home, the community and the school.

Considering the rather small budgets allocated to inclusive and quality education pilot projects

implemented by UNICEF, modelling is a very efficient core role of the organisation in the CEE/CIS

region.

Partnerships and leverage of resources. The five visited countries are middle-income countries, with

many active partners and agencies. This made it difficult to identify causal linkages between UNICEF’s

efforts and the governments’ political will and financial resources to address child rights violations and

equity issues in education.

There is evidence, however, of UNICEF’ success in leveraging funds and partner expertise for

education inclusion in most countries. UNICEF models and pilots have often been taken up by other

partners. In Turkey, UNICEF’s work in child protection and education was strongly supported by the

European Union’s Children First programme. In Armenia a World Bank loan is addressing school

preparedness through a half-day provision scheme originally piloted by UNICEF. In the former

Yugoslav Republic of Macedonia, USAID is supporting the expansion of extracurricular activities

based on a model developed by UNICEF. In Serbia the Ministry of Education leveraged funds from the

World Bank to expand the work initiated by UNICEF on the development and implementation of the

Schools without Violence standards.

UNICEF has been a partner of choice for the World Bank which has sought to complement

infrastructure and rehabilitation investments with technical expertise in capacity development such as

training. In Armenia, UNICEF is training pre-primary teachers working in institutions supported by the

World Bank.

Regionally, UNICEF has engaged in a fruitful partnership with the UNESCO Institute for Statistics on

the Out-of-School Children Initiative, strengthening and harmonising data collection processes for out-

of-school children across the region.

Nationally, UNICEF has developed strong partnerships with civil society organisations on key inclusion

issues, particularly out-of-school children, anti-discrimination and children with disabilities (Kosovo*,

Serbia). UNICEF has also taken further steps to engage with the private sector. In Turkey, a partnership

with twenty private media outlets was established to support the campaign on violence-free schools.

Monitoring & Evaluation. Independent child rights monitoring to strengthening accountability of duty-

bearers with respect to CRC commitments is core work of UNICEF and is operationalized through

support for a range of data and research related activities on children.

UNICEF has been consistently advocating for the development of quality assurance systems and

monitoring frameworks for education throughout the period. To this end, UNICEF’s interventions have

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xvii


aimed both to generate baseline data (on inclusion, on child-friendly school, on out-of-school children)

and to monitor progress according to relevant indicators. Although Child-friendly schools and other

education quality standards have been developed, national capacities to manage and monitor the

implementation of complex and cross strategies relating to education inclusion are lacking. There has

also been little focus on monitoring learning outcomes at country level: in the former Yugoslav Republic

of Macedonia UNICEF has nonetheless worked on literacy and numeracy assessments.

UNICEF has been supporting the strengthening of education data collection processes, working

alongside Education Management Information System units (Kosovo*, Armenia) and National

Statistical Offices. In Turkey, UNICEF has been playing a catalytic role in supporting the Government

to strengthen its quality assurance systems and in brokering partnerships to reinforce independent

monitoring of results for all children.

UNICEF’s support to Multi-Indicator Cluster Surveys (MICS) has also contributed to fill the data gap

on education for Roma children for instance (Serbia and the former Yugoslav Republic of Macedonia)

and on school readiness while developing capacities of national stakeholders on data collection and

analysis.

Independent Voice’ for children and adolescents. UNICEF has been successful in influencing the

thinking and decision-making of ministry officials and development partners on education inclusion.

Child-friendly schools principles are known, accepted and validated by stakeholders across countries.

Information generated by studies on education exclusion has increased the awareness of policy makers

on recognition of different types of marginalisation and different education obstacles faced by children

and their families and has positively impacted on policy decisions. In Turkey UNICEF played a key

role in advocating for girls’ education, out-of-school children, child labour and children engaged in

seasonal work, resulting in two large-scale programmes to address these issues. In Armenia and

Kosovo*, UNICEF brought children with disabilities and special needs education to the centre of

education debates. In Serbia, UNICEF raised awareness about drop-out in primary and secondary

education among education ministry officials but also officials from other ministries and civil society

organisations.

Over the evaluation period UNICEF has focused little on addressing demand-side bottlenecks related

to social norms: this is still mostly done by civil society organisations. It has also put an overall focus

on access in the first period covered by the evaluation. Equity in learning is a more ambitious and more

challenging agenda, requiring drastic system changes in teachers’ and directors’ recruitment and

management, in evaluation of learning outcomes and in classroom and school practices. The increased

corporate focus on equity has led to concrete steps being taken by the Regional Office and by some

countries to support national debates on equity in learning. This was done through the regional analysis

of PISA results (Regional Office) and the generation of new evidence on learning outcomes (former

Yugoslav Republic of Macedonia).

Knowledge exchange. UNICEF has played an important role as knowledge broker at regional, national

and local levels. Regional publications are valuable to stakeholders and UNICEF CO education staff as

they also give an international perspective on region-specific achievements and challenges. The

Regional Office has also contributed to capacity development of national officials through the

organisation of study tours focusing on quality education and school effectiveness – these were timely,

well received and contributed to the promotion of child friendly school principles in policy documents

and school practices.

Strategies, education sector plans, and normative frameworks have been developed with UNICEF

support, reflecting child friendly school principles and inclusive education standards and indicators,

thus providing a base for information-based planning, monitoring inclusiveness of schools and quality

assurance of education. Particularly efficient in influencing Ministerial decision-making have been the

independent studies generating knowledge and providing evidence to policy makers on the importance

of inclusive education and modalities of investments. Further, advocacy and research contributed to

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xviii


fresh policy commitments in critical areas like measuring child well-being, addressing developmental

delays in early childhood, and deinstitutionalisation of children in care through foster-parenting, and

child justice.

Research studies on issues pertaining to equity in education access, learning and governance have been

instrumental in influencing decision-making at national level in all countries, particularly when

addressing issues of investment modalities. UNICEF has also taken steps to strengthen national

networks on specific issues. In Serbia UNICEF is supporting the Network for Support to Inclusive

Education established to foster horizontal cooperation and information exchange over 120 practitioners

and experts. Expansion and “mainstreaming” of inclusive education has however proven difficult in the

evaluated countries. More technical assistance from the regional office is required on this as well as on

how to build partnerships for leveraging influence and funding at the country level.

In the context of decentralisation, UNICEF has increased its knowledge sharing activities across local

governments both on technical and financial issues including local budgeting for children, inter-sectoral

cooperation for inclusive education and out-of-school children and child participation (Kosovo*, former

Yugoslav Republic of Macedonia).

Facilitating national dialogue. UNICEF has had considerable success in making governments’ reforms

and partners’ priorities more relevant to the realisation of the right of all children to be included in

quality learning. UNICEF has constantly advocated for the most disadvantaged groups, whether

children with disabilities, minority children, marginalised boys and girls and out-of-school children

resulting in concrete legislative and policy changes.

UNICEF has also increased collaboration for children’s rights. In Serbia, UNICEF has supported social

accountability for children's rights by strengthening cooperation between the Ombudsman, the

Parliamentary Committee for the Rights of the Child and the National Council on Child Rights.

UNICEF has been efficient in conducting evidence-based advocacy work at the highest level

(Parliaments, politicians, high-level policy-making bodies) but also with the general public and children

themselves. This has been done through campaigns for the enrolment of excluded children such as

Roma children in Serbia, children with disabilities in Armenia or girls in Turkey.

UNICEF has reflected children’s voices in strategy and policy development, not least by promoting

child-friendly school and inclusion principles.

UNICEF’s work has significantly contributed to system changes in countries. Challenges to the

effectiveness of UNICEF’s work include weak national capacities for planning and management, weak

capacities at decentralised levels, lack of multi-sector cooperation and lack of integration of education

systems.

Relevance

The evaluation period has seen a change in how UNICEF operates, moving from projects and outputs

to a system approach focusing on system changes for greater results and sustainability.

UNICEF’s work in education inclusion is in line with key human rights instruments such as the

Convention on the Rights of the Child and on the Rights of Persons with Disabilities but also the

Salamanca Statement on inclusive schools.

Through its work, UNICEF follows up in countries the recommendations of the Committee on the

Rights of the Child and adopts a rights-based approach to programming, targeting those children whose

right to education is violated and supporting countries establishing effective child rights monitoring

mechanisms.

UNICEF was constrained in its work in the early 2000s by the lack of supportive legislation with regard

to ethnic minorities and marginalised groups. UNICEF’s efforts to create an enabling environment for

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xix


the inclusion of all children in school also stem from this initial legislative gap which has taken some

time to fill. The focus on the enabling environment aimed to protect the rights of marginalised children

but also to enable rights holders to claim their rights based on newly established legislation.

UNICEF has put an overall focus on access but less so on learning in the first period covered by the

evaluation. Equity in learning is a more ambitious and more challenging agenda, requiring drastic

system changes in teachers’ and directors’ recruitment and management, in evaluation of learning

outcomes and in classroom and school practices. Steps have been taken however by the Regional Office

through the regional analysis of PISA results and by some countries (former Yugoslav Republic of

Macedonia).

Sustainability

All evaluation countries have seen an increase of their Gross National Income between 2002 and 2012

and the trend of total public expenditure on education as a percentage of GDP (for all levels of

education) is generally increasing over the period in the 5 countries. This is an enabler in terms of

guaranteeing the sustainability of the education outcomes achieved during the decade. Armenia,

however, was more seriously hit by the economic crisis than the other countries. This could potentially

have a negative impact on the education sector in the long term.

In middle-income countries there are increased opportunities of partnerships with the private sector as

corporate social responsibility is expanding. This is an opportunity not only for UNICEF but also for

governments and schools to capitalise on and expand good practices for inclusion.

Four main factors hindering sustainability have been identified by the evaluation.

Firstly, the capacities of officials sometimes at national but particularly at local level are weak,

including around monitoring and evaluation functions.

Secondly, cultural and societal attitudes and mind sets continue to be a barrier to the inclusion

and the performance of traditionally marginalised children in the education system.

Thirdly, local and decentralised authorities and schools do not all have the necessary support

to reach out to the most vulnerable children and to expand and sustain inclusive practices,

monitoring of at-risk children and equitable approaches to access and learning.

Lastly, the inter-sectoral coordination and cooperation required in policy development and

implementation for all children to be included in and learn through education remains a

challenge.

Conclusions

Trends over time suggest that, over the past decade, primary education enrolment rates have increased

or stabilized in the five countries, gender parity in primary education has improved in Kosovo* and in

Turkey, and both girls and boys have increasingly gained equal opportunities to progress from primary

through to lower secondary. However, while Roma children access primary education, they start

dropping out in lower secondary in all visited countries. Nevertheless, significant progress in lower

secondary attendance is observed for both Roma boys and girls, for example in Serbia where the

attendance rate of the 11-15 year old children has increased. There has been an increase in the number

of children with special educational needs being deinstitutionalized to attend mainstream schools

(whether in special or mainstream classes) in Armenia, Serbia, Turkey and Kosovo*, while in Turkey,

the number of children with special educational needs enrolled in regular schools doubled between 2009

and 2012.

UNICEF’s work has significantly contributed to system changes in countries. Challenges to the

effectiveness of UNICEF’s work include weak national capacities for planning and management, weak

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xx


capacities at decentralised levels, lack of multi-sector cooperation and lack of integration of education

systems.

While policy planning processes have become more evidence and results-based, the decentralisation of

education functions and the critical role of local education officers and school directors require a holistic

capacity development approach for all schools to be able adequately to engage with equity access and

equity learning.

Education sector planning has become increasingly challenging as more ministries are becoming

engaged in the process. Harmonisation of policies and plans across sectors has also become an issue in

countries where sectors are still thinking in silos. Addressing poverty related barriers to education and

the inclusion of children facing complex and overlapping disadvantages requires a holistic approach to

education and a genuine collaboration between ministries, organisations and agencies.

For the environment to be truly enabling, societal norms and standards regarding diversity, inclusion

and non-discrimination need to support policy development and implementation. By increasingly

making their voices heard, minority group representatives have contributed to changing mind sets.

Progress is, however, slow and there is a need for more or different type of work on changing societal

norms and social and cultural practices around demand for education and learning.

The focus of both policies and practices has mostly been on enrolment and retention of marginalised

groups. While many teachers have been trained and some are applying child-centred active learning

methods, classroom practices remain mostly traditional and teacher-centred. More knowledge, know-

how and readiness from teachers, directors and officials is needed to enhance quality and tackle equity

in learning outcomes and participation and achievement of all children.

Expansion and “mainstreaming” of inclusive education has proven difficult in the evaluated countries.

More technical assistance from the regional office is required on this as well as on how to build

partnerships for leveraging influence and funding at the country level.

Main recommendations

General recommendations

Rec 1: Intensify the realisation of equitable participation and learning i.e. move from quantitative to

qualitative inclusion.

Rec 2: Focus on operationalising “transformation” in a systemic sense, in order to achieve systemic

changes.

Recommendations for the Regional Office:

Rec 3: Focus on knowledge generation and exchange across the region.

Rec 4: Generate discussion around the concept of inclusion and its implications for education systems

at all levels.

Rec 5: Advocate the cross-sectoral dimension of inclusion to build new partnerships and leverage funds.

Rec 6: Facilitate the identification of quality standards for inclusive education which rather than

reinforcing exclusion cover social norms, the active involvement of all learners in learning and the

benefits of diversity for all learners.

Recommendations for the Country Office:

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas xxi


Rec 7: Support national education institutions to make inclusive policies a reality on the ground through

technical assistance placed directly in institutions.

Rec 8: Raise awareness of all stakeholders about the broad meaning of inclusion, going beyond children

with special educational to encompass all excluded children.

Rec 9: Strengthen M&E systems to monitor the qualitative aspects of inclusion particularly around

perceptions and mindsets and classroom interactions.

Rec 10: Create linkages in the monitoring architecture enabling to draw conclusions on the impact of

social issues on education, such as ethnicity and poverty.

Rec 11: Strengthen inter-sectoral coordination and the links between inclusive education and social

policy development.

Rec 12: Support the paradigm shift required by a truly inclusive education system, particularly in terms

of organisation of teaching and learning in school and the holistic and cross-sectoral support required

by inclusion.

Rec 11: Strengthen the education continuum from preschool to primary to secondary education to

ensure continuity in cognitive and emotional development and to trigger increased enrolment and

attendance in education.

Final Evaluation Report: Multi-Country Evaluation of Regional Knowledge and Leadership Areas 1


1 INTRODUCTION

UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States

(CEE/CIS) has contracted PROMAN S.A. within the framework of the Multi-Country Evaluation of

Regional Knowledge and Leadership Areas. Under five strategic result areas, evaluations shall

demonstrate how UNICEF’s contributed to system changes, which themselves, in turn, had an impact

on the life of children. This evaluation responds to the fourth key result area, i.e. Inclusion of all Out of

School Children in Quality Learning.

1.1 Background

In line with UNICEF’s general policy shift from implementing projects to a combination of sub-national

work and policy work at the national level and in sector-wide groups, the CEE/CIS region has over the

past 10 years also adopted a systemic approach to programming. As a consequence, programming

evolved from small interventions to influential integrated programmes, aiming at concrete changes at

institutional, societal and individual level. In continuation, and on the basis of systemic changes,

UNICEF’s work targets specific changes on the life of children at the impact level. Such changes could

include, inter alia, preparedness for secondary education, improved employment opportunities, or

stronger participation/inclusion in society.

Under the five strategic result areas, the Regional Knowledge and Leadership Agenda (RKLA) has a

dual purpose, i.e. (i) informing and guiding UNICEF’s future work in CEE/CIS, and (ii) generating

evidence for documenting how UNICEF contributed over the past decade to outcome and impact results

for children. Strategically, it will be assessed how equity gaps are being reduced, and how indicators –

as reflected by indicators related to the most disadvantaged children – are closing up with national

averages.

Despite average primary school enrolment rates in the CEE/CIS region of above 90-95%, there exist

strong sub-national disparities with significant numbers of children from specific groups receiving no

basic education at all. According to UNESCO’s Global Monitoring Report (GMR) 2010, this refers to

3.7 million children of primary and lower-secondary school age, and to 1.6 million children of pre-

primary school age, in addition to a further 12 million of adolescents being estimated to be out of school.

Marginalisation is characterised by social, cultural and economic conditions, and refers to issues of both

access and educational outcomes, i.e. educational quality impacts. Children most likely to be out of

school comprise a broad variety of profiles, depending on the country and territory and on the level of

schooling:

children from ethnic and linguistic minorities, including Roma4;

children with disabilities;

gender-related disadvantages, affecting both boys and girls;

children from the poorest households;

children who are working;

children performing below academic standards.

4 Apart from Roma there are other ethnic minorities that are excluded both in the countries object of this evaluation

and in those that are not included but who will still be able to draw lessons from this MCE; for example, in Kosovo

(UNSCR 1244) minorities include Roma, Ashkali, Egyptian (RAE), in the former Yugoslav Republic of

Macedonia there are Albanian, Turkish, Vlach, Serbian, Roma and Bosniac minorities.



Children of pre-primary age, as well as adolescents, from all of the groups above are particularly

affected; also, combinations of disadvantages are rather common and aggravate the degree of

marginalisation.

Results, visions and goals of UNICEF’s Education Strategy and Medium Term Strategic Plan (MTSP)

are closely interrelated regarding UNICEF’s approach towards inclusion in CEE/CIS for the past 10

years. Such approach is based on the 2007 Regional Education Strategy Note5 which sets the vision

that “every child in the region will access and complete basic education of good quality”. This in turn

results in 4 long-term (2015) education goals and operational strategies for the CEE/CIS region, within

the framework of UNICEF MTSP:

Education disparities and exclusion will be reduced, and the last 10-15% of children out of

school in the region will be included in formal quality education systems.

The quality and relevance of basic education will be improved in order to reduce school drop-

out, and to increase completion and achievement rates.

All children will enrol in formal primary school education by the age of six years and all

children ages 3 to 6 will participate in at least one year of free pre-primary education.

All countries and territories will create education systems designed to build social cohesion and

tolerance to reduce tension and prevent conflict, especially in areas of “frozen conflict” and

ongoing ethnic tensions, and will integrate education in emergency preparation and response to

restore schooling to affected populations in all emergency (including natural and manmade

disasters) and post-crisis situations.

1.2 Purpose and objectives of the evaluation

The purpose of the evaluation is (i) to assess the extent to which UNICEF’s contributions to programme

interventions addressed major child rights violations in education, reduced equity gaps and led to impact

results for children – in terms of reduction in the number of children out of school and improved quality

of education/learning outcomes; and (ii) to generate learning on practices, innovations and models

across a critical mass of countries addressing similar issues, to be used during various UNICEF country

programming processes.

According to the ToR the objective is therefore

(i) to document and report on impact results in terms of changes in children’s life;

(ii) to assess and demonstrate how such results were made possible through system changes; and

(iii) to document the contribution of UNICEF to these system changes.

In addition, the evaluation will ensure that mechanisms for the assessment of impact results embrace

issues of

(i) contextual adequacy (regarding tailoring programming approaches to the qualitative differences

between inequities);

(ii) coordination (with other stakeholders);

(iii) coherence (across policies and other supporting interventions); and

5 Listed in Appendix 3 as document # D026. This document provides a regional orientation and guidance to

Country Offices and thus, makes the link between the Regional Theory of Change and interventions at country

level to bring about system changes.



(iv) protection (in terms of adequacy of the response of the respective education systems to

protecting vulnerable groups, including Human Rights and gender issues).

Ultimately, the evaluation will make a contribution towards improving future programming by adapting

it to the specific groups of children out of school.

1.3 Underlying Theory of Change in relation to the evaluation object

The reduction of equity gaps and impact results achieved through changes in the underlying education

systems (at national, regional or local level/s) is based on a regional Theory of Change (ToC), based on

the mainstreaming of the UNICEF Child-friendly Schools (CfS6) framework. While the CfS concept

served as a global guide for UNICEF’s work since 1995, it was only in 2001 when a complex and

comprehensive CfS approach was conceptualised, both at national and community levels.

Considering UNICEF’s work with countries over the past decade, CfS models have emerged as a

holistic initiative providing a comprehensive range of quality interventions in education. Based on a

successful implementation of CfS programmes in over sixty countries world-wide, and in many

CEE/CIS countries, a Regional Analysis of Education in CEE/CIS was undertaken in 2005. The

subsequent CfS Study Tour to Thailand, organised by the CEE/CIS Regional Office, was a key

milestone towards establishing a more systemic approach, which was in sharp contrast to the start

situation in the early 1990s when the issues of child rights and reduction of equity gaps were addressed

through isolated project-type activities. This was then followed by the Regional Education Strategy

Note (2007) and its four long-term education goals and operational strategies for the CEE/CIS region,

within the framework of UNICEF MTSP (cf Section 1.1).

The ToC reflects UNICEF’s historical and programmatic shift in approach from project based work to

focussing on systemic changes, thus bringing together the Regional ToC with the MoRES (Monitoring

of Results for Equity Systems) framework for effective planning, programming, implementation,

monitoring and managing for results to achieve desired outcomes for the most disadvantaged children.

The ToC expects a realisation of child rights and the reduction of equity gaps through changes in

systems at national and sub-national levels.

Therefore, the ToC is directly related to the evaluation object, in order to assess to which degree

upstream work of UNICEF led to impact results (changes in the life of children) and outcome results

(system changes). This is particularly reflected in the Regional Education Strategy with its clear goals

related to inclusiveness and quality; (ii) its paradigm shift from project to system support; (iii) and the

subsequent translation of the regional strategy into technical guidance through regional position papers,

studies and evaluations.

1.3.1 Regional Theory of Change and reflexions at country levels

The Regional ToC refers to “the reduction of equity gaps and impact results achieved through changes

in the national (regional/local) systems, partly due to the contribution of UNICEF” (ToR, p.5), and as

such to the progressive realisation of child rights and equity.

6 The abbreviation CfS is preferred here over the more common abbreviation CFS (all with capital letters). The

reason is that the concept of child-friendliness is considered to be a close unit also in linguistic terms, thus not to

be divided by two capital letters of rather equal value. Whereby CFS basically stands for Child [and] Friendly

[and] Schools, the abbreviation CfS rather stands for Child-friendly [and] Schools, highlighting the concept of a

school being child-friendly, and clearly expressing the comprehensiveness, togetherness and uniqueness of the

complete expression “CfS”.



The Regional ToC has guided the assessment of country approaches towards system changes, in

particular regarding (i) the analysis of core roles against country specific processes of education

legislation, financing, policy- and decision-making, and (ii) the analysis of core roles against school-

level application of system changes.

The desired sustainable systemic impact is thus closely related to the removal of system bottlenecks to

inclusive education at country level. This evaluation assessed the degree to which UNICEF’s core roles

i.e.,

policy advice and technical assistance;

modelling;

facilitating national dialogue towards child-friendly social norms;

enabling knowledge exchange,

monitoring & evaluation;

leveraging of resources from the public and private sector;

giving children and adolescents ‘a voice’,

contributed to (i) the improvement of education legislation, policy- and decision making; (ii) the effect

on creating a more enabling environment for education systems to become more inclusive (i.e. the

removal of specific bottlenecks), and (iii) the impact of the application of system changes at school

level.

The analysis applied in this evaluation has further been based on a reflexion of “country pathways”7 in

influencing system changes against the overarching MoRES determinants. Country visits provided an

opportunity for an in-depth validation of the relevancy, effectiveness, efficiency and sustainability of

country-specific strategies with regard to the achievement of concrete impact at the various system

levels.

While analysing and assessing country-specific reflexions of the ToC, the centre of attention has been

on the processes underlying such reflexions, and on how well processes have worked at country level.

1.3.2 Mainstreaming the Child-friendly School approach

Concepts underlying the CfS approach reflect a quality environment and act holistically in the interest

of the whole child, which includes his or her health, nutrition and overall well-being. Similarly, child-

friendly schools are concerned with the child’s contextual situation outside school, and thus reach out

to families and communities.

The Convention of the Rights of the Child (CRC) sets the parameters for educational quality in terms

of “child-friendliness”– not only as far as the “education articles” Articles 28 and 29 are concerned, but

also regarding Articles 12-14 in terms of promoting the child’s right to freedom of expression. This also

needs to be seen within the larger framework of the EFA Dakar Goals and the Millennium Development

Goals (MDGs) “Achieve Universal Primary Education” (MDG 2), “Promote Gender Equality and

Empower Women“ (MDG 3), and “Develop a Global Partnership for Development” (MDG 8).

7 The notion of “country pathways“ refers to background Information formulated by the countries having

participated in this multi-country evaluation and which was attached in annex to the TORs. These “country

pathways“ provided initial details on how system changes, to which UNICEF made a contribution together with

its partners, aimed at achieving a greater inclusion of out of school children in quality learning. These were

continuously reflected upon during the course of the in-country consultations.



The conceptual framework of child-friendly schools8 defines a school as “child-friendly” if it is

rights-based,

child-seeking and inclusive for all children,

healthy and protective;

gender-sensitive,

effective for learning;

promoting the participation of children, families and communities, and

quality-based.

“Child-friendliness” can thus be described as a child-centred learning environment which sees and

understands the whole child in a broad context, thus making education relevant due to the consideration

of the reality of children’s lives. Such a quality environment will be (i) effective for learning and (ii)

inclusive and protective of all children, and likewise involved with families and communities.

1.3.3 A programmatic shift towards a focus on systemic change

Since national contexts and priorities differ from a world-wide perspective, the related CfS standards

attained specific regional characteristics. Also, between 2006 and 2009, the Regional Office started to

promote the shift from project-focussed approaches to an increasingly systemic approach in order to

mainstream key elements of the CfS approach into national education laws and policies. The CfS helped

the articulation of a national vision for quality and inclusive education based on child rights which were

mainstreamed in the national education laws and policies. More upstream strategies opened up

increasing opportunities for UNICEF to participate in the sector-wide education debate with a large

variety of donors including the Global Partnership for Education (GPE), formerly known as the

Education for All Fast Track Initiative (EFA-FTI)

As from 2006, the Regional office benefitted from Education Thematic Funding, and prioritised future

programming across all Country Offices on system changes. The regular and more predictable funding

aimed at strengthening the programmatic shift and developing new standards for quality education (or

revising existing ones) in line with the intensified systemic orientation.

In particular, quality of education and equity on learning outcomes became key focal areas of UNICEF’s

work in the region, which aimed at a greater inclusion of out of school children in quality learning –and

to which UNICEF seeks to make a significant contribution alongside with its partners in development.

1.3.4 Monitoring for equity: the MoRES framework

The conceptual framework of MoRES is used as an ex-post reference to UNICEF planning, with the

following central principles:

Equity Refocus: Reconfirms UNICEF’s commitment to promote the use of data and evidence

in advocacy and programming for implementation of essential services to address the critical

needs of marginalised and disadvantaged children.

Management for Results: Can only be achieved if there is real time and frequent data to act

upon. While building up on existing approaches (like results-based management), MoRES

addresses the critical need for intermediate, real time process/outcome measures between

8 CHABBOTT, C. (2004): UNICEF’s Child-friendly schools framework: A desk review. New York: UNICEF.



routine monitoring of inputs and outputs as a part of development programmes, on the one hand,

and monitoring of high level outcomes/impact every three to five years, on the other.

Bottleneck and Barrier Analysis: Underscores that there are critical conditions or determinants

which either constrain or enable the achievement of results for particular groups of children.

The four determinants are enabling environment, supply, demand and quality of services and

goods.

The MoRES approach is applicable to all country contexts. This is especially relevant where

cooperation is focussed on policy advocacy. Ten determinants of bottlenecks are grouped in four

different headings (enabling environment, supply, demand, and quality).

Code9 Determinant Description


E1 Social norms Widely followed social rules of behaviour

E2 Legislation/Policy Adequacy of laws and policies

E3 Budget/Expenditure Allocation and disbursement of required resources

E4 Management/Coordination Roles and accountability/coordination/partnerships

Supply

S1 Availability of essential commodities/inputs Essential commodities/inputs required to deliver a

service or adopt a practice

S2 Access to adequately staffed services,

facilities and information Physical access (services, facilities, information)

Demand

D1 Financial access Direct and indirect cost of services/practices

D2 Social and cultural practices and beliefs Individual/community beliefs, awareness,

behaviours, practices, attitudes

D3 Timing and continuity of use Completion/continuity in service, practice

Quality

Q1 Quality of care Adherence to required quality standards (national or

international norms)

MoRES determinants refer to four levels of planning, programming, implementation, monitoring and

analyses of results which feed into each other and are complementary:

9 Matching codes are also included in the Evaluation Matrix (Appendix 4) and the Overview of

Interview/FGD/Round Table items against indicators and MoRES Determinants (Appendix 5), in order to show

in detail the relation of evaluation questions and related indicators (and survey tools) to the MoRES framework

and the respective determinants.



Programmatic level

Level One is about effective equity focussed planning to identify bottlenecks and barriers for achieving

results (situation analysis and strategic planning)

Level Two is about monitoring implementation of UNICEF’s inputs and activities which contribute to

addressing child deprivations

Level Three is about programme assessment/monitoring, analyses and timely actions to remove local or

group specific barriers and bottlenecks, strategic adjustments to programme interventions at all levels and

informing policy dialogue

Level Four is about monitoring trends in the situation of children (validating outcomes and estimating

progress towards reducing child deprivations)

Overall, MoRES (and its analytical framework) facilitates linking UNICEF’s support to the

strengthening of policies and systems to concrete changes in the life of children.

The underlying emphasis is on

(i) the flexibility of the approach;

(ii) the economy of indicators to be monitored (regarding their ability to feed back into

programming); and

(iii) the ways and means to identify, collect and collate the relevant data.

1.3.5 Human Rights-based approach to programming

UNICEF Country Programmes in the CEE/CIS region are generally geared towards the achievement of

the Millennium Development Goals (MDGs) on improving nutrition, education, gender equality,

maternal health, reducing child mortality, HIV/AIDS and addressing child poverty. This implies

supporting the Government in meeting its obligations under the CRC and the Convention on the

Elimination of all Forms of Discrimination against Women (CEDAW).

UNICEF aims at ensuring inclusion of all children, young people and women in the provision of basic

education, health and child protection services with their increased and genuine participation. Together

with government and NGO counterparts, United Nations agencies and donors, UNICEF country

programmes for CEE/CIS seek to ensure that state policies provide a better framework for the protection

of children’s rights, with existing legislation amended to comply with CRC principles, and with

administrative frameworks and institutional development support to provide an effective

implementation of the programme. The Human Rights-based approach (HRBA) to programming is also

reflected in UNICEF’s work on budgets, C4D, modelling and other core roles aligned with the HRBA.

UNICEF regularly works with Government, international and local organisations, civil society, media,

children and young people (i) to advocate for effective policies to benefit children, (ii) to support

innovative programmes to care for and protect children, and (iii) to ensure that policy makers and

community representatives will encourage and facilitate the meaningful participation of children and

young people in their communities.



2 OVERALL FRAMEWORK OF THE EVALUATION

2.1 Scope and context of the evaluation

Based on the purpose and objectives described in Section 1.2 above, the evaluation has a clear focus on

impact. In line with the ToR, the evaluation analyses UNICEF's contributions to programme

interventions that “led to impact results for children” (ToR p.9) and well-functioning “practices,

innovations and models”(ibid.) in the concerned countries and territories which are addressing similar

issues, and which can be used by UNICEF in the future for country programming processes.

In particular, the evaluation provides evidence about the impact results “in terms of changes in children's

life” and the reduction of equity gaps on the basis of existing analyses and further data sources.

Furthermore it explains how these results emerged through changes on the systemic level and in which

way UNICEF contributed to these changes.

Accordingly the main issues of the evaluation are “the inclusion of out-of-school children, primarily in

terms of equity in access to basic education” and “quality learning – and particularly equity in learning

outcomes – as a key determinant of inclusion”. Thereby the evaluation does not undertake country level

evaluations or assess specific interventions within certain key results areas but rather applies a “multi-

country approach to validate the regional theory of change” and “document UNICEF's contribution to

similar results […] in a group of five countries addressing issues of exclusion from education”.

The prime audience for the evaluation comprises

UNICEF Senior management at Headquarters (HQ);

members of UNICEF’s Executive Board;

strategic partners and donors (including the Government of Norway being the main provider of

Education Thematic Funding).

In addition, a secondary audience will be governments and partners at country level.

The evaluation questions are clustered according to the OECD-DAC criteria, i.e. Impact, Relevance,

Effectiveness, Efficiency and Sustainability, whereby the focus lies primarily on the impact and outcome

results. Furthermore, they include the issue of “Human rights-based approach to programming”. In

response to the evaluation questions, the evaluation tried to establish “clear linkages between impact

results for children, system changes and UNICEF’s contribution”.

2.2 Target countries and territories for the evaluation

In line with the retrospective element of the RKLA – i.e. generating evidence of UNICEF’s contribution

towards outcome and impact results for children in a significant number of countries and territories –,

the ToR refer to five countries and territories where significant impact results in terms of changes in

children’s life have been achieved during the past 10 years. These are (in alphabetical order)

Armenia;

Kosovo (UNSCR 1244);

the former Yugoslav Republic of Macedonia;

Serbia; and

Turkey.



On the basis of the indicators and progress highlighted in the Background Information Sheets provided

with the ToR, the target countries and territories for the evaluation can be related to key thematic areas

as follows:

Country/

Territory

Key thematic

area(s)10 Achievements / Status11

Armenia

Education for

children with

special needs

Number of children in special schools decreased by 60% in 2011-

2012 compared to 2007-2008

Percentage of children with certified disabilities attending regular

schools is 70%

In regions where UNICEF programmes have been implemented

(Yerevan and Tarush), the proportion is even 75-80%

Kosovo

(UNSCR 1244)

Drop-out

Gender parity

Drop-out at primary and lower secondary education levels

reduced from 1.67% in 2002/03 to 0.5% in 2010/11

Gender Parity Index (GPI) in primary school increased from 0.79

in 2004 to 0.93 in 2011

Greater participation of girls in primary education

The former

Yugoslav

Republic of

Macedonia

Roma enrolment

Knowledge in

numeracy and

literacy

Primary school enrolment of Roma children increased from

61.1% in 2006 to 85.6% in 2011

Substantial increases in numeracy and literacy (reading and

writing) by at least 10% against the baseline for (i) 4th grade

students; and (ii) Early Grade teachers (1st-3rd grade)

Serbia Roma enrolment

Quality of education

Primary school enrolment of Roma children increased from 66%

in 2005 to 91% in 2010

Turkey School enrolment

Gender parity

350,000 children (250,000 girls and 100,000 boys) enrolled as a

result of the 2003-2007 Girls Education Campaign Haydi Kızlar

Okula

Gender disparity in education reduced from 7.15% in 2003 to

0.37% in 2010-11

2.3 Strategic principles and ethical considerations

2.3.1 Participatory approach

Since beneficiaries need to feel concerned and have ownership of their proposed intervention,

PROMAN always has been a strong advocate of the participatory approach. Consequently, all

stakeholders were involved right from the start in order to avoid counterproductive actions from their

side. The evaluation team endeavoured to meet, speak and consult with as many stakeholders as possible

given the time schedule. However, just as the overall study outcomes needed to be lucid and

straightforward, so also was the process, facilitated by the consultant team, be both participative and

transparent.

10 This is in no way meant to be exclusive, since we are fully aware that all issues of inclusion have relevance for

all the target countries. This table however links the key indicators provided in the Background Information Sheets

provided in the ToR to the visited countries, which also served as a foundation for deciding on the selection and

deployment of the regional experts. In doing so, we want to ensure that all issues of inclusion are adequately

covered particularly with a view to the whole evaluation process. 11 Cf Background Information Sheets for the respective countries, contained in Appendices 3-7 of the ToR for this

evaluation. Quantitative data will be validated during the country visits.



2.3.2 School selection for field visits

Sampling considerations were not applicable in this evaluation due to its explorative nature, and

also due to the very limited time that was available for each country visit, which did not allow for truly

representative surveys.

Nevertheless, selection guidelines have been communicated to the participating COs whereby the six

schools to be visited per country needed to include:

schools of an average standard (not very strong or very weak ones),

with and without a particular focus on inclusiveness issues;

with different types of composition (single and mixed sex schools);

with different socio-economical environments (poorer against “better-off” areas).

Due to the small size of schools to be visited it was acknowledged that a random selection of six schools

would have contained a considerable risk of bias. Therefore a purposive sampling approach was applied.

Additionally, data from the school visits were triangulated with three more data from other sources, i.e.

(i) data from the quantitative analysis;

(ii) data from the document analysis; and

(iii) data from the consultations with Ministries, Development Partners, NGOs and other

stakeholders at central levels.

A balanced selection of schools was thus ensured not only by having provided the COs with the

abovementioned categories on how to identify schools, but also through a final validation of the schools

identified by the evaluation team. During the validation of the school selection the team not only

focussed on the initial criteria, but also looked at a broad variety of other issues. In the former Yugoslav

Republic of Macedonia, for example, the schools represent a number of different characteristics which

were to the full satisfaction of the evaluation team: (i) a rural school with mixed ethnicity; (ii) an urban

school out of Skopje with single ethnicity, and with a known focus on literacy issues; (iii) a large school

in an urban area out of Skopje with mixed ethnicity (including Roma) and with a strong focus on

multicultural education, and which has been involved in CfS piloting; (iv) a very big school in an urban

environment outside Skopje with mixed ethnicity, and with a strong focus on inclusiveness for SEN

students; (v) an urban school in Skopje with a history of being difficult to manage, and with very

crowded classrooms; (vi) an urban school in Skopje, with specific challenges related to socio-economic

backgrounds, and with a high representation of SEN children; (vii) an urban school outside Skopje,

ethnically mixed, which served as a former CfS pilot school. A similar validation process was pursued

in Kosovo (UNSCR 1244), Serbia, and Armenia.

In Turkey, all schools were identified by the Ministry of National Education. The validation process,

however, did not show a bias in favour of specific criteria. Otherwise, the team would have either asked

to replace the proposed schools, or the information collected would not have been taken into

consideration. This, however, was not necessary. We can confirm that our criteria were fully respected

by UNICEF during the selection process, in line with the main purpose of the school visits having been

to collect additional evidence from stakeholders for further triangulation.



2.3.3 Framework for Interviews and Focus Group Discussions

In order to maximise the efficiency of inputs (manpower and time) and outputs (assessment outcomes,

i.e. data), the number of interviews and FGDs to be conducted in every country/territory was designed

as follows:

Estimated number of activities per country/territory

Interviews Round table discussion Focus Group

Discussions Classroom observation

Total number:

5-8 per country/territory

Total number:

1 per country/territory

Total number:

2-3 per school, i.e. 12-18

per country/territory

Total number:

1 per school, i.e.

6 per country/territory

Ministry of Education:

Planning/EMIS;

Inspectorate basic

education;

Department of ECE

Department of special

needs education

Department of non-

formal education

Others (if existing):

Teacher Union

representative(s)

Teacher Training

College

representative(s)

NGO (dealing with

inclusion)

Development partners

UNICEF CO staff

(duration: 2-3 hours)

School level:

Teachers

Students (mix of grades,

male and female)

Parents (if available)

8-12 participants per

group

(duration: 1 school hour)

Different subjects across

all schools, focussing on

subjects which easily

allow for active

participation of students,

particularly language

subjects, social science

FGDs take place at the school level, with the principal being responsible for making the final selection

of participants.

2.3.4 Protection of informants

The Evaluation Team ensured that the evaluation process was ethical and that stakeholders were

protected and stayed anonymous. In line with UNICEF evaluation standards and other existing

international standards like the ESOMAR standards12, the team protected the confidentiality, dignity,

rights and welfare of all participants in the evaluation process, particularly children. The rich contextual

experience of the national experts further warranted respect for the values of the beneficiary community,

and were further utilised to prevent any ethical dilemmas or issues that could emerge.

Efforts were made to include both men and women, boys and girls, as well as representatives from

different ethnic, geographic, or other relevant groups. Equity dimensions (or related limitations) were

taken into account during the process of school selection.

2.3.5 Capacity building and transfer of knowledge

12 http://www.esomar.org/uploads/public/knowledge-and-standards/codes-and-guidelines/ESOMAR_Codes-

and-Guidelines_Interviewing-Children-and-Young-People.pdf.



While conducting the study, we included a considerable element of capacity building for all those

involved in the multi-country evaluation. Through a cooperative manner, not only the regional

consultants but also relevant officers from the COs were included in the evaluation process. Sharing

and discussing about the evaluation process as a whole, including a discussion on the core findings,

provided all involved people with crucial knowledge and enlightening new insights. This was confirmed

by the COs during the debriefing sessions held at the end of the country visits.

Overall, the main focus of the capacity building was a common understanding of the underlying

evaluation process (which could be replicated in future).

The consultancy explicitly acknowledges work already been done in order to build upon previous

experiences, and in order to provide additional inputs within the larger framework of UNICEF CEE/CIS

programming. Such transfer of knowledge from the international experts to the regional counterparts is

considered to be both beneficial for the recipients, and also indispensable for the overall validity of the

evaluation when utilised for future programming and follow-up assessments.



3 EVALUATION FRAMEWORK AND METHODOLOGY

The evaluation design consists of six key methodological components, which are outlined below with

regards to their subject purpose, practical implementation and limitations:

Extensive document/desk review ( section 3.1)

Review of national and international education outcome data ( section 3.2)

Key informant interviews and Focus Group Discussions (FGDs) ( section 3.3)

Self-assessment by national educational staff ( section 3.4)

Qualitative data analysis ( section 3.5)

Sociological sensing ( section 3.6)

It needs to be noted that all findings are based on these information sources. In particular, qualitative

data collected in the field were triangulated on the basis of all data sources which verified the findings.

A key role played the key informant interviews and FGDs which were documented in great detail

during the field visits on the basis of an Evaluation Matrix template for completion after every single

interviews/FGD.13

3.1 Extensive document/desk review

Subject to the extensive document/desk review are the following types of documents:

Document type Description

Guidelines

All material that is relevant for evaluation design and reporting, general guidelines,

standards, norms, etc., related to the evaluation subject, and UNICEF Programming and CfS

standards

Evaluations Project/programme evaluation reports

Policy Papers Policy, position or strategy papers or Country Programme Action Plans (CPAP), Country

Programme Documents (CPD) or UN Development Assistance Frameworks (UNDAF)

Studies (Quantitative) studies or assessments for statistical analyses, e.g. Multiple Indicator Cluster

Surveys (MICS), Demographic and Health Surveys (DHS), etc.

Reports Project/programme reports, e.g. progress reports of UNICEF, other DPs and Government

Government

documents

Action plans and budgeting, e.g. Medium Term Public Expenditure Framework (MTEF),

education sector plan

Other types Other types of documents apart from those mentioned above, which are relevant for the

evaluation or provide further background information

The documents are one of the key sources providing data regarding the indicators as represented in the

evaluation matrix (see Appendix 4). The purpose of this review is to gain knowledge about the

indicators, both those already defined and those which need to be further developed, all in relation to

the guiding evaluation questions as defined in the ToR.

Since the desk review was entirely explorative, i.e. the changes, influencing factors, contributions and

cause-and-effect relationships were to be discovered, an inductive analysis approach was applied (see

section 3.5). Therefore, the original evaluation questions and resulting indicators (as identified in the

evaluation matrix) have been refined throughout the research process in order to provide a

comprehensive picture of the contribution of UNICEF to the lives of the children and the underlying

ToR.

13 The master template is included in Appendix 5. The completed templates have not been included in order to

protect the anonymity of the respondents. Cf also Section 2.3.4: Protection of informants.



3.2 Review of national and international education outcome data

The review of national and international education outcome data focuses particularly on MICS,

TransMonEE, PISA, UIS and DHS data. The international survey data is used to identify trends

regarding the impact results and differences between the performances of the countries and territories

in the observed timeframe. Especially, the analysis was carried out in a way to differentiate the impact

over groups, particularly targeting the most vulnerable groups (e.g. Roma children). As a consequence,

the data review is mainly used to answer questions concerning the impact, relevance and sustainability,

such as:

Do the impact results reported by countries and territories in terms of greater inclusion and

improved quality of education really represent a change in the lives of children, in terms of the

realisation of their right to quality basic education and/or reduction of equity gaps?14 (IM.1)

Have UNICEF’s interventions at country level that were designed to influence inclusive

policies and system changes, been specifically targeted at the most marginalised children, those

children whose right to education is violated, in particular children with disabilities, Roma

children, girls, children from poor rural areas, children performing below academic standards,

and children with multiple disadvantages? (R.1)

Are there indications that the impact results (greater inclusion of marginalised and vulnerable

children in basic education) will or will not be sustained over time in countries and territories

where they took place? (S.1)

The outcome data has been analysed with descriptive statistical methods.15 Where data availability

allowed it, these methods included the development of time-series as well as basic trend analysis. The

following table indicates the existence of international surveys and administrative data:

Country/Territory MICS16 DHS PISA TransMonEE17 UIS18

Armenia 2000, 2005,

2010 2001-2011 2001-2011

Kosovo (UNSCR

1244)19

The former Yugoslav

Republic of Macedonia

2005-2006,

2011 2001-2011 2001-2011

Serbia 2005, 2010 2003, 2006,

2009, 2012 2001-2011 2001-2011

14 In the interest of consistency, we always refer to the original evaluation questions as stated in the ToR. However,

this particular question is a bit imprecise, so we understand it as “How is the realisation of children’s’ rights to

quality basic education and/or reduction of equity gaps represented by the impact results reported by countries in

terms of greater inclusion and improved quality of education?” In principle, this question will then be approached

in three steps: (1) What do the figures say about the degree of change in education outcomes? (2) To which degree

do government and any other relevant authority accept the figures as accurate? (3) Do teachers‘, parents‘, and

children‘s perceptions of improvements match up with the level of change the figures imply? 15 E.g., univariate analysis, group comparisons, trend analysis. 16 A survey programme developed by UNICEF which assists countries in collecting and analysing data in order

to fill data gaps for monitoring the situation of children and women. 17 The TransMonEE data were solely considered within the timeframe from 2001 to 2011. 18 The UIS data were solely considered within the time frame from 2001 to 2011. 19 No MICS, DHS, PISA and TransMonEE data available for Kosovo (UNSCR 1244).



Turkey 1993, 1998,

2003, 200820

2003, 2006,

2009, 2012

2001-2011

As the table shows, the database is unfortunately not consistent for all visited countries and territories.

MICS data was only available for the former Yugoslav Republic of Macedonia and Serbia, DHS data

for Armenia and Turkey, PISA data for Serbia and Turkey, and TransMonEE only for Armenia, the

former Yugoslav Republic of Macedonia and Serbia. Furthermore, except for TransMonEE, even

within the same database, the surveys were not conducted simultaneously. Finally, the surveys are based

on different research designs and accordingly used different instruments with partially different research

questions. These limitations made it impossible to conduct a general cross-country comparison. It was

rather necessary to analyse the data for each visited country individually and put the results into the

overarching context of the evaluation.

3.3 Key informant interviews and Focus Group Discussions (FGDs)

During the country visits, consultations were undertaken with key informants such as, inter alia,

education policy makers, planners, administrators, Development Partners (DPs), teachers, students,

parents and other partners/stakeholders in the field of education21. Particular attention was given to

those who experienced exclusion/inclusion themselves.

Stakeholder consultations are carried out through interviews, FGDs and also through Round Table (RT)

discussions.22 Consultations were implemented “in order to fill knowledge gaps and carry out a rigorous

triangulation of the information collected through the desk review” (ToR p.16). Therefore, in-depth

information about the implementation on the system/macro level and the network relations between the

involved stakeholders was analysed. The key issues of these consultations were the coordination

mechanisms between the government and UNICEF and its partners and their contributions to the

removal of system bottlenecks to inclusive education. Furthermore, UNICEF’s contribution to

government’s policy formulation, implementation or fine-tuning through its work in terms of

knowledge generation, evidence-based advocacy and policy advice, and technical support were

assessed.

The interviews followed semi-structured guidelines. A set of key topics and related guiding questions

(deriving from the ToR, the initial document analysis of key documents and the emerging evaluation

matrix, cf Appendices 3 and 4) was expanded with further relevant issues raised by the interviewees.

The interviews were documented and again analysed according to qualitative methodology as outlined

below. In addition, interaction patterns between students and teachers were observed during the school

20 The DHS Data Turkey 2008 was not accessible as raw dataset. However, the CO Turkey provided the country

study “All children in school by 2015. Global Initiative on Out-Of-School Children” which included some

statistics of the DHS data 2008. Where applicable, figures and statistics were added to this report. 21 For a detailed list of stakeholders consulted please refer to the table in Section 2.3.3. On average per country, 6

FGDs with 12 participants each were carried out. Together with 12 interviews at school level and a further 15

interviews at central level. In addition, a Round Table discussion comprising 15 participants on average was

carried out. For all countries together, approximately 360 FGD participants, 60 interviewees at school level, 75

interviewees at central level and 75 RT participants directly contributed to the evaluation, thus a total of 570

stakeholders. 22 A Round Table Discussion is a specific group discussion where participants discuss and debate a specific topic.

The specific feature of a RT discussion is that each participant has equal rights to participate, which is facilitated

through the circular seating layout usually applied in such discussions. The RT is chaired by a facilitator (the

international consultant) who ensures that the equal rights to participate are not infringed upon by speakers who

tend to dominate the process.



visits in the field, in order to assess the degree of student participation in line with active learning

principles.23

3.4 Self-assessment by national education and UNICEF CO staff

Complementary to the desk study, the key informant interviews and the FGDs, a self-assessment by

national education staff and UNICEF CO staff was carried out.24

The self-assessments are perceptive and, due to the purposive sampling strategy of the countries and

territories as well as the staff, not statistically representative. Accordingly they were primarily carried

out to fill knowledge gaps and to triangulate the information gathered by the other means of data

collection.

The self-assessment serves to assess the implementation process and the impact on the institutional

level by identifying the level of awareness of the major partners with regards to UNICEF’s programme

goals and strategies at country level. This also contributes to the analysis of the effectiveness of

government’s interventions in removing system bottlenecks that determined or contributed to the

exclusion of marginalised children from education.

The self-assessments focused on a set of items within the framework of the interviews around the key

dimensions of the impact results and their work and relevance in the national education framework.25

3.5 Qualitative and quantitative data analysis

3.5.1 Theoretical background

The analysis of all qualitative data gathered from the desk study, the interviews and the self-assessments

follows a circular approach26, whereby the evaluation questions are guided by the ToR and refined on

the basis of the data material. The starting point is the transformation of the evaluation questions into

indicators. This translates into the evaluation matrix (see Appendix 4).27 Within this framework, the

documents, interviews and self-assessment results represent the analytical cases, i.e. the data required

for the indicators respectively to answer the research questions.

23 Please refer to Appendix 5 for a complete set of evaluation tools, including the guidelines for interviews and

FGDs, together with the respective recording forms. Training to the national experts will be provided at the initial

workshop in Kosovo (UNSCR 1244). 24 Self-Assessment Sheets (SAS) are perceptional assessment tools and provide the respondents with 50 items to

which respondents need to give an indication to which degree they agree with a given statement. The tools are

included in Appendix 5. 25 The original technical proposal foresaw a period of two weeks for every country visit which however was

decreased following UNICEF’s request to reduce costs. In line with the reduction of time for the field visits, the

inclusion of the self-assessment into the interview schedule replaces the original proposal to administer semi-

structured questionnaires for the self-assessment. As discussed and agreed during the Geneva briefing, forms for

the self-assessment have not been included in the evaluation tools provided in Appendix 5 in order to avoid

“prepared responses”. 26 Or, in a wider sense, the Grounded Theory Methodology. However, in the interest of easier understanding, we

will refrain from referring to this specialised methodology. 27 An initial version of the Evaluation Matrix was developed during the first evaluation team workshop between

02 and04 July 2013. This version has since then been refined twice, with the latest refinement undertaken after

the pilot field phase in Kosovo (UNSCR 1244). Appendix 4 contains this final version which has been applied

during all the field visits, and which also serves as a foundation for the reporting structure of this Evaluation

Report.



On the basis of the evaluation matrix, the data were worked through and relevant content was

assigned to one or more indicators for which they provided information for answering the

evaluation questions.28

Then, each indicator was firstly analysed individually in order to achieve findings about the

actual changes in children's lives and the contributions of UNICEF to these changes.

In the next step, noticeable patterns, i.e. repeating combinations of particular characteristics

(e.g. dependency of impact results on particular framework characteristics) were identified.

This approach aimed at identifying empirical connections between different indicators by integrating

them into a causal model.29 In this case, cause-and-effect relationships were identified to draw

conclusions about the factors which influence the changes in children’s lives and which determine

the effectiveness and efficiency of UNICEF’s contributions.

The results were then quantified (scored) by the evaluation team as far as possible in order to build and

substantiate the hypotheses. After identifying relations between different project characteristics, the

findings were integrated into overarching concepts and conclusions about common factors of success

and failure, which again allowed for developing general recommendations for the future design of

UNICEF’s projects. The objective of this last step was to formulate a theoretical model with a limited

scope, which devises concrete contexts, conditions and strategies and their consequences in their

relational references to a particular phenomenon, i.e. the impacts results for children.

Within the evaluation team, the analysis of the qualitative data was organised individually, so each

evaluator worked on a separate set of data (i.e. documents, interviews, self-assessments). In order to

achieve a homogeneous results picture, an intensive exchange between the evaluators was ensured by

regular meetings, with workshops at the end of each step of data collection. During these workshops

the individual results were integrated into a consolidated evaluation matrix.30

Accordingly, the analysis did not only rely on data triangulation but also included peer reviewing

as the results of each evaluator were re-examined by the two others. As a consequence, both the validity

and reliability of the analysis results was assured at highest standards.

3.5.2 Overview of evaluation questions, indicators and related information

As a first step, the general answerability of the evaluation questions was assessed and presented in the

Inception Report. In principle, all evaluation questions were regarded as being “answerable”, a fact

which was then confirmed during the field visits.

In line with the theoretical background explained above, Appendix 4 contains the final evaluation

matrix with all the evaluation questions and subsequent details on indicators, data types, availability,

data sources, methods and related instruments. In particular, the table shows the related MoRES

determinants for the evaluation questions and indicators (using the codes introduced in Section 1.3.4

above).

28 Relevant information which did not fit in the grid was grouped and, during the refinement process of the

Evaluation Matrix, new indicators were developed on the basis of the data material. 29 This is an inductive approach which aims at developing a model (see following paragraph), not testing it. We

try to understand HOW it works and not to prejudice THAT a particular model works. There is no model yet to

test. 30 For confidentiality reasons, and in order to protect respondents, Evaluation Matrices compiled by the Regional

Consultants (for every visited country) are not contained in the report. However, they can be made available upon

request to UNICEF RO.



It should be noted that indicators are both at a very high level and mostly generic, i.e. without specific

timelines and targets. Whenever indicators start with “Degree of...” or “Value of...”, they were scored

on the basis of a 3-level “traffic light” system. The same scoring was then applied to the quantitative

data. A complete overview is contained in Appendix 4b.

When reading the findings, for some of the more descriptive findings there will be no direct causal link

or explanation apart from some fairly good hypotheses; however these statements should be seen as an

invitation to national stakeholders and country offices to further analyse these and drawing conclusions

feeding into new policies and strategies.

3.5.3 Methodological limitations

During the development of the initial evaluation matrix (as contained in the 1st and 2nd draft of the

Inception Report) a number of potential data gaps were addressed by scrutinising available secondary

data and/or by establishing which further data needed to be collected.

In order to mitigate this possible limitation, the evaluation matrix was reviewed in terms of the

probability of data availability during the pilot phase in Kosovo (UNSCR 1244).31

3.5.4 Data reliability

While on the one hand the country specific results are quite heterogeneous – meaning that not many

consistent conclusions for the entire region can be drawn –, on the other hand the results from the

different data sources provide a quite consistent picture, at least with regards to the overall assessment

of the different evaluation questions. The relatively high accordance of (statistical, document based)

secondary data with primary empirical data from the field visits at least indicates high reliability and

validity of the entire database. Furthermore, the mixed methods approach and the task division among

the evaluation team members32 allowed for data and investigator triangulation, which minimises the

probability of systematic errors, such as introduced biases of interviewees or evaluators.

Data gaps have been addressed where required. These gaps were mainly caused by missing (e.g.

international benchmarks on resource efficiency) or incomparable (e.g. MICS vs. DHS data sets)

statistical data. However, in most cases the missing data could be substituted with qualitative

information from the field and/or further background document studies. This in particular refers to the

lack of quantitative data for Kosovo (UNSCR 1244).

3.6 Sociological Sensing

Sensing is a term coined by Harvie FERGUSON and commonly used in Phenomenological Sociology.

It is an important qualitative method especially in cases where concrete data from stakeholders is not

readily available, but where phenomena can be observed by the researcher in the field and compared

with his/her own experience in a variety of settings. By characterising observed phenomena and

comparing them to similar observations, sensed assumptions can then be utilised for further research by

means of questionnaires and items for interviews. Some approaches and tools for impact assessment

refer to the term Guessing when recommending to set boundaries by references to another variable, or

to employ triangulation (using several separate approaches/data sources to estimate a quantity and to

compare the results). Sensing can equally contribute to such a process in order to make data accessible

in more challenging circumstances.

31 Cf also Section 4.3. 32 Cf also introduction to Chapter 5.



With regards to the assessment of (changing) mindsets and behaviour patterns, sensing can generate an

important pool of observations which can subsequently be enriched and/or validated through

stakeholder consultations. Particularly during the execution of the interviews and FGDs, the researcher

focuses on additional aspects apart from the spoken word (e.g. assessing the atmosphere, observing own

physical reactions, receiving a subjective perception of the situation within which the interview/FGD is

placed). In other words, the researcher places him-/herself immediately within the context of the

interviewee and ‘senses’ various aspects of the determining environment, “dissolved into qualitatively

distinct aspects of consciousness”33, in order to get a “feel” for the situation the interviewee is in (also

allowing for inductively ‘guessing’ behavioural and attitudinal consequences at higher levels).

During the course of this evaluation, sociological sensing played an important part in facilitating a sound

understanding of the respective societies’ environment. The ability of the researchers to become aware

of, for example, political constraints and/or ethnical tensions assisted them in putting observed

phenomena into context, and find meaning or coherence in underlying reasons of various opinions of

the interviewees, particularly if these would feel “odd” if the context of the interviewee would be

neglected.

33 Cf Harvie FERGUSON (2006): Phenomenological Sociology. Experience & Insight in Modern Society.

London, Thousand Oaks CA and New Delhi: SAGE Publications Ltd., p. 115.



4 ISSUES HAVING EMERGED DURING THE INCEPTION PHASE

This section outlines some issues which emerged during the inception phase34, particularly after the

finalisation of the pilot visit to Kosovo (UNSCR 1244).

4.1. Inception phase up to the completion of the draft inception report

Following the submission of the draft Inception Report, Country Offices (COs) were contacted in order

to facilitate the preparation process for the implementation phase of the evaluation. The assistance of

the COs was crucial in order to ensure that (i) the additionally requested data envisaged to account for

the indicators became available; and that (ii) the field visits could be prepared, particularly in terms of

logistical issues such as inviting participants to round table discussions and fixing appointments with

stakeholders listed for interviews, as communicated to the COs.

The draft Inception Report was distributed within UNICEF Regional Office (RO) including the

Regional M&E facility. A comprehensive set of comments was received which informed the drafting

of the Final Inception Report. Comments were then discussed at a 2-hour-teleconference with UNICEF

on 29 July, with main issues relating to the formulation of indicators, their linkage to the MoRES

framework and the answerability of evaluation questions. Outcomes of this discussion were then

reflected in the Final Inception Report.

At the end of the initial inception phase, it was further agreed that the approval of the final inception

report would be foreseen for the time after the completion of the Kosovo (UNSCR 1244) field visit, on

the understanding that the evaluation team was mandated to continue with the document review and the

preparation of the country visits even though a formal approval of the Inception Report would have

been received at that time.

4.2 Inception phase up to the completion and approval of the final inception report

One core outcome of the UNICEF teleconference during the initial inception phase was the agreement

to review and validate the evaluation matrix35 against the experiences to be made during the pilot visit

to Kosovo (UNSCR 1244). The main task would be to review and verify the “maximum catalogue” of

indicators during the actual research process.

On the 13th of September, the pilot phase concluded with a video conference with the Regional Office

during which the outcomes of the field visit were presented and reflected upon against the proposed

methods, approach and tools including the evaluation matrix. Furthermore, any other outstanding issues

were reviewed and discussed.

4.3 Key outcomes from the pilot visit to Kosovo (UNSCR 1244)

Generally, the pilot visit in Kosovo (UNSCR 1244) validated the approach and the methodology,

and led to a number of further refinements.

During the first week of the pilot visit, all evaluation tools were applied with all target groups. At the

onset of the pilot mission, the regional consultant for Kosovo (UNSCR 1244) received an in-depth and

condensed training on the application of the tools (which was then expanded for the training of the

complete team of regional consultants during the second week).

34 The Inception Phase comprises two phases, i.e. (i) the timeframe for completion of the draft inception report;

and (ii) the timeframe for completion and approval of the final Inception Report. 35 Cf Appendix 4.



Following the comments received on the first two drafts of the Inception Report, a key element of the

briefing again referred to the fact that guidelines for interviews and FGDs must not be understood as

questionnaires, but need to be seen as guidelines for discussion. This fact however was clearly

understood by all regional experts, also due to their vast experience in conducting qualitative field

research themselves in the past for a considerable period of time.

All evaluation tools proved themselves relevant and contextually appropriate during field

implementation. No changes were made to the tools following the Kosovo (UNSCR 1244) pilot phase,

and an intensified training particularly for the classroom observation sheet was included in the team

training workshop in order to ensure comparability of observations. Further, the selection of schools

for school visits was counter-checked and found to be in line with the selection guidelines provided

to UNICEF COs.

The Kosovo (UNSCR 1244) CO provided an opportunity for an expanded composition of the Round

Table (RT) discussion by including participants from non-English environments, which proved to be

very useful. The Kosovo (UNSCR 1244) model of conducting an expanded RT has been conveyed

to other COs and recommended for adoption, and has subsequently been applied during the field

visits with the exception of Serbia.

During the team training workshop, a key focus was on the validation of the evaluation matrix.36

Utilising two workshop days for highly intensive and participatory consultations, the Evaluation Team

validated the evaluation matrix with a number of changes which are documented in the final matrix

contained in Appendix 4 of this report.

In principle, the team confirmed the “maximum catalogue” of indicators and further confirmed the high

probability of receiving information related to the indicators and evaluation questions. It was agreed to

maintain the scope of the “maximum catalogue” which was validated and thus reconfirmed by the

international core team during their team workshop in Frankfurt in October, immediately following the

field phase.

Regarding the operationalisation of generic indicators requiring a qualitative measurement,37 it was

agreed to utilise a 3-level “traffic light” scoring system.38

Finally, the Kosovo (UNSCR 1244) pilot phase concluded with a debriefing presentation to the CO

Head and staff which has also been shared with the Regional Office.39

36 It was agreed by UNICEF and the evaluation team during the 29 July teleconference that indicators are both at

a very high level and mostly generic – due mostly to the formulation of the evaluation questions in the TOR. It

was also agreed that during the pilot phase the Evaluation Matrix would be revised and validated, and that a final

version would be included in the Final Inception Report. 37 I.e. generic indicators commencing with formulations such as “Degree of …“; “Extent of ...”; “Value of change

of …”. 38 This is shown in detail in Appendix 4b. 39 The country-specific presentations will not be included in the reporting (Inception Report and Evaluation Report

respectively) since the evaluation is not supposed to focus on country-specific achievements. They can however

be requested through the Regional Office.



5 EVALUATION FINDINGS

This chapter assesses the achievements towards the realisation of inclusion40 according to five

organising criteria as follows:

Changes in the lives of children (impact results)

Equitable access to basic education

Education quality and learning achievements

Equality in learning opportunity

Effectiveness of systemic changes for the most vulnerable children

Relevance of UNICEF’s approach

Relevance towards the most vulnerable

Relevance towards prevailing education sector policies and international standards

Relevance towards the work of national partners

Contribution of UNICEF to system changes through its core roles (efficiency)

Sustainability of results for children and system changes

Additional findings regarding adequacy, coordination, coherence, protection, HR and gender have been

mainstreamed under these five organising criteria. In addition, the cross-cutting issue of a Human

Rights-based approach to planning will be considered, and data reliability will be assessed.

The data analysis followed a mixed methods approach, combining secondary data from documents and

statistical surveys with empirical evidence from the field visits. With this type of methodological

triangulation the reliability of the results could be considerably improved as the data sources (project

reports, statistical databases, interview results, self-assessments etc.) were widely independent (e.g. the

persons who were interviewed or filled out the self-assessment sheets were most likely not the same

persons who wrote the project reports or who conducted the statistical surveys).

Since the document analysis and the empirical data collection were divided among the evaluation team,

an additional investigator triangulation was feasible which again improved the reliability of the results.

Starting point of the analysis were the available documents and statistics, which were used to fill the

evaluation matrix with adequate indicators and data providing information about the achievement of

these indicators.41 These preliminary results were then compared with the empirical data and amended

or as far as necessary revised. Qualitative data collected in the field were triangulated on the basis of all

data sources which verified the findings.42

40 Inclusive education promotes the “recognition of the need to work towards ‘schools for all’ / institutions which

include everybody, celebrate differences, support learning, and respond to individual needs” (Salamanca

Statement, World Conference on Special Needs Education, held in Salamanca, Spain, June 1994). 41 If the empirical data fully confirmed the statements made in the documents, the text sticks as closely as possible

to the original wording in order to maintain a consistent language. 42 A key role played the key informant interviews and FGDs which were documented in great detail during the

field visits on the basis of an Evaluation Matrix template for completion after every single interviews/FGD. The

template is included in Appendix 5.



5.1 Changes in the lives of children (impact results)

5.1.1 Equitable access to basic education

(1) In the visited countries and territories, the key focal areas for UNICEF in terms of improving

inclusion relate to the inclusion of children with special educational needs (particularly those

with disabilities), children from ethnic and linguistic minorities including Roma, and

children from low socio-economic backgrounds.

(2) All visited countries and territories place considerable importance on universal enrolment. The

following chart visualises the steady increase of the adjusted net enrolment rate (ANER)

between 2002 and 2012 for Armenia and the former Yugoslav Republic of Macedonia. In the

case of Turkey, the rate remains rather stable at an already high level. In contrast, the ANER in

Serbia appears to decline considerably. However, as the empirical data gathered in the field

does not provide any explanation to support this finding and the values in the UIS database are

flagged as “national estimations”, this development might rather be a statistical artefact due to

improvements in the measurement approach than a factual trend.43 This assumption is also

backed up by the fact that the rates for 2005 and 2006 are almost 100, which is – even for a

well-developed country – possible but not very likely.

Adjusted net enrolment44 rate, primary

Data source: UIS

(3) Interviews and focus group discussions have confirmed that governments in all visited countries

and territories succeeded in achieving increased access to education for children who had been

excluded from education or dropped out early through catch-up programmes (in particular

43 As can be seen in the following, the “national estimations“ in the UIS data centre from Serbia do not appear to

provide reliable information. The analysed indicators show, particularly in contrast to the TransMonEE and MICS

data (cf Sections 5.1.1.3 and 5.1.2) as well as the PISA data (cf Section 5.1.2), a negative or inconsistent

development, which does not allow for a plausible interpretation. 44 The adjusted net enrolment is the number of pupils of the school-age group for primary education, enrolled

either in primary or secondary education, expressed as a percentage of the total population in that age group.



girls). Governments have abolished mechanisms hampering access to education, and are

increasingly aware of the importance of school readiness for school attendance and completion

in particularly for marginalised children.

(4) With regards to the documentation of the rate of Out-of-School children (OOSC) in primary

education the database drawing from national statistics is relatively scarce. Furthermore, the

data which is available produces a partly ambiguous picture, as can be seen in the chart below

which has been based on all available UIS data.

Rate of out-of-school children of primary school age, both sexes (%)

Data source: UIS

(5) For Armenia, it is interesting to observe an increase from 2002 to 2004 followed by decline

until 2007 to about the half of its starting value. Likewise, the former Yugoslav Republic of

Macedonia shows an exceptional improvement of the rate of out-of-school children over a

period of just eight years.

(6) In Turkey, the development follows an absolute linear trajectory line with an average decrease

of 0.4 per cent per year, which, at least from a probabilistic point of view, is rather unlikely to

happen in reality. Again Serbia shows a reverse development (cf §2), which cannot be

substantiated by empirical data from the country visit but rather suggests improvements in the

registration system.45

5.1.1.1 GIRLS AND BOYS

(7) National averages of gross and adjusted net enrolment rates do not display major gender

disparities in education except for Armenia, where girls appear to have a better access to

primary education than boys.

(8) As the following figures show, Armenia and the former Yugoslav Republic of Macedonia

feature a positive development during the observation period, while for Serbia the ANER seems

to decline considerably. However, as already discussed in paragraph 2 and footnote 42, the

45 The verification of this hypothesis requires further monitoring of the indicator in the future.



latter finding is rather a result of a changing measurement approach than of factual changes of

empirical data. Unfortunately for Turkey ANER data is only available between 2009 and 2011,

which makes it difficult to estimate a ‘trend’. However, the values provided allow the

assumption that the enrolment remained stable at a very high level.

Adjusted net enrolment rate, primary,

Armenia:46

Adjusted net enrolment rate, primary,

the former Yugoslav Republic of Macedonia:

Data source: UIS Data source: UIS

Adjusted net enrolment rate, primary, Serbia:47

Adjusted net enrolment rate, primary, Turkey:

Data source: UIS Data source: UIS

(9) The gross enrolment ratios displayed below confirm the patterns for the adjusted net

enrolment rates for Armenia and the former Yugoslav Republic of Macedonia. As the UIS

database provides values for the entire observation period for Turkey, it is also possible to

confirm the assumption made above about the constantly high enrolment, whereas the

convergence of the ratios for female and for male pupils indicate that campaigns such as the

Girls Education Campaign had a positive effect on gender equality. With regards to Kosovo

(UNSCR 1244), data is only available for the school years from 2009/2010 to 2011/2012. While

it is not possible to make a statement about the long term trend, at least the comparably high

values for the last three years suggest a currently satisfactory participation in primary education,

both, of boys and girls.

46 There may be calculations errors as there is mismatch between UIS data and data provided by National

Statistical Service. 47 For the reasons stated in §2 and §6 (cf. also respective footnote) the statistical data for Serbia from the UIS data

centre is not taken into consideration in the assessments in this section.



Gross enrolment ratio, Armenia:

Gross enrolment ratio, primary,

the former Yugoslav Republic of Macedonia:

Data source: UIS

Data source: UIS

Gross enrolment ratio, primary, Serbia:48 Gross enrolment ratio, primary, Turkey:

Data source: UIS

Data source: UIS

Gross enrolment ratio, primary, Kosovo (UNSCR

1244):

Data source: MoEST, Kosovo (UNSCR 1244)

(10) The figures below show the attendance rates disaggregated by gender for Armenia and

Turkey. As can be seen the attendance rate for first-graders (age group of six and seven year

old children) and adolescents increases considerably during the observation period in both

countries and territories, both for boys and girls. In Armenia the development of the attendance

48 For the reasons stated in §2 and §6 (cf. also respective footnote) the statistical data for Serbia from the UIS data

centre is not taken into consideration in the assessments in this section.

85

90

95

100

105

110

115

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Gross enrolment ratio in primary education in Armenia

Female Male

85

90

95

100

105

110

115

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Gross enrolment ratio in primary education in the Former Yugoslavian


Female Male

85

90

95

100

105

110

115

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Gross enrolment ratio in primary education in Serbia

Female Male

85

90

95

100

105

110

115

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Gross enrolment ratio in primary education in Turkey

Female Male

85

90

95

100

105

110

115

2009/2010 2010/2011 2011/2012

Gross enrolment ratio in primary education in Kosovo

Female Male



of first-graders is particularly impressive49, while Turkey attendance rates improved remarkably

over the time span between 1993 and 2003, especially when looking at the children of the ages

of secondary school.

(11) The figures show further that both countries feature a gender gap in High School attendance.

While in Armenia the attendance rate for girls is slightly higher than for boys, in Turkey on

average boys appear to attend school until a later age than girls. For Turkey it is also interesting

to note that while the attendance rates remained more or less stable between 1993/1994 and

1998/1999, it increased considerably between 1998/1999 (mainly because of compulsory

primary education expansion to 8 years) and 2002/2004.

School attendance rate, Armenia: School attendance rate, Turkey50

Female school attendance rate, Armenia:

Female school attendance rate, Turkey:51

Male school attendance rate, Armenia:

Male School attendance rate, Turkey:

Data source: DHS Data source: DHS

49 To be noted: Armenia changed the school starting age from 7 to 6 years in 2008. 50 The DHS data of 2008 was taken out of the Turkey country report “Global Initiative on Out-of-School-Children”

[sic], 2012. The data may not be accurate or consistent to the data calculated with previous DHS data since there

was no access to raw data. 51 No new data from DHS 2008 available.



(12) In conservative areas of visited countries and territories, participation of girls remains a problem

in particular for girls whose mother tongue differs from the official language of instruction.

This difference has a bigger impact on girls since boys learn the official language on the street,

while girls do not have that opportunity because they are at home.

(13) Being already at very high levels above 90% or 95%, the literacy rate of young women aged 15

to 24 increased in all three countries. However, only in Serbia the increase proved to be

statistically significant (p ≤ 0.05)

Young Women Literacy Rate (age 15 -24; the former Yugoslav Republic of Macedonia and Serbia):

Young Women Literacy Rate (age 15-24, Turkey):52

Source: MICS and DHS53

52 No new data from DHS 2008 available. 53 No MICS data available for Armenia, Kosovo (UNSCR 1244) and Turkey; no DHS data available for Kosovo

(UNSCR 1244), the former Yugoslav Republic of Macedonia and Serbia.

0

10

20

30

40

50

60

70

80

90

100

The former Yugoslav Republic ofMacedonia

Serbia

MICS 3

MICS 4



5.1.1.2 GENDER PARITY

(14) All visited countries and territories show a Gender Parity Index (GPI) of nearing or even

exceeding 1.0 which is a strong indication of girls and boys having equal access to schooling

(net intake rate). The rate for Kosovo (UNSCR 1244) needs to be seen against a reported GPI

in primary school from 0.79 in 200454, and which represents a significant improvement.

Gender Parity Index (GPI) for adjusted net intake rate in primary education

Source: UIS; for Kosovo (UNSCR 1244): own calculations on the basis of data from MoEST, Kosovo (UNSCR 1244)

*) The GPI for Kosovo (UNSCR 1244) has been estimated on the basis of the number of students in first class, which does

not account for drop outs. However, it provides a good approximation of the intake rate as their number (i.e. some hundred)

is usually considerably lower than the number of new students (i.e. some ten thousands).

(15) In Turkey, it is also interesting to note the steadily increasing GPI by educational year and level

of education, with a significant improvement particularly for girls.

Turkey, GPI by educational year and level of education

Source: MoNE Statistics Formal Education

54 Background Information Sheet for Kosovo (UNSCR 1244) attached to the TORs; this could however not be

verified with the calculation method used for this chart.

0.90

0.95

1.00

1.05

1.10

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

GPI for adjusted net intake rate in primary education

Armenia Kosovo* Serbia The former Yugoslav Republic of Macedonia Turkey

70

75

80

85

90

95

100

105

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Primary education Secondary education



(16) A similar pattern can be observed for (i) the GPI for the transition rate from primary to

secondary education; and for (ii) the GPI by net attendance rate. This is a strong indication of

(i) girls and boys having equal opportunities to progress through the educational system; and

(ii) of remaining in school and actively attending after initial enrolment.

Gender Parity Index (GPI) for transition rate from primary to secondary education

Source: UIS; for Kosovo (UNSCR 1244): own calculations on the basis of data from MoEST, Kosovo (UNSCR 1244)

*) The GPI for Kosovo (UNSCR 1244) has been estimated on the basis of the number of students in sixth class, which does

not account for repeaters or drop outs. However, it provides a good approximation of the intake rate as their number (i.e.

some hundred) is usually considerably lower than the number of new students (i.e. some ten thousands).

Gender Parity Index (GPI) by net attendance rate in primary school age55

Source: DHS (Armenia, Turkey); PROMAN’s calculations using MICS databases

(the former Yugoslav Republic of Macedonia, Serbia)

55 The figure is based on own calculations of the DHS and MICS data sets. In order to make the data comparable

the GPI has been calculated for the ‘core age’ of primary education according to the ISCED classification (cf UIS

2011:30).

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1

1.01

1.02

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Armenia The former Yugoslav Republic of Macedonia Serbia Turkey



5.1.1.3 ETHNIC MINORITIES

(17) The data on school attendance disaggregated by ethnic minority show that in Serbia the

attendance rate of 11-15 year old Roma children increased considerably in between 2004/05

and 2010/11. This trend exists for both Roma girls and boys (see solid red and blue lines).

Roma children vs. all children attendance rate, Serbia: Roma girls vs. Roma boys attendance rate, Serbia

Roma girls vs. all girls attendance rate, Serbia: Roma boys vs. all boys attendance rate, Serbia:

Data source: PROMAN’s calculations using MICS databases

(18) School attendance of Roma children, especially after the ages of 10 or 11, remains problematic

in all visited countries and territories as illustrated by the example of the former Yugoslav

Republic of Macedonia (see chart below). The percentage of Roma children going to school

decreases with each age group and merely improved in time. The situation was improved in

2006/2007, when the former Yugoslav Republic of Macedonia introduced compulsory nine-

year education by extending the cycle to include the pre-primary year, or the 6-year-olds.56

UNICEF provided specific support in the preparation for the introduction of this year in

compulsory primary education.

(19) Significant disparities remain regarding attendance of ethnic minorities. The graphs (below)

based on MICS data show that the older Roma children at the end of primary school stop going

to school regularly – this trend is found for both boys and girls. Even if there was an

amelioration (especially for Roma boys from 2004/2005 and 2010/2011), the drop-out rates for

Roma children already after the age of 13 do not catch up with the national average between

the two MICS’s. Children from ethnic minorities still face a lot of difficulties to continue with

secondary and tertiary education.

Roma children vs. all children attendance rate, the Roma girls vs. Roma boys attendance rate,

56 Likewise, Serbia also introduced a 9th year of compulsory education as a pre-primary year which resulted in an

increased attendance of Roma children in the ages 5-7.



former Yugoslav Republic of Macedonia the former Yugoslav Republic of Macedonia

Roma girls vs. all girls attendance rate,

the former Yugoslav Republic of Macedonia

Roma boys vs. all boys attendance rate,

the former Yugoslav Republic of Macedonia


(20) In order to tackle the drop-out issue, UNICEF collaborates with NGOs in some countries and

territories in order to increase the enrolment of Roma children, through, inter alia, learning

centres providing pre-school education/kindergarten classes, language classes, homework

support and assistance, and parenting programmes. Coupled with the EDC centres, UNICEF’s

support to enable enrolment without birth certificate has contributed, according to UNICEF

field office, to an increase in Roma enrolment to 95% in the northern part of Kosovo (UNSCR

1244), i.e. Mitrovica North. In cooperation with the Municipality department of labour,

continuation of educational participation of RAE children was stimulated through the

development of a certified VET programme (high school level).

Kosovo (UNSCR 1244), share of ethnic minorities in

primary education enrolment

Kosovo (UNSCR 1244), drop-out rates in primary and

lower secondary education (1-9)

Source: Own calculations on the basis of data from MoEST,

Kosovo (UNSCR 1244)


0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

Share of ethnic minorities in primary education enrolment

0.00%

0.50%

1.00%

1.50%

2.00%

2002/2003 2005/2006 2009/2010 2011/2012

Drop out rates in primary and lower secondary education (1-9)



(21) Field data confirm that the degree of access of Roma children in mainstream classes is

satisfactory until the end of primary education. Interviews with educational staff in countries

and territories with a substantial Roma population show that Roma communities are recognised

as one of the most marginalised groups. Interviewees also demonstrated quite a good

understanding on the importance of the inclusion of Roma children in mainstream education.

Interviewees were able to describe interventions aiming to facilitate the integration of Roma

children in school, in particular introduction of Roma mediators/pedagogical assistants, and

attached classes. This especially refers to drop outs, girls and returnees.

In Serbia there are notable improvements in terms of attendance of 11-15 year old Roma children, net attendance

rate going from 74% in 2005 to 89% in 2010. Data show that the older Roma children stop attending school at

the end of primary school already. This is also the case in the former Yugoslav Republic of Macedonia.

5.1.1.4 CHILDREN WITH SPECIAL EDUCATIONAL NEEDS

(22) Although there are no comparable data across the visited countries and territories with regards

to education for children with special educational needs, national databases for Armenia and

Turkey show significant progress regarding mainstreaming efforts.57

(23) In Armenia, where the focus on mainstreaming children with SEN has been particularly strong,

this is clearly evidenced by the rapidly decreasing number of children in special schools. This

can be directly related to the UNICEF-supported systemic shift towards inclusion as realised

by the Armenian government.

(24) Likewise, the general systemic shift towards inclusion resulted in a steadily increasing number

of students attending regular schools in Turkey (either in special education classes at

mainstream schools or in mainstreamed classes).

Armenia, number of children in special schools

Turkey, number of SEN students attending regular

schools

Source: MOES annual Statistical Data Bulletin for 2010-2012 Source: MoNE Statistics Formal Education

(25) In addition to the mainstreaming process, in Armenia special pedagogues develop individual

education plans for SEN children, although the follow-up by mainstream teachers remains

problematic. In the other visited countries and territories, not all teachers in the schools visited

developed or applied an individual education plan for SEN children, based on a regular

assessment of students’ developmental and learning progress. Even though monitoring the daily

progress is foreseen by some national laws, not all teachers possess the required student

assessment or lesson planning skills.

(26) Wherever SEN schools are transformed into resource centres, like in Kosovo (UNSCR 1244),

57 For the former Yugoslav Republic of Macedonia and Serbia, no specific mainstreaming data are available in

the same way as they are available for Armenia and Turkey.

0

20000

40000

60000

80000

100000

120000

140000

160000

180000

200000

2009 2010 2011 2012

Number of students in special education class and inclusive

education in regular schools (primary and secondary)



they can be considered as an intermediate step towards inclusion of SEN children in mainstream

schools. However, the majority of school buildings are not yet designed to provide access for

SEN children. Recently, some new school buildings have begun to create the necessary

facilities of children with special educational needs. This can be one of the reasons why the

number of SEN children in special schools and attached classrooms58 has remained rather stable

over the past decade.

Kosovo (UNSCR 1244), number of children in special schools and attached classrooms


(27) Schools visited also demonstrated some mixed experience with “catch-up” and “second

chance” programmes. On the one hand, vulnerable and marginalised children were reached as

part of catch-up programmes and were subsequently attending school. However, they usually

did not perform well in terms of learning achievements.

5.1.1.5 CHILDREN FROM DISADVANTAGED BACKGROUNDS

(28) The common problem of all countries and territories is the prevailing substantial difference in

attendance between children living in the poorest households and those in the richest

households, and the rural and urban area at the secondary level. The graph below shows the

non-attendance of children per wealth quintiles. However, at least in the former Yugoslav

Republic of Macedonia, the percentages of non-attendance over all five wealth groups decrease

significantly (p ≤ 0.05) between the two time points. Still in Serbia, the percentage of the poorest

children not going to school appears to have increased or at least remained stable over time

while the share of ‘richer’ children declined considerably.

58 Attached classrooms are special classes within mainstream schools.

340

360

380

400

420

440

460



Percentages of children not attending school (primary and secondary education) per wealth quintiles in the

current school year in the former Yugoslav Republic of Macedonia and Serbia59


(29) Recent data from Armenia show a lower existing youth unemployment rate compared to the

former Yugoslav Republic of Macedonia and Serbia. However still about 40% of the Armenian

youth were without a job in 2009 to 2011. The former Yugoslav Republic of Macedonia has

encountered a successive decrease in youth unemployment from enormously high rates of 65%

in the early 21st century to about 55% in 2011. In all the three countries, the youth

unemployment rate rose after the financial crisis in 2007/2008, with Serbia experiencing the

highest visible increase.

Youth Unemployment Rate (based on TransMonEE data):

59 The figure is based on own calculations of the MICS data set. In order to make the data comparable the OOSC

rate has been calculated for the ‘core age’ of primary and secondary education according to the ISCED

classification (i.e. ISCED 1 -3; between 6 and 18 years, cf. UIS 2011:30, 33, 38).



(30) Although education in the visited countries and territories is free of charge, non-tuition costs

substantially burden low-income families. In some cases, pupils are commonly forced to take

private tutoring to gain access to the next educational level. In Turkey, the Conditional Cash

Transfer (CCT) has been introduced in 2003 and subsequently been taken over by Government.

The World Bank is studying whether CCT should be given at community (or school) level

instead of household level, to make people aware of the ‘conditions’ for cash transfer (the why),

i.e. the impact of education on the child’s development. CCT has also been introduced in the

former Yugoslav Republic of Macedonia, although the effects of it have not yet been evaluated.

5.1.2 Education quality and learning achievements

(31) The percentage of repeaters in primary education is one of the indicators for quality education.

In Serbia there was a visible reduction from 1% to 0.4% in a ten year time span. However, if

percentages do not show a significant change over time this does not imply that quality of

learning did not improve. In Armenia, the percentage of repeaters was already very low i.e.,

0.1-0.2%. In actual numbers (100-200 children less repeating) the differences are, therefore,

not statistically significant.

(32) Another indicator for education quality is the Pupil:Teacher ratio (PTR). In the former Yugoslav

Republic of Macedonia and Serbia, PTRs have been improving especially between 2000 and

2003. Armenia maintained an already good PTR, but the PTR is varied between sparsely

populated regions and Yerevan (up to 40% difference between the lowest region and Yerevan).

The declining number of students especially in village schools results in low PTR (even below

efficiency levels). Average PTRs, therefore, may hide problems of teachers shortages and/or

teacher deployment.

Percentage of repeaters in primary education (% of all students enrolled

Data source: TransMonEE

-

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Armenia

The former Yugoslav


Serbia



Pupil:Teacher Ratio in primary education :

Data source: TransMonEE

(33) In the former Yugoslav Republic of Macedonia and in Serbia, national data show significant

progress regarding (i) the reduction of early school leavers in primary and secondary education;

and (ii) the improvement of the completion rate in primary and lower secondary education.

While these data are not directly comparable to international data, they are nevertheless a good

indicator for the improvement of educational quality.

Former Yugoslav Republic of Macedonia, early school

leavers in primary and secondary education

Serbia, completion rate in primary and lower

secondary education

Data source: Gender statistics indicators, the former

Yugoslav Republic of Macedonia - State Statistical Office

Data source:

Statistical Yearbook of the Republic of Serbia, 2013

(34) Since 1995, UNICEF promotes a CfS framework, a holistic initiative providing a

comprehensive range of quality interventions in education. The CfS approach was

conceptualised by the end of 2001. Curriculum frameworks in all visited countries and

territories now incorporate CfS related elements. Country programmes and strategic plans

acknowledge that these concepts need to be mainstreamed and, thus, reflected in teacher

education and training in order to guarantee that children benefit from a learner-centred and

competency-approach in the classroom.

10.0

12.0

14.0

16.0

18.0

20.0

22.0

24.0

Armenia

The former Yugoslav


Serbia

0

100

200

300

400

500

600

700

800

900

2002 2003 2004 2005 2006 2007 2008

Early school leavers in primary and secondary education

90

91

92

93

94

95

96

97

2008/2009 2009/2010 2010/2011 2011/2012

Completion rate, rate, primary and lower secondary education



In the former Yugoslav Republic of Macedonia, the National Strategy “Steps Towards Integrated Education”

(D311) acknowledges that “school curricula are an essential element in the development of a strategic approach

towards an integrated education system”, and that “any references leading to negative stereotyping and

intolerance will have to be removed; a critical way of presenting subjects will have to be introduced in order to

encourage analytical thinking” (“Country Programme Action Plan 2010-2015 between the Government of the

Former Yugoslav Republic of Macedonia and United Nations Children’s Fund”, D098). As a result of the

UNICEF supported intervention in the area of curriculum development, a baseline study on multiculturalism and

inter-ethnic relations in education was undertaken in 2009) This will allow monitoring at the impact level,

regarding a change in stereotypes and prejudice among students and teachers

The Kosovo (UNSCR 1244) Curriculum Framework (KCF) focuses on learner-centredness, competencies,

integrated teaching and learning, flexibility, mobility and transparency. It sets out the vision for developing and

implementing a learner-centred and competency-based curriculum in Kosovo (UNSCR 1244) integrating and

reflecting the fundamental values and principles of human rights, living together, social justice and inclusiveness.

The KCF also requires alignment with teacher education and training, assessment and school and classroom

management. In this respect, teachers will need upgrading and training. A teacher licensing system has been

developed to ensure more coherent professional and career development and is linked to the teacher performance

evaluation practices and mechanisms that serve to improve quality of teaching and learning in classrooms.

(source: D075)

(35) By accepting the holistic approach focusing on all aspects of quality education from a

structural perspective and within a school environment, Governments have generally been

successful in ensuring that the majority of marginalised children may progressively benefit from

inclusive education. Key interventions include (i) awareness raising and stimulating measures,

(ii) reviews of curricula and textbooks, (iii) extending opportunities for all ethnic communities

for mother tongue education, (iv)and publishing of pedagogical documents in community

languages. Other measures include the introduction of pedagogical assistants to facilitate the

integration of SEN children in mainstream education, and school-community mediators

bridging the gap between school and home culture for Roma children.

(36) Active learning in the classroom has not only been a focal area of UNICEF support over the

past 10 years (initially through specific Active Learning programmes, later through the CfS

initiative). It plays a central part when strengthening inclusion of learners in the teaching and

learning process. However, document study and interviews confirmed that implementation of

inclusive, quality education at classroom and school level is hampered by weak competence of

teachers in regular schools on how to relate to marginalised children in an enabling way. The

development and use of monitoring and feedback systems on the impact of teacher training on

the quality of learning of all children, together with building school-parent cooperation and

local support networks to influence beliefs of all stakeholders have a positive impact on making

education more inclusive and the improvement of the quality of teaching and learning.

In the former Yugoslav Republic of Macedonia, an Inter-Active Learning pilot initiative (concluded 2003)

developed into a network of 75 schools, creating a new learning environment, developing children’s skills (from

being passive learners to active problem solvers), and gearing classroom teaching to meet the individual needs

of the children. UNICEF and the Ministry of Education and Science are currently expanding this initiative across

schools nationwide.

In Serbia, in order to provide continuous support to teachers and schools in implementation of inclusive

education, the Ministry of Education, Science and Technological Development (MoESTD) has established the

Network for Support to Inclusive Education (NIE) in 2010, within a World Bank supported project. This innovative

mechanism comprises more than 120 practitioners and experts on inclusive education working in mainstream

education and NGOs, covering practically the whole territory of Serbia. There are 14 schools of good inclusive

practice in Serbia which are initiating local actions, creating local inclusive networks and hosting study visits for

other schools. NIE members work with children, teachers and parents with the aim (i) to advance inclusive

practice; (ii) to provide continuous professional support and consultations for achieving inclusive education; (ii)



to provide coaching, consulting and supervision support to education institutions and individuals (teachers,

parents and students in particular). From 2012 it is supported by UNICEF and KulturKontakt Austria. The

network is expanding.

In Turkey, models for decentralised, school-level decision-making to support the implementation of inclusive

education were established and although changing government priorities impact on their sustainability, the

school-based approach to inclusive education has not been changed.

(37) While there is a significant number of teachers who apply child-centred active learning

methodologies (above all novice teachers), inclusion in terms of equity in learning opportunity,

still remains problematic. According to the focus group discussions, teachers do not prefer to

employ the active learning principles due to several reasons as follows: lack of training and

their limited knowledge on practical manner; an overweight curriculum; time limits; large class

sizes and classrooms; lack of an assistant; lack of classroom equipment and practical

demonstration of knowledge. Especially, standard national tests lead teachers to use traditional

methods of instruction. Many teachers do make more use of active learning strategies when the

class sizes are small. However, question and answer methods are still dominant in most classes

observed.60

(38) Data for Turkey provide further support towards the evidence of increasing literacy rates, as

shown in the charts below.

Turkey, youth literacy rate, population 15-24 years,

both genders

Turkey, share of 15-year-old pupils who are at level 1

or below of the PISA combined reading literacy scale

Data source: UIS Data source: eurostat

(39) Serbia and Turkey participated in previous PISA examinations (2003, 2006, 2009 and 2012).

The PISA performance scores show a general improvement over time for both countries from

2003 to 2012.

(40) In particular, the relative improvement over time is stronger for girls. In Reading, girls

remarkably perform above average. In terms of impact on the access level, the PISA results

indicate that the gender parity shifted more in favour of the girls.

60 In order to assess the degree of active involvement of learners in the teaching/learning process, an adapted

version of the Flanders Interaction Analysis technique was included in the classroom observation sheets (cf

Appendix 5). Patterns of teacher and student interaction are highly interdependent, whereby the implementation

of student-centred methodology strongly encourages active student participation.

95.0

95.5

96.0

96.5

97.0

97.5

98.0

98.5

99.0

2004 2005 2006 2007 2008 2009 2010 2011

Youth literacy rate, population 15-24 years, both sexes (%)

20.0

22.0

24.0

26.0

28.0

30.0

32.0

34.0

36.0

38.0

40.0

2003 2006 2009

Share of 15-year-old pupils who are at level 1 or below of the PISA combined reading literacy scale (Turkey)



PISA scores on Science (all) PISA scores on Science (females)

PISA scores on Mathematics (all)

PISA scores on Mathematics (females)

PISA scores on Reading (all)

PISA scores on Reading (females)

In Armenia, 60% of all early grade teachers have been trained on improved techniques, in line with international

good practices for teaching numeracy, and 40% have received similar training on teaching skills to improve

literacy. Support to improving teaching methodology in literacy and numeracy resulted in concrete improvement

in students’ outcome. The preliminary results of a longitudinal study in schools whose teachers have been trained

in early numeracy show improvement in teachers’ knowledge to apply new techniques (from 34% in 2009 to 49%

in 2012) and student outcomes (from 38% to 58% per cent respectively). (source: D044)

5.1.3 Equality in learning opportunity

(41) Whereas section 5.1.1 demonstrated achievements in terms of physical access, particularly the

reduction of the number of out-of-school children, this section reports on the degree to which

learners are able to equally benefit from being in school. This is of particular importance when

considering a qualitative inclusion of marginalised children, which requires active participation

and not just the mere (passive) presence in the classroom.



(42) In connection with the assessment of policy reforms, it has been found that stereotyping and

intolerance towards ethnic minorities is being successfully addressed at least at the political

and legislative level. However, there remain challenges in translating such legislative

parameters into direct pedagogical practice since this directly touches upon critical mindset

issues. Interviews with NGOs representing minority groups and with stakeholders at school and

community level confirmed fierce resistance from local, school and community level

stakeholders resulting in discrimination and segregation.

(43) The following charts analyse specific types of teacher and student interaction patterns, whereby

particular interest is given to those patterns which are conducive towards actively including the

learners in the teaching and learning process. For example, seeking active dialogue with the

learners, involving them and utilising their ideas for further discussion, is much more conducive

to “living learning” than lecturing or asking questions with the only intent that the student has

to provide an expected response. In the charts, conducive behaviours are bolded and shaded in

white; out of all visited countries and territories, only 24.5% of teacher interaction patterns

could be regarded as conducive to active participation of learners.

(44) In a next step, the ratio of conducive to non-conducive interaction patterns is calculated which

ideally should be close to 1.0 or even higher; the higher the value, the more likely the teacher

is to employ student-centred techniques which in turn encourage active participation on the side

of the students.

(45) For the average of all the visited schools in the visited countries and territories, the CBPR is

very low with just 0.34. Although the school visits cannot be regarded as representative, this is

still an indication that the teaching approach still remains very teacher-centred, with the main

elements of interaction used by the teacher referring to asking questions (based on teacher

ideas), giving directions and lecturing.



(46) Likewise, the analysis of student patterns focuses on active participation on the side of the

learner, primarily as expressed through the initiation of interaction (as opposed to merely

responding to a teacher question). The degree of active involvement of learners in the teaching

and learning process can then be expressed as Pupil Initiative Talk Ratios (PITR) as shown in

the charts below. As with the teacher ratios (CBPR), this ideally should be close to 1.0 or even

higher; the higher the value, the more likely the learner is to actively participate in the

teaching/learning process. Again, for the average of all the visited schools in the visited

countries and territories, the PITR is very low with just 0.35; if the rather large number of off-

task activities is considered, the PITR even drops to just 0.25.

(47) According to information generated through interviews and FGDs, girls seem to have benefitted

to such a degree that they, at least in the visited countries and territories, are no longer

structurally excluded from the teaching and learning process. In terms of learning achievements,

COs and Ministries alike report gender equality in enrolment terms, and even that girls are

outperforming boys qualitatively in their learning achievements. Nevertheless, one needs to be

aware that in terms of sensitisation (as a key mechanism in reducing equity gaps at a

perceptional level) there are structural limits of which a sensitisation campaign such as the

Girls’ Education Campaign in Turkey can achieve.

The Turkish Ministry of National Education, supported by UNICEF and other development partners, has

addressed causes of low enrolment, attendance and transition of girls in between 2003-2010, through:

The establishment of conditional cash transfer (World Bank, 2003) to facilitate participation in

education for girls from economically disadvantaged families, resulting in an increase in girls’

enrolment in 2005;

The development of a catch-up system to address the problem of over-aged girls and boys (UNICEF

2005-2007), and addressing the late enrolment of girls;

Institutionalising the Girls’ Education Campaign (UNICEF GEC, 2001-2010) through strengthening

monitoring mechanisms for out of school children and by making local authorities more accountable

for addressing the issue through provincial report cards.



(48) Although the increase in percentage of Roma children in mainstream classes is satisfactory (see

chapter 5.1.1.1), children from ethnic minorities do not yet benefit equally from being in school.

(49) While teachers are using strategies of remedial teaching for Roma children, such as a student

coaching systems, or working with children individually, there are also teachers who believe

that a specific curriculum should be implemented for Roma children. According to them,

children from ethnic minorities cannot reach the same level as other students, even if they

participate in special programmes.

(50) When it comes to providing SEN children with equal learning opportunities, children with

mental disabilities in the mainstream classroom seem to be the most problematic group as far

as teachers’ abilities to address their needs are concerned, particularly regarding children with

more severe and/or multiple impairments. Interviews still confirm resistance against inclusion

of SEN children.

(51) SEN children with hearing, visual, and deaf-mute impairments are more (easily) included in

learning activities than children with mental disabilities. Examples of integration are teachers

adjusting the seating more in front, providing different ways to express knowledge, offering

opportunities to reply through art/drawings. Interviews confirm that children with mental

disabilities, especially severe mental disabilities, are perceived as more difficult to integrate in

mainstream education than children with physical impairments.

(52) On the positive side, both parents from ethnic minorities and with SEN children participating

in the focus group discussions in several of the visited countries and territories stated that

teachers did not show discriminative attitudes and behaviours towards their children.61 On the

contrary, there were examples of integration in Kosovo (UNSCR 1244) and Serbia mainly

relying on teachers goodwill and persistence to assist.

(53) At the other side of the spectrum, classes have been visited where teachers demonstrated

inclusive education in practice, i.e., (i) no discriminatory language, and (ii) SEN children

benefiting from active learning, promoting their intellectual and physical development.

Stakeholders from these example schools confirm that mainstream education provides an

environment for marginalised children which is more conducive to learning and development

than special, segregated environments. Condition is that teachers feel responsible and there is a

will to integrate all. Even these stakeholders, however, see severe barriers regarding successful

application of inclusive education.

(54) In summary, Governments have been successful in ensuring that marginalised children may

benefit from inclusive education by accepting a holistic approach focusing on all aspects of

quality education from a structural perspective and within a school environment. Key

interventions include (i) awareness raising and stimulating measures, (ii) reviews of curricula

and textbooks, (iii) extending opportunities for all ethnic communities for mother tongue

education, (iv) publishing of pedagogical documents in community languages, (v) the

introduction of pedagogical assistants to facilitate the integration of SEN children in

mainstream education, and (vi) school-community mediators bridging the gap between school

and home culture for Roma children.

(55) However, interventions have not always led to inclusion in terms of equity in learning

opportunity. Normative frameworks and education standards have been developed but need to

61 Of course, discussions were held with parents (from ethnic minorities and with SEN children) whose children

are in school. The opinion of parents whose children are not in school might be different.



be made operational to ensure equal benefits for all children promoting respect for the

multicultural and multi-ethnic nature of society. Education may be inclusive in terms of

physical access, it is not yet inclusive in terms of ensuring continuous development of children

regardless their ability, background, ethnicity.

(D103) “Government programmes and the legal and policy framework serving to advance inter-ethnic

communication, so far focused on the prohibition of discrimination rather than the positive promotion of

multicultural principles. No obligation is imposed to ensure the implementation of these principles and no

mechanisms are specified by which to stimulate respect or sanction disrespect of these principles at national

and school level”.

5.2 Effectiveness of systemic changes for the most vulnerable children

(56) Over the past 10 years, UNICEF in the CEE/CIS region has adopted a systemic approach to

programming. Programming evolved from small interventions to integrated programmes

contributing to system changes. Changes at the system level were deemed necessary to create

a more enabling environment, conducive to increased equitable access to and equal

participation in education of improved quality.

(57) One of the most important system changes in the last decade in the region is the extension of

compulsory education for one year, through introduction of preparatory pre-school

programmes. This has generated direct systemic impacts regarding drop-out prevention, length

of education, and an enhanced focus on quality issues. As part of their activities, UNICEF

Country Offices have contributed significantly towards such implementation and the related

efforts to include vulnerable groups into the process.

(58) As could be observed in all visited countries and territories, changes in legislation and policy

have created the foundation for an enabling environment towards greater inclusion. As a result

the number of marginalised children in mainstream schools increased, the number of out-of-

school children decreased, parity of enrolment and attendance between boys and girls in basic

education, improved PISA performance (Turkey and Serbia), and an increase in the

participation in preparatory pre-school programmes. From a quantitative point of view, all

visited countries and territories confirm that their most vulnerable children now gain on the

degree of their representation in the education systems.

(59) Even if the practical transfer – regarding translation of policies into educational practice – must

still be regarded as “work in progress”, the mere existence of legislation which supports

previously excluded population groups is an important step in changing social norms: the

previously “excluded” have now become empowered to directly refer to existing legislation in

case of their rights not being heard and respected.

(60) Stereotyping and intolerance towards ethnic minorities has been addressed in all visited

countries and territories, at least at the political and legislative level.

(61) While laws, policies and strategies are important milestones and foundations for future work,

they are in the first instance declarations of intent and as such require further operationalisation.

For example, Armenia uses the term “transformation” when referring to the changing role from

former “special schools” (as spaces of exclusion for children with SEN) to “resource and

assessment centres”. This is indeed a transformatory process which will ultimately impact on a

systemic understanding. Likewise, any law, policy or strategy needs to transform education into

a quality learning experience for all.



(62) The translation of existing legislation into pedagogical and social practice is still in its infancy

in most of visited countries and territories. For example, in Armenia a conceptual

misunderstanding could be observed, resulting in equalling “special needs” with “disabilities”

(thus neglecting “special needs” on the basis of children coming from poor rural areas, children

performing below academic standards). This however precludes the achievement of “inclusion”

in terms of children equally benefiting from being in school.

(63) Key reasons why system changes have not yet been fully successful in achieving inclusive

education are (i) weaknesses in education planning and management at all levels, including

teachers; (ii) weaknesses in decentralised decision-making, (iii) lack of multi-sector

cooperation, (iv) education systems not operating in an integrated way, and (v) systems of

education financing not explicitly supporting inclusive education.

(64) Education planning is not in all visited countries and territories based on an analysis of reliable

and accurate data. Although data collection systems exist in most of the visited countries and

territories, not all are used for informed and transparent decision-making and/or information

sharing. National and local level capacity to manage and monitor implementation of policies

and programmes remains weak and countries and territories lack effective quality assurance

systems to monitor inclusiveness of education.

(65) In some of the visited countries and territories, decision-making is centralised with weak

communication between national, local (municipalities) and school levels. In one country,

schools were apparently not informed on measures taken by the Ministry to foster inclusion. In

other visited countries and territories the transfer of authority from central to Municipal level

resulted in school management being politicised (i.e. Kosovo [UNSCR 1244], where the

municipal education directors are politically appointed and are accountable to the mayor, not to

the school director/management).

(66) In most of the visited countries and territories, key education institutions such as (i) Ministries

of Education, (ii) in-service and pre-service teacher training institutes, (iii) Faculties of

Education, and (iv)schools function in isolation. There is hardly any feedback on teacher trainee

performance during apprenticeship. In contrast, the achievement of inclusive education requires

a holistic approach, and education systems which parts are interconnected with, among others,

institutes for social development. Likewise, programmes of initial teacher education have not

always been transformed to support inclusive education, especially regarding subject teachers

(higher grades of primary and secondary schools). This lack of competencies is then sometimes

substituted by extensive in-service trainings.62

(67) Nevertheless, whenever inclusion has become a reality in schools or other pedagogical

environments (e.g. pre-schools, teacher training institutions), this is certainly a process which

is unlikely to be reversed. Achieving inclusive education requires more time to unfold,

especially if a growing acceptance of (and respect for) diversity is coupled with a society-wide

sensitisation and resulting mindset change.

(68) In line with the findings above, the four systems change contributions and its specific

determinants, as reflected in the MoRES framework, show most progress in the areas of (i)

legislation/policy and (ii) access to services, facilities and information. This particularly refers

62 Serbia CO reports, for example, that teachers’ training faculties (1st to 4th grade) went through certain

transformations, resulting in quality improvements in lower grades of primary schools.



to the adequacy of laws and policies (even though there is still a need for their practical

application).63

(69) Consequently, this shows an advantage of systems change contributions related to the areas of

(i) Enabling Environment and (ii) Supply over the areas of (iii) Demand and (iv)Quality. This

can be explained by the fact that determinants related to the “Demand” side include more long-

term oriented changes in individual/community beliefs, behaviours, practices and attitudes,

often based on a more long-term continuity of enabling practices. Likewise, the aspect of

installing quality standards at all system levels is an undertaking which requires considerable

time in order to become systemically observable.

5.3 Relevance of UNICEF's approach

5.3.1 Relevance towards the most vulnerable

(70) Country programmes reflect UNICEF's shift from project to system approach, i.e. the shift from

specific interventions to the introduction of strategic approaches and models and new roles in

supporting system reforms. The shift has thus contributed to the development and amendment

of legislative initiatives in three main areas, i.e. (i) better reflecting principles of inclusive

education, (ii) ensuring access of marginalised children to mainstream schools, and (iii)

protecting of child rights. Due to the high degree of inclusion-oriented legislation and policies,

all visited countries and territories confirm that children from their minority groups now gain

on the degree of their representation in the education system.

(71) Not all countries’ and territories’ general laws protect the rights of specific groups of children

yet. For instance, some of the countries’ and territories’ labour laws are not applicable to

seasonal working children, and in Armenia “inclusive education” is rather narrowly defined as

“children in need of special conditions for education” – practically referring to disabilities even

though the formal process of their certification is not connected to disability certification.

(72) Whereas changes in legislation and policy have contributed to greater equity in physical access,

the approach was less relevant in addressing bottlenecks which hamper equity in learning

opportunity. Mechanisms of formal classroom and method-oriented education that cause

selection, categorisation and exclusion, in particularly of minority children, have not yet been

addressed.

(73) Although participation, completion and transition rates of Roma children in mainstream

education increased, there is an observable increase in the number of majority parents taking

out their child from mixed-ethnicity schools and enrol them in majority schools, a trend

minority parents seem to be in favour of since they also tend to prefer their children to be in a

single-ethnicity school.

63 Appendix 4b provides an overall rating for all 10 determinants. The two areas with most progress (i.e. (i)

legislation/policy and (ii) access to services, facilities and information) are colour-coded green, based on a 3-level

“traffic light” system, as agreed following the pilot phase in Kosovo (UNSCR 1244). The remaining areas received

a “yellow“ score, indicating ongoing work in progress. Since no determinant has been found to be unsatisfactory

(which would have been indicated by a “red” score), the effectiveness of systemic changes can also be described

as being generally satisfactory, with the understanding that more time is required for most interventions to show

their impact. The scores were assigned jointly by all evaluators on the basis of their own judgement, referring to

information and data collected through the completed evaluation matrices and the interviews conducted in the

field.



(74) However, as could be observed in all visited countries and territories, changes in legislation and

policy have created important foundations for an enabling environment towards greater

inclusion. Even if the practical transfer – regarding translation into educational practice – must

still be regarded as “work in progress”, the mere existence of legislation which supports

previously excluded population groups is an important step in changing social norms: the

previously “excluded” have now become empowered to directly refer to existing legislation in

case of their rights not being heard and respected.

(75) Equity in developmental and learning opportunities can ultimately only grow if there is space

for encounter of children with and without special needs, of children coming from various

background and ethnicities. As such, UNICEF’s approach has been relevant for preparing the

ground in the respective countries and territories towards a more inclusive environment, as now

reflected in all policy and legislative frameworks. In order to unfold relevance even further,

“respect” needs to grow among the stakeholders and final beneficiaries, the learners in schools

and (ultimately) universities.

5.3.2 Relevance towards prevailing education sector policies and international standards

(76) UNICEF country programmes are generally based on national policy and budgetary

frameworks, during which they address the rights of children, women and young people, often

outlining social and economic policies benefiting vulnerable groups. UNICEF programme

designs have been guided by the principles of the Convention on the Rights of the Child (CRC),

the Millennium Development Goals, and the goals of A World Fit for Children, and the growing

focus on the equity agenda. Programme goals, expected results, and strategies are framed within

the five focus areas of the UNICEF medium-term strategic plan (2006-2013) and use identical

or similar progress indicators.

(77) UNICEF support aims at completing outstanding issues related to the Millennium Development

Goals, with focused attention on inclusion of marginalised groups of children. Country

programmes show references to the 2008 Concluding Observations of the Committee on the

Rights of the Child, and the 2007 Concluding Observations of the Committee on the

Elimination of Discrimination against Women (CEDAW).

5.3.3 Relevance towards the work of national partners

(78) As Governments pursue reforms of their basic social services systems to ensure access to

quality services for all, they have sought the continued assistance of UNICEF. UNICEF thus

supports Governments to: (a) align their regulatory frameworks with reformed laws and

policies, in line with international standards; (b) model implementation of key provisions of

new legislation in order to develop lower-level procedures and regulations; and (c) generate

knowledge and evidence on children and adolescents.

(79) Inter-organisational cooperation between Ministries of Education and UNICEF is perceived to

having been done in harmony and close cooperation between ministries and UNICEF, as

established by the perceptional analysis, and as shown in the charts below:



Consideration of national partner issues in UNICEF programming (MoE perspective)

(80) Ministries and UNICEF (together with other DPs) agree in their perceptions that (i) cooperation

and support has been instrumental for mutual strategic planning with a view to inclusion; (ii)

capacity building supported the realisation mutual objectives; (iii) :partnerships contributed to

reducing social and educational inequalities; and (iv) key inclusive education strategies are

mutually shared between Ministries and UNICEF (together with other DPs).

Consideration of national partner issues in UNICEF programming (UNICEF CO perspective)

(81) Perceptions regarding the relevance of joint strategies between Ministries and UNICEF/other

DPs are even more positive on the side of Ministries, with UNICEF being slightly less positive,

although still to a significantly positive degree.



(82) Direct technical assistance from the regional office does not yet play such an important role to

the country offices, but would be considered useful if it can be related to the strengthening of

capacities for providing support to partners in implementing their programmes in line with

UNICEF priorities. This also refers to the building of partnerships for leveraging influence and

funding at the country level.

(83) C4D strategies (as a regional initiative) are perceived by UNICEF CO staff to complement

systemic reforms at the respective countries and territories, also ensuring that services meet the

needs of the most vulnerable and are widely used by them. However, there appears to be lack

of awareness towards how C4D strategies could be utilised in light of the needs of the respective

country offices and their contexts of operation (particularly in the field of inclusion).

5.4 Contribution of UNICEF to system changes through its core roles64 (efficiency)

(84) The perceptional analysis of current perceptions of Policy makers and other Development

Partners shows that UNICEF practices are generally viewed as very efficient in contributing to

system changes, as shown in the chart below. 65

The charts make reference to the score values (“consent values”) of the underlying perceptional questionnaire,

whereby 1.0 is the value for “strongly agree” and 4.0 the value for “strongly disagree” (with 2.0 for “agree”

and 3.0 for “disagree”). Since a score value of 2.5 represents neutrality66, i.e. showing neither agreement nor

disagreement, the charts visualise the degree of deviation from the neutral zone, i.e. the degree of agreement (or

disagreement) with a given statement.

Efficiency of UNICEF practices as assessed by policy makers and DPs

(85) It is interesting to observe that the perceptions of Policy makers and other Development Partners

very much correspond with the perceptions of UNICEF Country Office staff. The only

64 It should be noted that one programme/intervention could generally combine more than one core role. One

example is ‘giving a voice to children’, which is a cross-cutting role that can also be played under other core roles,

particularly since ‘giving a voice to children’ is always a priority for UNICEF and can thus be considered in

almost all interventions. 65 Detailed perceptional data are contained in Appendix 5, with reference to perceptional tools SAS-1 and SAS-

2. 66 Due to the 4-levelled scale of the perceptional questionnaire, it was intentionally not possible for respondents

to select a neutral response; it was rather the intention that respondents clearly show their personal tendency.

1.00 1.50 2.00 2.50 3.00 3.50 4.00

<- AGREE ---------------------------------------- Consent Value ------------------------------------ DISAGREE -->

6 Interaction and dialogue with development partners on inclusive, child-friendly and social norms has influenced the

thinking about the functioning of education systems within my

Department

1.73

11 Information from studies and assessments on reasons for

education exclusion did not change the way MoE/ my Department thinks about education delivery

2.71

19 Through capacity building, I understand why and recognize how children are being excluded from education

2.04

50 The inclusive quality education/CfS concept and approach are not really adequate for the situation my country is in

3.27

Efficiency of UNICEF practices as assessed by Policy Makers and DPs



difference is that CO staff view the impact of “information from studies and assessments on

reasons for education exclusion” higher than the policy makers themselves, although the

perception of a significant contribution is observed by both groups of stakeholders.

Efficiency of UNICEF practices as assessed by UNICEF staff

(86) UNICEF professional staff is subjectively confident regarding their competencies to play an

effective role at country level and influence the development and support of inclusive policies

in the education sector. The self-assessments revealed that staff feel very familiar with the

regional vision for basic education and the regional education strategy, although they would

wish for a stronger role of the Regional Office when it comes to the provision of more external

expertise on how to mainstream inclusive quality education. Existing regional studies are being

considered to be very valuable in having raised their own capacities regarding their

understanding of inclusion.

Perceptional confidence by UNICEF CO staff

(87) According to current perceptions of UNICEF CO staff, UNICEF CO staff does not yet always

act in a multiplier role, i.e. being more accessible for colleagues (and others) in relation to

1.00 1.50 2.00 2.50 3.00 3.50 4.00

<- AGREE ---------------------------------------- Consent Value ------------------------------------ DISAGREE -->

6 Interaction and dialogue with development partners on

inclusive, child-friendly and social norms has influenced the thinking about the functioning of education systems within

MoE (national and/or decentralized levels)

1.73

11 Information from studies and assessments on reasons for education exclusion did not change the way the MoE thinks

about education delivery3.00

19 Regional studies have given me the insight in and

understanding of how children are being excluded from education

2.00


3.24

Efficiency of UNICEF practices as assessed by UNICEF staff



information exchange on how to mainstream inclusive quality education, or how to trigger

system changes.

(88) According to a number of UNICEF CO staff a critical review of the education reality,

particularly at school level, in the individual country is sometimes insufficient. Although

aligning with international policies and approaches is generally a feasible approach, taking over

examples of other countries cannot replace a proper analysis of the respective education

practices.67 In order to overcome this deficit the institutional memory of local partners should

be used more systematically as their experiences can contribute considerably a context-specific

adaptation.

(89) UNICEF has created, in all the visited countries and territories, a solid foundation of mutual

trust which has proven to be a prerequisite for the degree of government acceptance when it

comes to infusing innovative ideas into education systems, and subsequently to integrate these

into legislation. Regarding partnerships with other DPs, UNICEF is in the very fortunate

position to have a strong acceptance at local, national and international levels. Regarding the

latter, the Regional Office certainly plays an important part when it comes to knowledge

generation and particularly knowledge distribution, a fact that was repeatedly stated by all

visited Country Offices. At international level, country- and region-specific achievements can

be brought into further debate, which in the past has also resulted in paradigm shifts towards a

greater acknowledgement of the issue of educational quality (as opposed to quantity) – the

revised M&E framework for the Global Partnership of Education (GPE), of which UNICEF is

a key partner, is a very striking example for this.

(90) Changes in social norms are, obviously, closely related to changes in mindsets, attitudes and

behaviours. These patterns have taken generations to manifest themselves, and it is likely that

it will take at least one generation to achieve changes at a large-scale societal level.

Consultations with a large variety of stakeholders have shown that, while progress could be

observed, there is still resistance towards greater inclusion. Such resistance is not only based

on attitudes, but also on deep-rooted fears that greater inclusion might negatively impact on

own privileges. Apart from intensified sensitisation efforts, such fears could also be overcome

by providing more information on the benefits of inclusion for all, i.e. the “excluded” and the

“included”.

In Kosovo (UNSCR 1244), It was a psychologist/pedagogue self-representing people with disabilities who

advocated that perceiving a child as being disable results in protective behaviour, taking over from the child. As

a consequence, the child will “learn” to define itself as being unable.

The Ministry of Education in Kosovo (UNSCR 1244) recognises that the starting point for inclusion, therefore,

lies in building parents’ and teachers’ trust in the child’s abilities, overcoming their felt need to protect, and their

feelings of pity or fear. The SEN Department is developing a training programme for teachers in regular schools

on how to relate to SEN children in an enabling way. The programme will strengthen teachers’ recognition of the

abilities of disabled children, by questioning assumptions (a disabled child is helpless), and by making emotions

(feeling sorry for a disabled child) conscious.

(91) UNICEF budgetary spending across activities over the past 10 years does not show a unified

picture across the visited countries and territories. The chart below shows the distribution of

67 This seems to be a common – and on short sight very efficient – procedure for many international agencies, as

it can be done much faster than developing new approaches for each country. However, in the long run it decreases

the overall programme efficiency as the transaction costs (i.e. making a concept from one country work in another)

on the operative level are considerably higher.



UNICEF’s contributions in the education sector of the five visited countries and territories,

against its core roles. As can be seen, the distributions vary considerably from country to

country. However, policy advice and technical assistance appear to play a substantial role all

the visited countries and territories, in particular in the former Yugoslav Republic of

Macedonia, Armenia and Turkey.

(92) Modelling is, next to policy advice and technical assistance, the second most well funded core

role, especially in Kosovo (UNSCR 1244), Turkey and Armenia.

(93) It is interesting to note that while policy advice and technical assistance, as well as modelling,

appear to be well funded in all visited countries and territories, Monitoring & Evaluation has

yet received only little funding allocations.

Financial contributions by core roles

(Source: Lists of interventions prepared by Country Offices)

5.4.1 Policy advice and technical assistance

(94) In the case of Armenia, Kosovo (UNSCR 1244), and the former Yugoslav Republic of

Macedonia, UNICEF's long-standing history of involvement started immediately or very

shortly after independence, putting UNICEF in a key position among development partners.

This generally generated a great deal of mutual trust, professional acceptance and reliability

between governmental authorities and UNICEF. Therefore, UNICEF, often in partnership with

other development partners, contributed in all visited countries and territories to significant

changes in policy and legislation, curriculum frameworks, and education planning.

In Serbia interviews with former senior Government staff confirmed, that the trust relationship between

government and UNICEF contributed to the establishment of a parliamentary committee on child rights, and that



the coordinating role of UNICEF among the DPs (as from 2001) paved the way to foster collaboration avoiding

fragmentation of DPs contribution (2008-12). Now there is an ombudsman, a support network for inclusive

education at national level and school support teams at decentralised levels.

(95) A number of legislative initiatives and amendments were adopted in all visited countries and

territories. UNICEF's support of the development of laws, by-laws, strategies and action plans,

contributed to ensuring that key issues of inclusion became reflected in the legislative

foundations of the visited countries and territories.

In Serbia, UNICEF has developed tools to support the implementation of legislation; (i) rapid assessment of

possible bottlenecks in implementing inclusive education basis for the definition of by-laws and other soft

legislation (2010), followed by new laws on primary and secondary schools (adopted in 2013) making the

education legal framework more enabling and child-friendly since these laws incorporate provisions for an

individualised approach to support inclusion, drop-out and violence prevention, and a mandate for an inter-

sectorial approach; (ii) coherent technical assistance and advocacy to ensure that new laws have a systemic,

inclusive approach through, among others, the establishment of various school-based mechanisms (education

care specialists for SEN children), external school support emphasising quality improvement; and (iii) drafting

new roles and responsibilities for inter-sectoral committees (assessing abilities of SEN children).

In Armenia, the Law on Education for Children with Special Educational Needs [2005] is in the process of being

amended68 to better reflect principles of inclusive education and ensure access of children with disabilities to

regular schools. With support from UNICEF, the Law on Domestic Violence was developed and submitted to the

National Assembly of Armenia for further consideration and adoption.

(96) UNICEF has also contributed to the drafting of laws and by-laws, and regulations on

mainstreaming Roma children in country contexts where Roma discrimination is evident (cf

also Section 5.1.1.3).

(97) In Turkey and Serbia, UNICEF advocated for improved policy-making based on evidence

generated through monitoring frameworks designed to prevent dropout and exclusion using CfS

standards, and independent studies. The example of the former Yugoslav Republic of

Macedonia, where UNICEF entered into a close cooperation with the EU on Achieving quality

education through providing inclusiveness and intercultural learning environment, is an

excellent example how policy advice can develop into a multiplier at top-level policy making.

In light of the fact that UNICEF has rather meagre resources to invest in any country, and

compared with the financial investments of other development partners, the outputs regarding

systemic change are certainly relevant and significant.

(98) Nevertheless, drafting and adopting CfS Standards within the legislative framework is not

sufficient since the application of those Standards at the grassroots level also matters. In general,

impact should be enhanced by strengthening national and local level capacities to manage and

monitor the implementation of a high number of complex strategies such as a holistic approach

to CfS.

(99) UNICEF’s support towards materials development for subjects such as Life Skills (the former

Yugoslav Republic of Macedonia, Armenia), provided for an incorporation of such issues into

primary education curricula for grades 1-9. Also, Life-Skills Based Education (LSBE) helps

students developing skills essential and relevant for their daily lives, including skills promoting

respect for the multicultural and multi-ethnic nature of their society.

(100) Local NGOs have served as implementation partners at school level, technically supported by

UNICEF COs. As a result, some 500 teachers have been trained in Life Skills Based Education

68 Amendment to the Law on General Education, expected to be adopted still in October 2013.



(Kosovo [UNSCR 1244]), and teacher training programmes enhancing teacher competencies

in active learning, child-centred approaches, and inclusive education strategies were developed

(Serbia, the former Yugoslav Republic of Macedonia, Armenia). Also, partnerships to increase

participation in educational planning and management were created, as were classes for

children with special needs in primary schools, and multi-ethnic schools. In Kosovo (UNSCR

1244), specific interventions have also contributed to creation of a pool of local expertise,

especially in the area of inclusive education, which are well positioned and substantially

contribute to reforming processes in the country.

(101) In summary, UNICEF’s promotion of inclusion and its technical assistance regarding child-

friendly, non-discriminatory policy formulation and legislation has contributed to national

policies aligned to international standards and conventions, like the CRC or the Salamanca

Statement (Kosovo (UNSCR 1244)).

(102) Strategies, education sector plans, and normative frameworks have been developed with

technical support from UNICEF, and reflect CfS and inclusive education standards and

indicators including the Early Learning and Development Standards. For example, minimum

standards for primary education are now being used as benchmarks for all primary schools in

Turkey, providing a base for information-based planning, monitoring inclusiveness of schools

(including standards for school safety, pupil support, ethics), and quality assurance of

education. UNICEF Turkey continues to support the Ministry in developing, testing and

expanding minimum standards, measurement tools and software for primary education

institutions, and in building capacity at decentralized levels of education in implementing the

standards.

(103) CfS standards have been mainstreamed in the nationally adopted Primary Education Institution

Standards (PEIS) with the support of UNICEF. In cooperation with teacher training faculties,

UNICEF also supported the subsequent development of teacher competencies and the

adjustment of pre- and in-service teacher training (the former Yugoslav Republic of

Macedonia).

(104) In addition to CfS Standards, child rights issues are reflected in key documents such as

Partnership Action Plans [2009] and Progress Reports. Key national priorities now include (i)

the development of and support to an inclusive system of education enabling equitable access

to quality education; (ii) developing non-violent and safe school environments; (iii) developing

and supporting early childhood development services; (iv) and improving the quality of

teaching and learning and promoting quality learning outcomes.

With long-term technical assistance provide by UNICEF CO, the Kosovo (UNSCR 1244) Education Strategic

Plan 2011-2016 has been developed as a six-year plan that links life-long learning and inclusion in education

and is based on a learning model that reaches out to all learners offering equal opportunities and quality

education. (source: D071). The National Action Plan for Children with Special Educational Needs (2007-2015)

has been developed and a new law on financial support for families with children with disabilities has been

adopted. (source: D075)

(105) UNICEF efforts during the past 10 years often commenced with advocacy activities and

intervention models within the scope child-friendly (quality-related) issues, resulting in the

development of new policies. In order to ensure sustainability, there was a clear focus on

strengthening capacities of schools as the smallest unit of education, although this has not

always been linked to the higher policy-making levels. The main strategy emerging from this

development relates to strengthening monitoring and evaluation capacities of educational

Ministries.



In Turkey, the database for identifying and monitoring the children who are out of the school system was

strengthened, and “Provincial Situation Reports on Access to Education” were prepared in order to monitor the

progress at district and province levels. The organisational capacity of the Directorate General of Primary

Education was strengthened in terms of managing the process and using the results of the developed system. The

developed system revealed the areas where research was needed; for example a quantitative and qualitative

survey regarding the issue of late enrolment in primary education has been conducted, and a workshop was held

to develop strategies ensuring timely enrolment (D165). Implementation of the strategy is ongoing, automatic

enrolment of children has been introduced.

(106) The efficiency of the CfS concept in contributing to national system changes was enhanced

once it was coupled with the development of related minimum standards for (primary)

education institutions. National legislative systems then facilitated the integration into the

education system, where the inclusive elements of the CfS framework finally could unfold

practical relevance.

5.4.2 Modelling

(107) Over the past 10 years, UNICEF has been successful in expanding pilot projects and

simultaneously integrating those into national policies. Basically, three periods can be identified

which have relevance to all the visited countries and territories: (i) the period until 2005, where

UNICEF provided support to education reforms and simultaneously infused theoretical

concepts into the education system by means of innovative approaches (e.g. CfS, Active

Learning project); (ii) the period up to around 2009, where outcomes from the pilot projects

became integrated into the legislative systems and thus contributed to providing the foundations

for larger education system changes; (iii) the period from 2010, where existing laws were

further refined and amended.69

(108) UNICEF supported, both technically and financially, the development of concept papers based

on model schools (CfS, quality education). Such model schools feature a holistic concept that

focuses on all aspects of quality education from a structural perspective and within a school

environment. In the early 2000s, such concepts were regularly approved by the relevant

Ministries of Education, and later further supported by UNICEF until the final integration in

into the respective legislative systems. CfS quality standards are now [2011-2013] referred to

in all visited countries and territories’ education sector plans, relevant policy documents and

strategic action plans, although with different localised names and conceptualisations.

(109) National education development programmes implicitly fully adopt the modelling provided by

CfS frameworks, e.g. regarding the element of active learning: “The programmes for initial

teacher training should be dynamic, flexible and in a process of permanent development. They

should prepare future teachers for progressive and proactive up-taking of the professional

commitments”. This is also reflected in the “Global matrix of teacher competencies” (D311,

Annex 2) which shows strong correlations to the CfS framework, particularly stressing the

importance of collaboration with parents.

69 As most clearly observable in Armenia with the pending Amendment of the Law on General Education.



In Kosovo (UNSCR 1244), the process of development of the education component was coordinated by the Kosovo

(UNSCR 1244) Education Centre (KEC) and was characterised by a broad participation of all involved parties:

government, Roma, Ashkali, and Egyptian communities representatives, civil society, political parties,

international organisations engaged in human and children’s rights (OSCE and UNICEF), etc., who all engaged

in continuous efforts for reaching consensus in all key issues. Roma, Ashkali and Egyptian communities and other

international agencies are meaningful and close partners of the Government of Kosovo (UNSCR 1244) ready to

cooperate for drafting and implementation of this important document. The education component of the strategy

for Roma, Ashkali and Egyptian communities complements in general the overall Strategy of Pre-University

Education in Kosovo (UNSCR 1244), but more particularly it supports measures addressing education of

endangered groups, promotion of diversity, equity, and inclusion in education. After the approval of this

component, the Ministry of Education and Science will be the key agency implementing this strategic document

in close partnership with all other involved parties.

(110) With regards to the efficiency of UNICEF’s efforts, modelling and piloting inclusive education

interventions that could be replicated, expanded or scaled up, the country visits have shown that

UNICEF activities in this field have been extremely meaningful. Taking into account that the

investment of rather small amounts of funding resulted in the infusion and subsequent

integration of IE strategies into the legislative systems, this is one of the most efficient core

roles UNICEF has been engaged in.

The former Yugoslav Republic and Macedonia and Armenia are examples where modelling and subsequent

mainstreaming (with regard to the CfS approach), particularly when coupled with capacity building at school

level, laid an important foundation for later integration of piloting experiences into policy making. Also, due to

the broad framework of “child-friendliness”, this opened up opportunities to address issues such as support to

multiculturalism, inter-ethnic relations and the reduction of school violence.

In Armenia, another example could be found (D043) where the revision of the TransMonEE database was piloted

with the aim of introducing a new focus on equity and on consolidated data not provided by other sources, thus

enhancing its use among UNICEF offices and other entities.

UNICEF Turkey supported the development and use of an on-line absence management model aimed at

monitoring children at risk of dropping out. Functional definitions of absenteeism have been made, risk

assessment forms prepared, and action plans developed. Schools have also been provided with practical

suggestions and activities on how to tackle non-attendance, once it's monitored. The model has been launched in

2011.

In Kosovo (UNSCR 1244), model community-based early child education centres were instrumental in

demonstrating how new pre-school laws translated into practice, as was the creation of drop-out prevention and

response teams at school and municipality level to demonstrate how the national prevention plan on drop-outs

worked in practice. Both pilot initiatives generated national level support to implement new legislation and

programmes.

In Serbia, Pedagogical Assistants (PAs) have been introduced in the Serbian education system after 5 years of

modelling as a result of successive and coordinated activities of several international and non-governmental

organisations (Open Society Fund, OSCE, Roma Education Fund, UNICEF). PAs were taken up by the Ministry

of Education and Science in 2009, when the Ministry, based on the existing experience introduced the pedagogical

assistants in the Law on Foundation of the Education System. There are currently 175 PAs working in the 35 pre-

and 140 primary schools in 85 local self-governments in Serbia supporting mainly Roma children. Their main

roles are to(i) extend assistance and additional support to children and students; (ii) to provide assistance to

teachers, psychologists/pedagogues; (iii) to cooperate with parents or caregivers as well as with competent

institutions, organizations, associations and the local self-governments.



5.4.3 Facilitating national dialogue towards child-friendly social norms

(111) Evidence-based advocacy, using independent studies and reliable data, has contributed to an

increased awareness of policy-makers on the exclusion from education of vulnerable children

including girls (Turkey) and Roma (Serbia) and SEN children (Armenia). UNICEF’s support

and contributions to Regional Conferences (such as in Serbia, Roma Decade) and national

dialogue (Turkey, identification of children at risk of non-attending and dropout) has put child-

rights issues on national agendas.

In Serbia, senior Ministry staff mentioned that due to workshops on dropouts organised by both CO and RO on

prevention of children dropping out, dropouts are now a priority area for both MoE and UNICEF. MoE has

incorporated the monitoring of drop-out indicators as a key task of the National Education Council and drop-out

prevention measures in school development plans.

The Ministry of National Education, Turkey has conducted a Gender Review and a study on birth registration,

together with three research studies in collaboration with the Education Reform Initiative (ERI).

In Armenia, interviews with key government staff have confirmed that policy makers are fully committed towards

mainstreaming SEN children, as evidenced by the transformation of previously special schools into resource

centres to provide support to mainstream schools with enrolled SEN children.

(112) UNICEF managed to position children with disabilities at the centre of national level equity

agendas, comprising inclusive education, the promotion of a social model of disabilities and the

advocacy for inter-sectoral cooperation in the provision of services along the life cycle.

(113) UNICEF has built partnerships and dialogues with a variety of official entities, civil

society/private sector actors, and international organisations. EU remains a very important

partner in most of the visited countries and territories, sharing many of UNICEF's values and

financially supporting work on preschool education, quality and inclusive education, justice for

children, violence against children and emergency response.

(114) Through the facilitation of national dialogue, UNICEF cooperates with governments in

increasing the quality of education services and ensuring that the most disadvantaged groups

have equitable access. Within the context of its capacity-building efforts towards achieving

such goals, UNICEF is closely involved in testing and demonstrating innovative changes in

service models.

(115) UNICEF is maximising the impact of its advocacy work by joining forces with professional

associations, NGOs, academics, the EU, the World Bank and others. UNICEF has, for instance,

taken part in the global OOSC Initiative, documenting findings and recommendation to infuse

in policy thinking on how to include out-of-school children in mainstream education. UNICEF

Turkey has built partnerships and entered into dialogues with more and more official entities,

civil society/private sector actors and international organisations. The EU remains a very

important partner sharing many of their values and financially supporting work on pre-school

education, justice for children, violence against children and emergency responses.

In Turkey, UNICEF worked with relevant actors, building a focus on equity and the rights of children and women.

This resulted in an agreement between UNICEF RO and the Ministry of Health, initially to support UNICEF

Humanitarian Action. The CO further helped to maximise Turkey’s contribution to the global post-2015 agenda,

leading country consultations in inequalities and education, supporting the Health consultations and gathering

views of young people and children.



(116) UNICEF efforts to address social norms and demand-side bottlenecks are less developed, and

have focused till date on establishing partnerships with key civil society organisations and

independent institutions. Efforts to influence attitudes, mindsets and behaviour towards

inclusive education have advanced in the areas of children from minority backgrounds, children

with special education needs, in early childhood education, and in the prevention of violence

against children.

In Turkey, UNICEF girls' education and Catch-up Education programmes raised the awareness on the

importance of girls' education and participation in education for all children. The impact on attitudes, mind-set

and behaviour is demonstrated by increased enrolment rates for girls and reduced numbers of over-aged children.

Interviews confirmed that societal change not only requires considerable time, some of the programmes had a

time-span of 10 years, but also local level participation in solving local level problems, i.e. late enrolment, early

school leavers and unequal access to education for girls.

(117) In Serbia, Kosovo (UNSCR 1244) and the former Yugoslav Republic of Macedonia, UNICEF

starts to support the exchange of good practices among local level governments, particularly in

the areas of local budgeting for children, inter-sectoral cooperation, and child participation.

There is a need to strengthen local level capacity in planning, managing and monitoring the

implementation of inclusive education interventions.

5.4.4 Enabling knowledge exchange

(118) Many of UNICEF's efforts to foster horizontal cooperation and exchange of experiences to

enhance child well-being and equity build on and reinforce the achievements of other core roles

in particularly Monitoring and Evaluation and Policy Advice and Technical Assistance.

(119) Exchange of information and knowledge from independent studies has been particularly

efficient in influencing Ministerial decision-making by providing evidence to policy makers on

the importance of inclusive education and modalities of investments [2002-2004].

In the former Yugoslav Republic of Macedonia, enabling knowledge exchange includes numeracy and literacy

assessments, the inclusive education assessment, and the CfS baseline study. The Country Office feels strongly

about the value of these studies to formulate and implement the UNICEF evolving agenda in Education in the

CEE/CIS region.

In Turkey, with UNICEF support, preschool education was significantly strengthened through a new inclusive

curriculum, increased capacity of officials, teachers and inspectors, heightened public awareness and initial

implementation of community-based services for disadvantaged communities.

(120) Research on the importance of addressing developmental delays in early childhood,

deinstitutionalisation of children in care through foster-parenting, and child justice, and

advocacy contributed to fresh policy commitments in critical areas like measuring child well-

being and equity.

In Serbia, UNICEF assisted the Government in drafting legislation regulating support to inclusive education from

the health, social welfare and educational perspective (inter-sectoral approach) to inclusion. Furthermore, based

on UNICEF analysis and advocacy, a steering committee comprising Ministry of Education and Ministry of

Labour staff redefined roles and responsibilities of personal assistants for children with special needs,

contributing to the deinstitutionalisation of the former system for special needs.



(121) Knowledge exchange at local level has a positive impact on belief systems of stakeholders with

a positive effect on making education more inclusive and the improvement of the quality of

teaching and learning.

In Serbia, a Network for Support to Inclusive Education (NIE) has been established to foster horizontal

cooperation and information exchange. Over 120 practitioners and experts on inclusive education, working in

mainstream education and NGOs nationwide, create local inclusive networks and host study visits for schools.

UNICEF is supporting the NIE since 2012.

(122) Regional and country level workshops on the prevention of children dropping out, resulted in

Governments acknowledging the notion of non-attendance and early dropout as issues of

exclusion. In Turkey, a model to monitor and act upon non-attendance and preventing dropouts

of children at risk was set up.

(123) UNICEF country office representatives across the visited countries and territories confirm the

substantial influence of the regional’s office support in enabling knowledge exchange thereby

generating knowledge and developing capacity. They particularly highlight the benefit of study

tours which focused on quality education, self-improvement of schools or physical education.

(124) Furthermore the interviewees refer to studies from the regional office as valuable data sources

for their work. They confirm to use the results from these studies for the development of

intervention strategies and the planning of activities within the individual countries and

territories. In particular, such studies helped to formulate and implement the UNICEF evolving

agenda in Education in the CEE/CIS region.70

(125) To enhance child will-being and equity, governments recognise the need for greater inter-

sectoral coordination if system reforms are to have a tangible impact on children. UNICEF has

been sought to play a convening role in the establishment and strengthening of such

coordination mechanisms, be they related to policy and planning at the national level or service

delivery at the local level. To strengthen commitment to an inclusive civic national identify

with respect to diversity further requires capacity of central and local bodies to facilitate

inclusive problem-solving processes and consensus building around community priorities.

(126) Education systems in the visited countries and territories remain to be largely subject-centred

and geared towards standardised testing against minimum learning standards. In Serbia,

interviewees stated that entry tests for pre-primary education were abolished and that tests are

instead taken after enrolment to determine whether specific support is needed, which indicates

a felt need to group children according to ability. Although experiences with mixed-ability

classes resulted in changes in the curriculum for SEN children, it was not yet understood that

inclusive education would not require a separate SEN curriculum anymore.

(127) Independent studies71 identified and mapped reasons for being out-of-school, key reasons being

(i) poverty coupled with ethnicity in particular mother tongue language not being the language

of instruction. Negotiations based on facts impacted on Government thinking,

70 For example, the systemic approach was promoted and advised by the Education for Some More than Others

study and the CfS standards development regional workshop; the focus on quality education and learning was

promoted through the Learning Achievements in the CEE/CIS Region study and the Developing Standards for

Quality Education study; the focus on equity and cross-sectoral cooperation was facilitated by the Regional

Position Papers on inclusion of Roma children and the Position Paper on inclusion of children with disabilities. 71 E.g. D104, D137, D159.



minority/minority issues became visible and acknowledged resulting in specific measures taken

to increase enrolment and participation in education for minority groups including girls.

5.4.5 Monitoring and evaluation

(128) As mentioned earlier, ethnicity and poverty often go hand-in-hand, and there are problems of

exclusion which are beyond the influence of an education ministry. These more “hidden

problems of exclusion” refer to domestic labour (reason for girls not being enrolled/dropping

out) or seasonal work (reason for minority children not attending school). Through independent

studies, situation analyses, and needs assessments on child labour, i.e. street/working children

and children working in agriculture (seasonal labour), supported by UNICEF and ILO,

awareness of increased inter-organisational cooperation and collaboration among different

ministries and sectors has been raised. In visited countries and territories, it is acknowledged

that social issues, including exclusion, need to be addressed through inter-Ministerial

collaboration.

(129) In Turkey, UNICEF’s independent assessment of the needs of non-attending children-at-risk

led to the development of a monitoring system and programme how to handle non-attendance

through an online monitoring system at school level plus the provision of practical advice to

schools and teachers on how to act (inform the family; pay home visits). Due to policy priority

changes at Ministry level the programme is being revised and will be re-launched in 2014-2015.

(130) Strengthening accountability both inside and outside the formal system is emerging as a

strategic area of focus. UNICEF has been recognised as playing a catalytic role in supporting

the Government to strengthen its quality assurance systems and in brokering partnerships to

reinforce independent monitoring of results for all children.

(131) Internal UNICEF country M&E systems, however, are not always providing steering-relevant

information as the instruments often focus on the activity level and are not taking sufficiently

into account the results level. In order to alleviate this problem, UNICEF assists countries in

collecting and analysing data in order to fill data gaps for monitoring the situation of children

and women through its international household survey initiative, the Multiple Indicator Cluster

Surveys (MICS). However, although the MICS’s certainly provide an important support to

national M&E systems, standard indicators and benchmarks are not always in place for all

projects.

(132) The achievement of quality in education is closely related to a sound process of quality

monitoring and evaluation, which is one of UNICEF’s core roles in cooperation with partners,

and in implementing an agenda of equity and child rights. Again, important steps have been

achieved in this regard, such as a general sensitisation on the importance of quality monitoring,

or even an initial monitoring of perceptions of key stakeholders regarding the respective

educational systems. However, these often isolated monitoring activities need to be further

structured and formally integrated into the M&E systems of the respective countries and

territories, and also need to be expressed as concretely formulated indicators (which requires

further support in varying degrees, especially with regard to capacity building, up to a point

where currently there is absolutely no quality monitoring taking place whatsoever).



Serbia provides a good practice example whereby standards and instruments (for monitoring pre- and primary

school-level inclusiveness) are being developed within the context of a comprehensive monitoring framework for

inclusive education. Such framework aims at facilitating quality monitoring of inclusive education at various

levels, i.e. pupil, classroom, school, local and national level. The Monitoring Framework is developed in

cooperation of the Institute for Psychology, UNICEF and the Social Inclusion and Poverty Reduction Unit of the

Government of the Republic of Serbia.

(133) Within the four levels of planning, programming, implementation, monitoring and analyses of

results, MoRES determinants at Level Four are about monitoring trends in the situation of

children, which aims at validating outcomes and estimating progress towards inclusion. Within

a qualitative understanding of inclusion, UNICEF has a perfect position to continue advocating

for the operationalisation of such a concept, particularly when combining efforts between local,

regional and international levels. The Regional Office has shown over the past 10 years how

overall coordination and advocacy efforts can result in strengthening of UNICEF staff

capacities (and the capacities of their counterparts) at decentralised levels.

5.4.6 Leveraging resources from the public and private sector

(134) Evidence has been provided through a variety of studies to policy makers on the importance of

inclusive education and modalities of investments. Inclusive elements originating from the CfS

model have since been referred to in the education sector plans of the visited countries and

territories, as well as in other relevant policy documents and action plans. The major challenge

remains in the expansion of interventions in order to ensure that the range of disadvantaged

groups is reached.

The Kosovo (UNSCR 1244) Education Sector Plan (KESP) 2011-2016 is an extensive undertaking that accepts

inclusion and life-long learning as the basis of the whole education system as a right and obligation towards its

citizens. The national development imperative is to ensure universal access and that the poor and vulnerable

groups and ethnic population have equal access to basic education, thus supporting opportunities for employment

and participation in economic activities. The right to education is guaranteed for all by the Constitution of

Republic of Kosovo (UNSCR 1244) (April 2008) and other applicable laws, where public institutions ensure equal

opportunities for everyone in accordance with their abilities and needs. Inclusion of minorities is addressed in

the Strategy for the Integration of Roma, Ashkali, and Egyptian Communities in Kosovo (UNSCR 1244) (2007).

(135) UNICEF entertains strategic partnerships with all major international development partners

present in the visited countries and territories, including financial institutions such as the World

Bank, with the aim of ensuring a coherent support to the respective Governments in the

enhancement of strategic reforms.

For example, in Armenia UNICEF provides continuous technical advice on methodological issues, while the WB

generally focuses on sustaining the infrastructural components. In Kosovo (UNSCR 1244), the UNICEF

contributes towards the Kosovo (UNSCR 1244) Education Strategic Plan 2011-2016 together with another 18

Agencies who provide on-going and planned external support towards education subsectors, coordinated by a

Memorandum of Understanding entered into by the Kosovo (UNSCR 1244) Government and all its donors.

In Kosovo (UNSCR 1244), while the WB is supporting (i) the development of a teacher database to facilitate

system performance assessment by MEST, (ii) further development of the EMIS, towards increased accuracy of

EMIS data, (iii) the standardisation of national examinations, and (iv) the development of an item bank (creating

an enabling environment for improved education system performance), UNICEF is coordinating with the WB and

GIZ regarding capacity development of central and local education authorities on data collection, their use for

planning, monitoring and policy making.



(136) UNICEF’s influence on effective donor cooperation becomes the strongest when teaming up

with an important multilateral partner, such as the EU, who provides a significant amount of

funding to the recipient country and simultaneously couples it with substantial disbursements

of funding.

In the former Yugoslav Republic of Macedonia, UNICEF managed to secure funds from the European Union in

support of the programme “Achieving quality education through providing inclusiveness and intercultural

learning environment”. This is an important partnership within the IPA72 Operating Structure and is expected to

be finally approved in 2014. What is of particular importance is that the Operation Identification Sheet73 mentions

a considerable number of quality indicators for inclusion, to which then disbursement tranches have been linked.

In Serbia and the former Yugoslav Republic of Macedonia, UNICEF actively participates in the donor

coordination mechanism - Programme-Based Approach (PBA) - that is established and led by the Government.

More specifically, UNICEF as the co-chair of the working group on human capital in the former Yugoslav

Republic of Macedonia, will ensure assistance to the health, education and social protection sectors compliment

the assistance provided by other donors including the EU, WB, and United States Agency for International

Development (USAID) and the Organisation for Security and cooperation in Europe (OSCE). UNICEF has used

the PBA approach to IPA planning to include a strong focus on quality ECD and education and system reform

and leveraged 3.66 million EUR for this priority (see also above).

In Turkey, UNICEF is increasingly seeking to maximise the impact of its advocacy work by joining forces with

professional associations, NGOs, academics, the EU, the UNCT, the World Bank and others. With respect to the

civil society sector, there is now a regular dialogue and exchange of knowledge. So has a MoU with the Union of

Bar Associations been signed.

Furthermore, partnerships with twenty private media (TV, radio, print and internet portals) were negotiated and

established to support the broadcasting and publication of the schools free of violence PSA and print materials.

The formation of these partnerships was paramount to the successful reach of the campaign achieved at minimal

cost.

(137) Although donor coordination and/or development partner working groups do not exist in all

visited countries and territories, interviewees confirm that the establishment of DP working

groups is beneficial towards aligning DP interventions and Ministerial strategic sector plans.

Such DP working groups could also ensure continuity of alignment and coordination of

development interventions after changes in Government. That continuity of interventions needs

safeguarding against Government changes has become clear after changes in the Governments

of Turkey resulting in the discontinuation of successful programmes, and of Serbia ending a

collaborative process building common understanding and avoiding fragmentation of

interventions.

(138) A holistic approach to the young child and cooperation between sectors benefits the overall

efficiency of cooperation. For example, “catch-up programmes” (Turkey) address the physical,

cognitive and emotional development of the young child, with special reference to the less

fortunate children, within a framework of growing integration. In Serbia, there are also

“Second-chance education programmes” for adults, and although funded through the EU, they

clearly support UNICEF’s efforts within the DP community.

In Turkey, the European Union-funded “Children First” project of 2005-2008 supported much of UNICEF work

in juvenile justice, child protection and education, whereby UNICEF concentrates on coordination, provision of

72 Through the Instrument for Pre-Accession Assistance (IPA), the EU provides assistance to the Candidate

countries to support their progressive compliance with EU rules and policies, in preparation for their accession.

European funding is granted to help countries to fulfil the political, economic and acquis-related criteria for

membership (the Copenhagen Accession Criteria) and to build up administrative and judicial capacity. 73 Pending official approval (expected still in 2013), therefore not yet included in the bibliography.



technical support, facilitating the exchange of experience and monitoring and documentation. Efficiency of

UNICEF's support increased by this multi-sectoral approach relating to early childhood development, child

protection and education.

(139) UNICEF’s concepts are reflected in curriculum frameworks, learning standards, and

frameworks of teacher competencies. Technical assistance has also been provided for the

development of education formula funding defining allocation of resources based on specific

needs of each individual child.

(140) Over the years, the emphasis has been shifted to the operationalisation of legal and regulatory

frameworks through capacity building, modelling, and strengthening monitoring systems.

Local and central authorities have been supported in reshaping existing policies and practices,

and teachers have been trained to implement the inclusive education model to progressively

support regular schools to include children with special needs in regular classes. Although

inclusive education has become a mainstream, overarching policy, the major challenge remains

in the expansion of interventions in order to ensure that the full range of disadvantaged groups

is reached. Further attention needs to be given to advocating the importance of decentralised,

school/community-based approaches, and to promote, support, expand, or sustain pilot models

designed to monitor children at risk, including the development of local monitoring and

informed decision-making capacity.

(141) Despite good relations built between UNICEF and Ministries of Education, some Governments

have recently change their priority negatively impacting on results achieved in terms of

inclusive education, in particularly in terms of intellectual and developmental access to quality

learning.

(142) Taking part in multi-country initiatives and dialogues, and increased collaboration with civil

society actors, UNICEF’s cooperation with international partners continues to place child

poverty and child rights on national agendas and to contribute to partner efforts to reform their

social welfare systems.

In Armenia [2013], UNICEF and the International Labour Organisation, with the full support of the International

Monetary Fund, promoted the Social Protection Floor initiative, as the platform of negotiation of a long-term

social protection programme. Despite progress made in inter-Ministerial dialogue between the Ministry of

Education and the Ministry of Labour and Social Issues, and the increased engagement of donors, it was not

possible in 2012 to define a long-term plan for the development of alternative care services for children, and the

reduction of placements in residential care, which is unanimously considered a priority in Armenia. The direct

engagement of the Minister of Labour seems, however, to indicate promising developments for 2013.

(143) Recent economic development and business environment reforms have placed some of the

visited countries and territories firmly as middle-income countries, with corporations

increasingly embracing corporate social responsibility also in terms of uplifting the quality of

life for disadvantaged population groups. This opens new opportunities for UNICEF to expand

its network of partners to include the private sector.

(144) UNICEF positioned itself as the partner of choice for the World Bank (WB) in providing

continuous technical advice on methodological issues, while the WB focused on sustaining the

infrastructural components.

(145) The documents analysed provide a number of examples on how UNICEF contributed to

leverage partners’ funding for activities focusing on inclusive education.



In Armenia, the adoption of the 2008-2015 Pre-School Strategy and Act on Alternative Pre-school Services

resulted partly from a partnership with the World Bank, based on the positive evaluation of a community-based

low-cost model supported by the country programme (D046).

In the former Yugoslav Republic of Macedonia, UNICEF’s activities were coordinated with USAID in the

implementation of their project on inter-ethnic education. Now USAID supports the expansion of extracurricular

activities based on a model developed by UNICEF. UNICEF and USAID were also working together to jointly

support teacher professional development in competencies needed for effective work in inclusive education.

In Serbia, the implementation of the Schools-without-Violence standards has led to improved reporting of violence

and a greater use of non-violent conflict resolutions (D129). The success of this approach allowed the Ministry

of Education to leverage further support from a World Bank programme.

UNICEF Turkey, in close collaboration with the General Directorate for pre-school education, mobilised EU

resources to strengthen pre-school education and to accelerate access in pre-school education. Standards for pre-

school education have been developed and pre-school curricula are being revised.

(146) In most countries and territories more than one ministry (e.g. MoE, MoL, MoSA) is engaged in

the education sector, which makes it a challenge to coordinate strategies among all relevant

stakeholders, since they do not necessarily pursue the same goals. Accordingly, national plans

need to be coordinated inter-sectorally and inter-ministerially on issues related to inclusive

education.

(147) The interview results show clearly that in all countries and territories major partners show a

high level of awareness about UNICEF’s programme goals and strategies. All interviewees

support UNICEF’s approach of inclusive education and agree with its implementation

strategies. However, it has to be acknowledged that inclusive education requires a change in

mindset of all involved stakeholders, including teachers, parents and peers, not only the

partners.

(148) Due to the number of actors and further influential (e.g. economic, societal, religious) factors it

is somewhat difficult to identify a causal linkage between UNICEF’s efforts and the

governments’ political will and financial resources to address child rights violations and

equity issues. Nevertheless, the document analysis provides some anecdotal evidence that these

efforts somehow contributed to increased funding by local authorities.

(149) UNICEF has taken considerable efforts to create alliances, mobilising partners, donors and

other duty bearers, and triggering national partnerships towards greater equity and

inclusiveness in education. The results of these efforts can be traced both by the document

analysis and the empirical data from the field.

In Armenia, UNICEF has joined with the NGO Bridge of Hope to support the government in developing policies

and practices in several areas such as: the closure of several special schools and the integration of children into

general education; the transformation of the closed special schools into Child Care and Protection Institutions;

the development of Community Centres that support parents and local authorities involved in education; and the

institution of several Inclusive Schools. Although the particular contribution of UNICEF to the formation of these

alliances cannot be assessed on an empirical basis, it can be assumed that due to its wide networking capacities

and the statements of the stakeholders involved in these networks, UNICEF played at least a fostering role if not

being the driving force.

In Turkey, UNICEF through its implementing partners collaborated with the Directorate-General for Basic

Education at the Ministry of National Education in three research projects, one on financial management of

primary education in Turkey, one identifying the basic determinants of attendance and non-attendance in primary

education, and a third one on basic determinants of the transition from primary to secondary education. The

results of these research projects were highly relevant for the policy development of the ministry and accordingly

well received by all stakeholders.



In Serbia, UNICEF partnered with the Institute of Psychology which developed a monitoring framework for

Inclusive Education. UNICEF also established a number of partnerships with NGOs who are important UNICEF

partners for advocating inclusive education. For example, UNICEF has been supporting work of the Network of

Organisations for Children of Serbia (MODS) which was formed in May 2011, and which currently consists of 93

Civil Society Organisations dealing with children and children rights protection. One of the top priorities in their

work is inclusive education.74

(150) It can be confirmed that UNICEF had considerable success in making governments’ reforms

and partners’ priorities more relevant to the realisation of the right of all children to be

included in quality learning in at least four of the five visited countries and territories.

In Armenia, UNICEF has had a central role in promoting Inclusive Education particularly with regards to the

2010-2015 Country Programme (D054). There in UNICEF clearly outlined the needs in the areas of rights of

women and children, child nutrition, HIV awareness, inequalities in pre-school, upper secondary and quality of

education, meeting the obligations of the CRC, and institutionalization. The Country Programme was apparently

recognised well by the government and accordingly led to a substantial reduction of the number of Boarding and

Special Schools and the reintegration of children in their original families and communities.

In Kosovo (UNSCR 1244), UNICEF implemented a teacher training programme for introducing innovative ways

of teaching and learning emphasising qualitative interaction and learning (D244). The curriculum, textbooks and

methodology were assessed so positively by the participants that the MEST apparently recognised the value and

importance of the community approach to early child education for rural communities. It finally encouraged steps

toward mainstreaming of the introduced innovative education practices in the entire territory.

In the former Yugoslav Republic of Macedonia, a Joint Programme on enhancing inter-ethnic dialogue and

collaboration has been set up (D105). The JP focused particularly on enhancing the capacity of central and local

bodies to facilitate inclusive problem-solving processes and consensus-building around community priorities and

to strengthen the commitment to an inclusive civic national identity with respect to diversity. This was the first

more comprehensive project in this area which informed the subsequent interventions by the Government and

partners in this area.

In Serbia, UNICEF together with NGOs supported schools to include Roma children, provided support to their

parents, and supported the development of active learning modules adjusted to the needs of Roma and other

vulnerable children. These modules are now incorporated in the Ministry’s/general in-service teacher training

programmes.

(151) It is difficult to compare UNICEF activities directly to those of other development partners,

particularly since UNICEF does not make financial contributions in volumes which would be

in any way comparable to those of, inter alia, the European Union, World Bank or other

development banks. Furthermore, as has already been stated, it will not be possible to

differentiate between contribution and attribution, i.e. to state which investment by UNICEF

exactly led to which result.

(152) Nevertheless, it can be stated that UNICEF’s investments are relatively low when compared to

those of other DPs. Due to UNICEF’s focus on infusion and integration of concepts into the

system, coupled with advocacy and policy advice, the need for capital inputs are rather minor,

yet very effective when considering the ultimate impact at system level (e.g. through adoption

of innovative policies, and/or change in perceptions towards UNICEF-specific issues, such as

inclusion in this case).

(153) The most adequate strategy and role to play for UNICEF depends on the country contexts.

However, in light of the relatively small capital investments made by UNICEF is hard to

74 Cf http://zadecu.org/?page_id=1360&lang=en .

http://zadecu.org/?page_id=1360&lang=en



imagine how UNICEF could operate in an even more cost-effective way then it is doing right

now, considering the system changes already (co-) initiated or even fulfilled in the visited

countries and territories.

5.4.7 Giving children and adolescents a voice

(154) Country Offices advocate for child rights with audiences such as Parliament, politicians, high-

level policy-making bodies, political parties and/or the general public. UNICEF was also an

important voice for children, and amplified the voices of children and youth – e.g. in the

Constitution debate and drafting of the national development plan and national child rights and

youth strategies. In some of the target countries and territories, among which Serbia, UNICEF

has supported the development of a social accountability for children's rights programme

aiming to strengthen the capacities of and cooperation among the Ombudsman, the

Parliamentary Committee for the Rights of the Child, and the National Council on Child Rights

to promote and monitor implementation of the Convention on the Rights of the Child.

(155) Campaigns, regional conferences, and evidence-based advocacy have made the issues of

disparity clearly visible75, particularly those focusing on (i) Roma children (Serbia, campaign

for the promotion of enrolment of Roma children in primary education), (ii) on children with

special education needs (Armenia, campaign for social inclusion of children with disabilities),

and (iii) girls' education campaigns which were instrumental to break the silence about girls'

exclusion from education (Turkey).

(156) Other examples of giving children a voice include (i) UNICEF's campaigns on birth registration

of Roma resulting in schools accepting enrolment of Roma children without birth certificate

followed by support to parents in the birth registration process (Kosovo (UNSCR 1244) North,

Serbia), and (ii) a conference on inclusion of SEN children in mainstream education, where

students of an autistic classmate made a plea for inclusive education based on their positive

experiences with inclusion (Serbia).

Interviewees such as self-representing Roma or people with disabilities were able to explain the problem from the

inside out, thereby giving a unique perspective to the problem of exclusion. They also showed beyond doubt the

changes inclusion had over their lifespan. An example from the field illustrates the process of UNICEF strategies

showing impact, also in terms of the key “stages”/core roles: A pedagogue/psychologist from one of the visited

countries and territories changed her self-perception from being disabled to being able, therefore she enrolled in

mainstream education up to University level [Policy and Legislation/equitable access], now influences teacher

training programmes for inclusive education [Facilitating national dialogue] and contributes to programme

development [Enabling knowledge exchange].

5.5 Sustainability of results for children and system changes

(157) In order to assess the sustainability of UNICEF's support it is necessary to put the achieved

results in a broader socio-economic context. Therefore the following figures provide an

overview about the development of the visited countries and territories during the observation

period:

(158) The first graph shows the development of the GNI per capita between 2000 and 2012 for four

of the five visited countries and territories. The trajectories indicate a largely consistent

development for all countries and territories, whereby Armenia, which also has the highest

relative increase, suffered above-average from the global financial crisis in 2008. On the other

75 However, it needs to be noted that it is difficult to attribute successes in giving children a voice to specific

activities.



hand the former Yugoslav Republic of Macedonia appeared to have been affected less

negatively by the crisis. Although the economic conditions might not be the only external

influential factor on the education sector, its development and particular the improvements

discussed in this report have also to be seen under the light of these general improvements, e.g.

with regards to the availability of government budget for education.

GNI per capita, PPP (current international USD) (Source: The World Bank, World Development Indicators)76

GDP growth (annual %) (Source: The World Bank, World Development Indicators)

(159) The figure above now shows the annual GDP growth of all five visited countries and territories.

Again the relative strong effect of the global financial crisis on Armenia gets visible, while the

former Yugoslav Republic of Macedonia and Kosovo (UNSCR 1244) show only small dips in

their economic growth. This finding allows the assumption that Armenia is less resilient to

external shocks and crises than the other four countries and territories. With regards to the

sustainability of the results of UNICEF’s work this is insofar relevant as it can be assumed that

the long term results might jeopardised by external factors in Armenia more than elsewhere.

76 No data available for Kosovo (UNSCR 1214).

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Armenia

The former Yugoslav Republic of Macedonia

Serbia

Turkey

-20.0

-15.0

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Armenia

Kosovo (UNSCR 1214)

The former Yugoslav Republic of Macedonia

Serbia

Turkey



(160) Despite relatively volatile parameters, the graphs below show a general positive trend over the

past 10 years for Armenia and Serbia for total public expenditure on education as a percentage

of GDP (for all levels of education). Unfortunately the data gaps for Kosovo (UNSCR 1244),

the Former Yugoslav Republic of Macedonia and Turkey are relatively big for a certain

conclusion about the long term trends. However, at least for Kosovo (UNSCR 1244) and Turkey

the trajectories of the last three to five years suggest a considerably positive development as

well.

Total public expenditure on education as % of GDP, for all levels of education (Source: TransMonEE for the former Yugoslav

Republic of Macedonia and Turkey; for Kosovo (UNSCR 1244) the share of the expenditure is calculated on the basis of the

GNI not the GDP.)

(161) UNICEF programming is not always in line with the reality of countries and territories at the

operational level (although UNICEF programming might well match the existing national

policies). The biggest bottleneck to sustainability are therefore the current mindsets of the

respective systems/societies which still marginalise children with special educational needs

and/or children coming from disadvantaged backgrounds, often coupled with a considerable

fear of teachers and professional staff to lose their job due to education becoming inclusive

(meaning that teachers from special schools would become redundant and subsequently lose

their job).

(162) UNICEF’s role within the context of other organisations – both at the government and

development partner level –, is an important one, since UNICEF covers the entire system

spectrum from top policy making down to grassroots involvement in the target countries and

territories. This in turn enhances sustainability of promoted and supported system changes since

these are usually carried by a large majority of stakeholders.

5.6 Cross-cutting issues of a Human Rights-based approach

1.50

2.00

2.50

3.00

3.50

4.00

4.50

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Armenia

Kosovo (UNSCR 1214)*

The former Yugoslav


Serbia

Turkey



(163) The Millennium Declaration, the Millennium Development Goals and the priorities of the

country-specific UNICEF medium-term strategic plans, are generally integrated into the

respective country programmes, clearly making reference to equity issues and human rights.

Country Programmes are additionally informed by the 2008 Concluding Observations of the

Committee on the Rights of the Child, and the 2007 Concluding Observations of the Committee

on the Elimination of Discrimination against Women.

The need for greater coordination among the sectors having responsibility for children has been documented in

two recent evaluations of the work of UNICEF Serbia in the areas of juvenile justice and schools without violence.

The need for greater inter-sectoral coordination has also been recognised by the Government as a critical

shortcoming which must be addressed if system reforms are to have a tangible impact on children.

(164) The Committee on the Rights of the Child in its 2008 Concluding Observations recommended

that Governments take measures to end all violence against children. The 2007 Concluding

Observations of the Committee on the Elimination of Discrimination against Women also

emphasised combating domestic violence as a continuing challenge to be addressed.

(165) Under the cross-cutting issue of Human Rights, UNICEF and its partners generally work to

identify and implement the changes in policies, systems and behaviour necessary to ensure

fulfilment of young people’s needs for life and livelihood skills and opportunities for self-

expression and participation in an inclusive society.

(166) Ensuring the active participation of every young person in public decision-making is important

for democracy, economic growth, disparity reduction and human development. Unemployment,

particularly among youth, is a major challenge in all visited countries and territories. Attention

needs to be paid by governments (and by UNICEF in designing their support and related advice)

to increasing secondary education enrolment and developing and upgrading young people’s

skills.

In October 2012 the precession for the Armenia CRC reporting process was held with the Committee of the Child

Rights in the Geneva. The preparation for the session and the list of issues informed the Mid-term review process

as well as the strategic plan of the Human Rights Ombudsperson's office. Especially highlighted by the Committee

were child protection issues, the need for de-institutionalisation, juvenile justice provisions and services for

children with special needs.

The Kosovo (UNSCR 1244) programme design was guided by the principles of the Convention on the Rights of

the Child, the Millennium Development Goals, and the goals of A World Fit for Children, and the growing focus

on the equity agenda. Programme goals, expected results, and strategies are framed within the five focus areas

of the UNICEF medium-term strategic plan (2006-2013) and use identical or similar progress indicators.

UNICEF Turkey seeks to base all its situation analysis, planning and programme implementation closely on

human rights standards and principles including empowerment of rights holders and their participation in

programming. A number of good practices are therefore fully integrated into office practice. The intermediate

result on child rights monitoring specifically seeks to promote the establishment of effective and sustainable

national child rights monitoring institutions and mechanisms, including the new ombudsperson system. The

Committee on the Rights of the Child considered Turkey's second and third periodic reports in June 2012 and

UNICEF efforts contributed to high-level government participation and to the targeted dissemination of the

Concluding Observations.

(167) In order to gain additional evidence regarding applied HRBA programming in regular activities,

the issue was included as an item for face-to-face interviews with teachers and principals.



Interviews confirmed that currently there is quite a high degree of advocating for inclusion, and

for promoting the inclusion of such principles into legislation and policies. Inclusion is

incorporated widely in legislation. In particular, school principals referred to the attention to

violence in school given by UNICEF.

(168) Further, there is a high degree of awareness among principals and teachers to ensure access to

school by marginalised children, albeit “access” is widely defined as physical/infrastructural

access. The focus is clearly on enrolment.



6 CONCLUSIONS AND RECOMMENDATIONS

6.1 Summary of the context of the visited countries and territories

This section briefly summarises the context of the visited countries and territories in order to put the

overall conclusions into perspective. It is important to note that the visited countries and territories had

very different starting situations, and that progress among the countries and territories need to be

understood against their specific contextual background.

6.1.1 Armenia

Armenia was the first non-Baltic republic which seceded from the Soviet Union, and has been

independent since August 1990 Following official recognition in 1991, Armenia signed the Convention

of the Rights of the Child (1992), adopted a Law on Children in 1996 and has subsequently signed

major international conventions in order to acknowledge and realise the rights of children. Despite

political will, poverty poses a major problem. While poverty rates in urban and rural areas are almost

the same, rural areas are more disadvantaged in terms of access to children with disabilities due to the

majority of inclusive schools being located in urban areas. One of Armenia’s major achievements to

which UNICEF contributed is the steadily growing number of children with disabilities attending

regular schools, and the effectiveness of education programmes which promote social inclusion, also in

line with the objectives of Education for All (EFA).

6.1.2 Kosovo (UNSCR 1244)

Kosovo (UNSCR 1244) is a territory having emerged from conflict which ended in 1999. The education

system had to be designed from scratch, resulting in many parallel focal areas. Government capacities

needed to be strengthened, and enormous challenges related not only to a lack of resources, but also to

security constraints. The peace building process and nation-building have always been an overarching

objective within which UNICEF programming needed to be placed. Here, the CfS initiative played a

very important foundation role. In addition to modelling and infusing innovative pedagogical practices

into the system, Kosovo (UNSCR 1244) pursued a strategy of providing policy advice nearly in parallel

to the piloting of the CfS and the infusion of support towards ECD and pre-school. The peculiarities of

the Kosovo (UNSCR 1244) context required the pursuing of a number of tasks simultaneously, and the

creation of a strong legal and policy framework with regard to non-discrimination and an inclusive

society has been one of the key achievements over the past 10 years.

6.1.3 The former Yugoslav Republic of Macedonia

The former Yugoslav Republic of Macedonia declared independence from Yugoslavia in 1991.

Following a brief war in 2001 between the country’s ethnic Macedonian majority and its Albanian

minority, a power-sharing agreement was reached which awarded ethnic Albanians greater local

autonomy and ultimately resulted in a peaceful and democratic style of government. Currently, the

former Yugoslav Republic of Macedonia is one of five candidate countries for EU accession. In

2006/2007, the former Yugoslav Republic of Macedonia introduced compulsory nine-year education

by extending the cycle to include the pre-primary year. A great deal of attention has since been given

to the education sector, resulting in a high enrolment rate in primary education, a steadily increasing

enrolment rate for children coming from Roma backgrounds, and a continuously improving

achievement rate in literacy and numeracy. UNICEF provided specific technical support in this process,

particularly since teachers felt overwhelmed with education reforms, and with the growing focus on

inclusion and educational quality issues.



6.1.4 Serbia

A former constituent republic of Serbia and Montenegro (1992-2006), the National Assembly of Serbia

declared Serbia to be the legal successor to the former state union in 2006. The Serbian context has

experienced the break-up of the former Yugoslavia and the establishment of a democracy which resulted

in Serbia now being a candidate country on the way to EU accession. The issue of social inclusion is

high up on the Government agenda, as is also reflected by the significant increase in primary school

enrolment of Roma children which has also been supported by UNICEF as one of its key focal areas in

the country.77 An initial piloting/modelling phase at the onset of their interventions ultimately led to

systemic changes through policy advice and reviews of underlying strategies to ensure inclusion of

vulnerable children. In parallel, the improvement of general educational quality has focused on the

enhancement of teacher competencies and of teachers' understanding of child-centred approaches to

teaching and learning. Serbia is an example for a successful early system involvement (through

modelling and piloting) which then allowed for addressing additional important systemic aspects, such

as monitoring and gradually infusing aspects like violence prevention or democratic citizenship/human

rights education.

6.1.5 Turkey

Contrarily to the other visited countries and territories, Turkey is one of the twenty most populous

countries in the world, expected to exceed a population of 80 million by 2015. As Serbia and the former

Yugoslav Republic of Macedonia, Turkey is one of five EU candidate countries for accession. Rural-

urban migration has also contributed to a context within which regional disparities need to be addressed,

particularly regarding school enrolment and gender parity issues. Turkey is a country with a relatively

large portfolio of activities supported by UNICEF. One of the most comprehensive was the ‘Hey Girls

Let’s Go to School’ Campaign to support girls’ enrolment which was implemented nationwide between

2003 and 2010, and which significantly contributed to systemic policy debate at Ministerial level.

Likewise, the adoption of national primary education standards, aiming at addressing the disparities

between schools in terms of the quality of education, represent a key issue UNICEF is supporting within

the Turkish context, alongside a programmatic shift towards secondary education.

6.2 Summary conclusions as a response to the evaluation questions

This section summarises the detailed findings presented in Section 5 against the evaluation

questions contained in the Evaluation Matrix, and on the basis of the achievement of the indicators

assigned to each question.78

6.2.1 Impact results

Impact results reported by countries and territories, as supported by quantitative evidence contained in

international databases, show concrete changes in the life of children in terms of (i) generally improving

Pupil:Teacher Ratios, enrolment rates and attendance rates over the past 10 years; and (ii) an increasing

benefit for previously excluded target groups such as children with disabilities, children from ethnic

minorities and/or children from poor social backgrounds. The realisation of children’s right to quality

education is indicated by constantly improving PISA scores and the establishment of quality standards

for education, in particular basic education. While there is clear progress towards gender parity in

77 The Serbia CO regards the process of EU integration as an important change driver, particularly in the area of

minority rights and Roma (i.e. the EU integration political criteria). 78 Please refer to Appendix 4b for a detailed overview of evaluation questions, related indicators and related

scoring.



education, challenges remain in some countries and territories (i) for Roma girls; and (ii) for boys

regarding school achievement.

Active learning was another focal area of UNICEF support over the past 10 years, meant to strengthen

inclusion of all learners in the teaching and learning process. While there is a significant number of

teachers who apply child-centred active learning methodologies, inclusion in terms of equity in learning

opportunity still remains problematic.

Most successful strategies used by countries and territories to realise children’s right to education and

to significantly reduce equity gaps refer to (i) the formulation of policy and legislative frameworks, and

to (ii) sensitisation strategies focussing on a change in discriminatory attitudes. Obviously, behavioural

and attitudinal change is a long-term process, but all visited countries and territories successfully

demonstrated a common approach aiming at attitudinal changes at various system levels. The realisation

of CfS standards clearly contributes towards the responsiveness of teachers, students and communities,

although this still requires attitudinal changes at the larger level of society to fully unfold its impact.

Local support networks and strengthening school-parent cooperation have a positive impact on making

education more inclusive and the improvement of the quality of teaching and learning.

In all visited countries and territories, UNICEF contributed to governmental policy formulation through

its work in terms of knowledge generation, evidence-based advocacy, policy advice and technical

support. This is particularly evidenced through a widespread occurrence of UNICEF-related concepts

in governmental publications and/or statements, even if those concepts are not always expressed in

UNICEF-specific terminology.

Regional initiatives through the Regional office influenced and impacted on the strategies and

interventions implemented at country level in the most successful manner through (i) study tours; (ii)

joint workshops RO/CO; and (iii) international lobbying for region-specific concerns by means of

utilising UNICEF’s international position in the DP community. More specifically, this refers to

UNICEF’s core roles of (i) modelling (by distributing and reflecting upon best practices); enabling

knowledge exchange; and (iii) partnerships building (particularly regarding leveraging influence and

funding, e.g. with EU partnerships).

6.2.2 Effectiveness

There is a clearly identifiable impact chain for UNICEF activities over the past 10 years. The impact

chain is an indicator for the effectiveness of UNICEF’s work in the visited countries and territories

regarding system changes, and thus chronologically expands over three steps: (i) Infusion of key

principles of inclusion (i.e. CfS, interactive learning, life skills) into the educational and societal system;

(ii) integration of such principles into the legislative system; and (iii) application of legislative

frameworks (laws, strategies) in the reality situation of the respective societies. UNICEF’s influence on

policies at country level has been particularly effective as a consequence of (i) designing and piloting

innovative models for replication and incorporation in national programme design; (ii) supporting

policy development; and by (iii) providing assistance towards setting norms and standards and/or

country-specific M&E systems.

Overall, UNICEF’s system approach towards programming has resulted in the introduction of systemic

changes at the level of the respective country governments and educational ministries. In cases where

system changes have not yet been fully successful in achieving inclusive education, these relate to (i)

weaknesses in education planning and management; (ii) weaknesses in decentralised decision-making,

(iii) lack of multi-sector cooperation, and (iv)education systems not operating in an integrated way.



In all visited countries and territories, UNICEF managed to establish a sound partnership with the

respective governments, and also with other developing partners (both international and national,

including local NGOs). The often sensitive issue of inclusion and/or overcoming marginalisation has

been addressed effectively due to the high degree of trust that UNICEF managed to create on the ground.

Due to this high degree of trust, UNICEF has generally been able to effectively (i) influence key policies

in terms of inclusive elements in the education system (related to e.g. child-friendliness, relevance of

the curriculum, life-skills based education, violence prevention); (ii) support implementers and

decision-makers throughout the mutual process of (anti-discriminatory) policy-making and practical

application; (iii) provide successful models in order to generate a meaningful platform and critical mass

of experience for scaling up pilot initiatives within the educational system.

6.2.3 Relevance

In general, UNICEF’s interventions at country level have successfully been designed to influence

inclusive policies and system changes, and have been targeted towards marginalised children and those

whose right to education was violated (e.g. children with special educational needs who were refused

access to mainstream education). Guiding principles of UNICEF (and other partners) have, as an

overarching strategy and as a result of a thorough consultation process, been matched to governments’

policies and reform agendas.

It needs to be acknowledged that, particularly during the initial phases of cooperation in the early

2000’s, the immediate relevance of UNICEF’s activities for the ultimate target groups (i.e. the

marginalised and excluded) has somewhat been structurally and systemically limited due to the

prevailing legislation at the time of UNICEF’s initial involvement. For example, larger political

contexts have (initially) denied the existence of marginalisation of certain ethnic groups, which then

made it difficult for UNICEF to specifically match such policies with their own programming; or, a

longer timeframe was needed to create systemic changes in line with UNICEF’s policy objectives.

However, relevance was always ensured by the fact that UNICEF has ultimately been successful in

supporting and creating a systemic environment which can be regarded as “enabling” towards

addressing child deprivations. Thus, based on the relevance for the target countries and territories and

its main stakeholders and beneficiaries, UNICEF has contributed to the removal of local/contextual or

group specific barriers and bottlenecks in the realisation of inclusion in all visited countries and

territories, since key strategic interests of minority children are now protected by law.

6.2.4 Efficiency

The degree of progress in terms of expected system changes has been an overarching indicator for the

efficiency of UNICEF support towards policy development and resulting legislations. Eventually, this

depends on the time required to translate existing laws (refined or even established on the basis of

UNICEF’s involvement and contributions) into pedagogical practice, in order to create an even greater

match between UNICEF programming and governments’ policies and (forthcoming) reform agendas.

The ultimate “value of change” in terms of efficiency – from a systemic point of view – is likely to be

greatest if a sound modelling/piloting strategy, coupled with policy advice and technical assistance,

eventually feeds into the acceptance of new (educational and societal) norms and standards, which of

course has particular relevance for inclusion. Most powerful, however, is the voice of representatives

from minority groups traditionally being excluded from education.

Based on such a “value of change”, UNICEF has managed to play a catalytic role in all the visited

countries and territories, in order to leverage partners’ funding (and commitment!) for inclusive

education reforms. At times, there have been gaps between the initial intention of an intervention and

the realities of implementation, for example when teachers were overwhelmed by reforms for which



they were not sufficiently prepared. Nevertheless, UNICEF has been successful in making

governments’ reforms and partners’ priorities more relevant to the realisation of the right of all children

to be included in quality learning.

6.2.5 Sustainability

Sustainability issues regarding the results for children and system changes relate to two categories, i.e.

(i) country-specific conditions beyond the influence of UNICEF (degree of a country’s political

stability, macro-economic stability, recurrent costs to be covered by a country’s national budget); and

(ii) the degree to which system changes can be maintained. The findings have generally widened the

focus towards increasingly addressing systemic issues in order to sustain once introduced system

changes over a longer time period. As such, inclusion needs to be seen within a broader framework of

inclusiveness requiring an education system which does not exclude any section of society or any person

involved.

Poverty appears to be a key barrier for children’s school enrolment and attendance, i.e. domestic labour,

seasonal work. Addressing poverty related barriers to education requires a holistic approach and

collaboration between different international organisations and between Ministries. Furthermore,

gender and other minority rights need to be mainstreamed in every level of decision-making.

Comprehensive (inter-sectoral) strategies are also required for integration of children with multiple

disadvantages and children without parental care.

Besides equitable access to education, sustainability of system changes within the area of education

also depends on the access to system resources outside the pedagogical realm. The findings have shown

that UNICEF‘s position within the Development Partner community (particularly at international HQ

levels, through the RO) has already been utilised to encourage quality aspects to be included in

milestones which trigger the release of donor funding, also beyond involvements related to Ministries

of Education only.

6.3 Recommendations

Based on the conclusions above, recommendations given in this section are directed towards their

respective audiences at (i) the regional level, and (ii) the country level. The key focus of the

recommendations is on UNICEF’s contribution to system changes and their translation into concrete

changes in the lives of children.

Recommendations are phrased in such a way that they can facilitate a management response according

to the Guidance for Management response to evaluations.79 Since this evaluation is an evaluation with

a regional focus, recommendations at country level are formulated as a pool of country-specific

recommendations, from which concerned countries and territories can select when discussing their

individual management response. Focus will be on improving country pathways to implement/realise

the ToC. In line with the management responses to be formulated by RO and COs, recommendations

will focus on (i) immediate actions (to be accomplished within a period of up to 60 days after approval

of the report) and (ii) actions to be accomplished within one year.

79 UNICEF Evaluation Office, May 2012.



6.3.1 General recommendations

The conclusions have highlighted the progress in terms of more quantitative impact results related to

access (both in absolute terms and in relation to enhanced equity of access) and learning achievements.

Given the underlying timeframe of this evaluation, i.e. 10 years, and the necessary duration to observe

impact changes at the system level, this has been an important starting point.

There is now a need to intensify the realisation of equitable participation, i.e. there is a need to move

from quantitative to qualitative inclusion. Education systems in visited countries and territories are

still very much based on formal classroom and method-oriented theories. Such systems have built-

in characteristics that are meant to select and therefore exclude, i.e. organising schools in age/year

groups, using methods which pre-determine at what level students should be and when, standardising

testing and examinations. Even the felt need for having minimum learning standards demonstrates

thinking in terms of categorisation and selection. Based on the conclusions presented in Section 6.2

above, a number of key recommendations emerge from the consultations at the visited countries

and territories. These were shared with all the Country Offices during the country debriefings, and

were then confirmed both in terms of relevance and feasibility. The recommendations have a strong

focus on operationalising “transformation” in a systemic sense, in order to achieve systemic changes.

It is generally recommended that on the basis of the achievements reached during the past 10 years,

UNICEF now particularly contributes to system changes which address the realisation of equitable

participation as outlined above. On the basis of an established generally enabling environment towards

inclusion – certainly an important achievement of UNICEF contributions towards systemic changes

over the past 10 years –, it will now be important to contribute to strengthening implementation

aspects in line with the contributory framework of the ToC.

6.3.1.1 SYSTEMS CHANGE CONTRIBUTIONS

In order to progressively realise child rights and equity, the ToC focuses on contributions towards

system changes to which UNICEF in turn contributes through its core roles (UNICEF contribution, see

6.3.1.2 in the next section). The status of the four systems change contributions80 is summarised below

on the basis of the detailed findings presented in Chapter 5, in order to show the general

recommendations emerging from the analysis. These will then be operationalised further in the specific

recommendations given in section 6.3.2.

Systems change contributions

Contribution Status

(Achievements, Challenges) General recommendations


Laws and policies for inclusive

education in place; political will

towards implementation at times

not supported with required

resources and not necessarily

carried by larger society

Focus on social rules of behaviour

(mindsets), in order to provide

support for existing legislation;

utilise political will for developing

accountability strategies and for

soliciting funding

Supply Physical access (services,

facilities, information) to inclusive

Strengthen service delivery of

governmental institutions charged

80 Every systems change contribution is broken down into specific determinants as reflected in the MoRES

framework. Please refer to Appendix 4b for a detailed assessment (scoring) of the respective determinants.



Systems change contributions

education significantly improved;

essential commodities/inputs to

deliver the service not sufficiently

in place

with improving inclusive

education, and with adopting

inclusive practices

Demand

Stakeholders are increasingly

aware of available opportunities

for inclusive education; current

practice does not meet the demand

(issues of cost and quality),

current mindsets preclude

utilisation of system change

Strengthen current practice of

implementing inclusive education,

particularly in terms of cost and

quality; secure community support

for inclusive practices

Quality

Increasing orientation towards the

establishment of quality standards;

some standards (e.g. standardised

testing) are meant to select and

therefore exclude

Refine quality standards; create an

understanding for effects of

standardisation on inclusion vs.

exclusion

6.3.1.2 UNICEF CONTRIBUTIONS TO SYSTEM CHANGES

This section provides a summary of the status (achievements and challenges) of UNICEF contributions

to system changes through its core roles, based on the detailed findings presented in Chapter 5. These

recommendations will be operationalised further in the specific recommendations given in section 6.3.2,

and will be of particular relevance for future programming since these will be under the direct influence

of UNICEF.

UNICEF contributions to system changes through its core roles

Contribution Status

(Achievement, Challenges) General recommendations

Policy advice and technical

assistance

UNICEF contributed significantly

to policy formulation in the

interest of children’s right to

education and the reduction of

equity gaps; policy

implementation has not yet

reached all system levels

Provide support towards

implementation of existing

legislation

Modelling

UNICEF contributions in this

field have been extremely

meaningful and efficient, and have

directly fed into the formulation of

policies and legislative

frameworks

Use modelling for the introduction

of new methods for monitoring

educational quality, and for

developing examples for

effectively implementing existing

legislation



UNICEF contributions to system changes through its core roles

Contribution Status

(Achievement, Challenges) General recommendations

Facilitating national dialogue

towards child-friendly social

norms

UNICEF managed to build strong

partnerships in order to

successfully advocate for changes

in national priorities in favour of

inclusion; local level capacities

for resulting policy

implementation not yet fully

developed

Strengthen local level capacities

for policy implementation;

strengthen inter-sectoral

coordination with a focus on

generating/deepening an inter-

sectoral perspective on inclusion

Enabling knowledge exchange

UNICEF advocacy and research

contributed to fresh policy

commitments towards inclusion;

policy commitments at times do

not match implementation

capacities at national and local

level, i.e. local governments

Provide support towards

improving implementation

capacities at national and local

level, including an exchange of

knowledge directed at

implementing agents of respective

governments and ministries

Monitoring and evaluation

UNICEF overall coordination and

advocacy resulted in the

establishment of M&E systems in

all visited countries and

territories; aspects of educational

quality (beyond learning

outcomes) not yet sufficiently

addressed

Further integrate quality aspects

into monitoring; create a linkage

between ethnicity and poverty;

include social issues which impact

on education; strengthen inter-

ministerial collaboration for M&E

Leveraging resources from the

public and private sectors

UNICEF contributed to leverage

partners’ funding for activities

focusing on inclusive education,

particularly regarding large and

influential partners (EU, WB); the

range of disadvantaged groups is

very diverse and is not yet reached

in a balanced way, and changes in

government can put successful

activities at risk of being

discontinued

Strengthen partnerships with large

international development partners

(e.g. EU) who have a stronger

influence on policy

implementation; strengthen inter-

ministerial cooperation also in

terms of increased opportunities

for leveraging resources for the

education sector

The ‘voice’ for children and

adolescents

UNICEF-supported campaigns,

regional conferences, and

evidence-based advocacy have

made the issues of disparity

clearly visible; there are structural

limits to what sensitisation and

advocacy can achieve

Use the ‘voice’ to sensitise the

‘included’ (i.e. those who are fully

integrated into the current system)

about the benefits of interacting

(and learning from) the ‘excluded’



6.3.2 Specific recommendations

6.3.2.1 RECOMMENDATIONS FOR THE REGIONAL OFFICE

Generally, the Regional Office should play a particular role in coordination, facilitation and

dissemination of the associated work and its documentary of outcomes. Support should be provided by

means of the various core roles of UNICEF, depending on the contextual needs of the various countries

and territories. In particular, the Regional Office should execute the core role of Knowledge Exchange

and Knowledge Generation.

Recommendation 1 (immediate action):

Provide support to Country Offices in soliciting partner engagement within the framework of

the partnership agreement, in order to facilitate the dissemination process of the evaluation

report, and subsequent discussions to be held at country level (also in terms to create ownership

for the findings and recommendations).


Prepare a regional meeting to provide a forum for exchanging key findings and

recommendations of the evaluation, and to reflect upon the conceptual issues raised therein (e.g.

qualitative vs. quantitative inclusion, physical access vs. equity in learning opportunity, use of

standardised testing vs. tracking learning progress of students).

Recommendation 3 (within 1 year):

Conduct a regional meeting (as recommended above) and document the key outcomes,

particularly regarding the conceptual discussions; exchange best practices and reflect upon

causes for observed shortcomings in specific countries and territories as observed in the

examples provided in the evaluation report.


Represent the regional position of UNICEF in meetings with other development partners, raise

the issues of importance of (i) inter-sectoral cooperation; (ii) inter-ministerial cooperation; and

(iii) and an inter-sectoral perspective of inclusion, also with a view to building new partnerships

and to solicit further support and funding.


Facilitate a process of identifying quality standards for inclusive education for the CEE/CIS

region which go beyond quantitative issues and learning outcomes, and which take into account

(i) mindset issues and attitudinal changes, (ii) active learning involvement of all learners and

(iii) benefits for those who are already fully integrated into the system and can learn from the

previously ‘excluded’, and (iv) the extent to which education systems are becoming inclusive.

As a starting point, such process could involve the countries and territories involved in the

evaluation, the results of which could then be used as a model to present to other countries and

territories in the region.

6.3.2.2 RECOMMENDATIONS FOR THE COUNTRY OFFICES



As discussed during the Validation Workshop, recommendations for the Country Offices are presented

as a pool of country-specific recommendations from which concerned countries and territories can

select when preparing their respective management response. Nevertheless, the evaluation team aimed

at formulating recommendations in such a way that most of the countries and territories can relate to

them, although the country-specific operationalisation might be different from country to country in

terms of their specific characteristics.

The first four recommendations are directly related to the Regional recommendations and should be

addressed immediately following the approval of the evaluation report; the subsequent

recommendations take into account important operational issues deriving from the conclusions and

general recommendations presented earlier.


Solicit partner engagement, disseminate the evaluation report, and organise a workshop to

discuss the findings and recommendations of the report. This could be done in two steps, i.e.

with an internal Government/UNICEF meeting first, and then with a joint meeting directed at

the larger development partner community in the country.


Convene a stakeholder meeting to reflect upon the conceptual issues raised in the evaluation

report (e.g. qualitative vs. quantitative inclusion, physical vs. intellectual access, use of

standardised testing vs. tracking learning progress of students). This could be done in

preparation for a regional meeting (cf Regional recommendation #3), but would also serve as a

basis for greater clarity in terms of theoretical concepts and their practical implications at

country level.


Represent UNICEF’s position in meetings with other development partners, raise the issues of

importance of (i) inter-sectoral cooperation; (ii) inter-ministerial cooperation; and (iii) and an

inter-sectoral perspective of inclusion, also with a view to building new partnerships and to

solicit further support and funding. This recommendation will be further supported by the

Regional recommendation #4.


In preparation for a Regional workshop (cf Regional recommendation #5), and together with

country-relevant stakeholders, identify quality standards for inclusive education for the

CEE/CIS region which go beyond quantitative issues and learning outcomes, and which take

into account (i) mindset issues and attitudinal changes, (ii) active learning involvement of all

learners and (iii) benefits for those who are already fully integrated into the system and can

learn from those who have been excluded from mainstream education.


Provide targeted support (technical assistance) to implementers81 who transform inclusive

policies and strategies into reality by means of practical application. This will aim at

strengthening the service delivery of governmental institutions charged with inclusive

education, with a strong focus on adopting inclusive practices at the implementation level. In

practical terms, UNICEF could provide TA to be placed directly at the implementing institution.

81 Implementers will primarily be Ministries of Education, but also include specific government bureaus and

departments charged with curriculum development, teacher training, monitoring and evaluation.




Sensitise stakeholders that “inclusion” refers not only to the issue of SEN children, but to all

who are excluded from education due to a broad variety of deprivations and marginalisations.

In particular, sensitise the ‘included’ about the benefits of learning from the ‘excluded’, in other

words, promote the concrete benefits for children attending regular classes to encounter and

interact with disabled children or children coming from disadvantaged backgrounds. In this

regard, aim at including adults from minority groups into the debate, in order to ensure ‘self-

representation’ of minorities who can share experiences on how to influence societal thinking.


Provide support towards strengthening the M&E systems, by putting a focus on monitoring and

assessing qualitative aspects of inclusion such as subjective perceptions, concrete classroom

interaction and societal mindsets82. Further, create a linkage in the monitoring structure which

enables M&E personnel do draw conclusions on social issues impacting on education (such as

ethnicity and poverty).


Strengthen inter-sectoral coordination at country level and aim at the emergence of a country-

specific inter-sectoral perspective on improving inclusion (which might even be reflected in the

M&E systems of other sectors); in countries and territories where donor coordination is not yet

established, work towards the formalisation of such structures, and ensure that partners are

meeting regularly (ideally under the leadership of the government or the respective ministry).

Overall, it will be important to establish a more holistic approach to inclusive education and

social policy development, and to utilise UNICEF‘s position within the Development Partner

community to encourage quality aspects to be included in important planning milestones, also

beyond involvements related to Ministries of Education only.


Through meetings and/or workshops, create a deeper understanding of the theoretical concept

of inclusive education, which is based on fundamentally different theories about learning than

year-group/method-oriented education. Achieving inclusive education therefore requires not

only a different way of organising teaching and learning (schools), but also a different

functioning of education systems, i.e. the way education systems interrelate and cooperate in a

multi-sectoral way. This includes a decentralisation of decision-making responsibility, and

school autonomy together with school finance. Ultimately (i.e. beyond the 1-year-timeframe),

approaches need to be developed to replace “old system thinking” with “thinking in terms of

inclusiveness” at all levels.


During partner consultations, promote a common approach from pre-school through to

secondary and even high school83. Such a deepened understanding towards the continuous

development of children will not only enhance the realisation of inclusion from an early age,

but will also thus promote the cognitive and affective growth of all learners at all levels of the

learning continuum. As an additional benefit, such approach might lead to strategies which

improve school attendance at the marginal ages of six to seven and fourteen years onwards

where the attendance is remarkably low in most countries and territories.

82 Societal mindsets should be monitored by Perceptional Surveys. 83 Within the learning continuum, pre-school will play a central role since the issue of inclusion can then be

introduced right from the start; and young children are much more likely to accept children from “other”

environments .



7 ADDITIONAL LESSONS LEARNT

This section presents additional important lessons learnt which go beyond the immediate evaluation

findings, but which the evaluation team considers to be relevant for UNICEF to consider during future

evaluations:

(i) The most important lesson learnt refers to the assessment of UNICEF’s contributions to system

changes. It is a common challenge in evaluations to link intervention activities aiming not only

at specific, regionally isolated or socially well-defined target groups but at system wide impacts

(i.e. nation-wide societal and/or political impacts), to the changes in the respective system that

occurred during the intervention period. The measurement of the indicators reflecting these

system changes and the documentation of the efforts undertaken by the implementing agency

to generate these changes is not always sufficient to verify the cause-and-effect hypotheses that

underlie the Theory of Change.

In fact the linkages between the causes and the (system) effects need to be reconstructed either

by searching for empirical evidence in the field which provides plausible explanations (as done

in this evaluation), or even better by collecting monitoring data throughout implementation that

demonstrates reliably the impacts on the micro level (e.g. individual schools, regionally). While

‘plausible explanations’ can always be questioned by alternative, as well plausible, explanations

that attribute the observed changes to external factors (e.g. typical ‘contagion effects’ such as

interventions of other agencies, general economic development), continuous monitoring can

provide a more reliable database. Although a number of documents contain substantial

information about micro level impacts (which has also been taken into account during this

evaluation), a continuous impact oriented monitoring system which provides coherent and

trans-nationally comparable data could improve the collection of relevant evidence.

(ii) One of the expected results mentioned in the Regional Education Strategy is “the creation of

education systems designed to build social cohesion and tolerance to reduce tension and prevent

conflict, especially in areas of ‘frozen conflict’ and on-going ethnic tensions...” To ensure

effectiveness in all programming, the creation of such inclusive education systems had to take

place “within existing national education frameworks and reinforcing existing mechanisms and

tools” (D027). With an initial focus on policy and legislation, the question how to arrange the

logistics i.e., the grouping of students, the scheduling of lessons, and school organisation, in

such a way that they support the implementation of inclusive education at classroom level was

not given sufficient prominence. Logistics need to follow didactics. If not, efforts to improve

the quality of teaching and learning (didactics) suffer.



DETAILS ON REGIONAL CONSULTANT TEAM

National experts provided valuable inputs regarding the respective local contexts and their specific key

thematic areas. They provided significant support to evaluation activities in the field (meetings,

consultations, interviews, focus group discussions) and participated in the analysis of country data and

information.

The national team is presented below in more detail (in alphabetical order of the visited countries and

territories):

Armenia

Susanna TADEVOSYAN is an Armenian national with over 20 years of experience

in issues related to social inclusion of children with special needs, particularly

disabilities. Being the leader of a national NGO for the protection of rights of

children and youth with disabilities, she has demonstrated strong leadership and

strategic skills in promoting the rights of people with disabilities in country policies

and practices from a social model perspective. We benefitted from her strong

partnerships with national, regional and local governments, NGOs, the local media

and also international organisations. During the past 15 years, Mrs Tadevosyan has

managed more than 58 projects within the thematic area of creating equal

opportunities for children and youth with disabilities and special needs in Armenia.

Kosovo

(UNSCR 1244)

Vlera KASTRATI holds Kosovar and Macedonian citizenship and is resident in

Pristina, Kosovo (UNSCR 1244). She has worked as an expert for research and

evaluation with a number of international organisations including UNICEF, where

she contributed to the promotion of gender as an integral part of the New Kosovar

Curriculum Framework (KCF) and Early Development Standards (ELDS). For the

European Commission (EC), she provided support for the integration of

marginalised groups within the framework of strengthening cross-cutting issues of

inclusive education in all EU sector-wide approach programme components. She

also undertook a territorial assessment on the Romani education population for the

Roma Education Fund.

The former

Yugoslav

Republic of

Macedonia

Serbia

Tatjana PERIC resides in Serbia and holds nationalities of both Serbia and Bosnia

& Herzegovina. Apart from an MA in Theory and Practice of Human Rights she

also holds a Certificate on Romology (Roma Studies) from the University of Novi

Sad, Serbia. She has a vast research experience on Roma issues in the region (e.g.,

inter alia, Albania, Bulgaria, Croatia, Macedonia, Montenegro, Serbia, Bosnia &

Herzegovina), and also in conducting evaluations in the areas of education,

employment and inclusion of Roma. She repeatedly worked as a cross-territory

expert for the Western Balkans regarding social inclusion of ethnic minorities, and

has published widely, as author, co-author, editor and contributor to international

publications.

Turkey

Cennet ENGIN-DEMIR is a Turkish national and an Associate Professor at the

Department of Educational Sciences of the Middle East Technical University in

Ankara. Holding a PhD in Curriculum and Instruction, she is a high-calibre

education expert with extensive knowledge of the Turkish education system. She

has been teaching undergraduate and graduate courses in education, also with a

particular focus on Social Foundations of Education and School and Society. Her

recent research included topics such as barriers of girls schooling, increasing

enrolment rates especially for girls, and academic performance and school

attendance of children in Turkey.



LIST OF PERSONS CONSULTED

Note: School staff consulted during field visits are not listed here in order to maintain their anonymity

UNICEF Regional Office, Geneva:

GAIA, Elena, Policy Analysis Specialist

LUZOT, Anne-Claire, Regional Advisor, Monitoring & Evaluation

MAHMUDLU, Siraj, Monitoring and Evaluation Specialist

TESTOT-FERRY, Philippe, Senior Regional Education Advisor

FIELD VISITS:

Armenia:

AHRENS, Henriette, Representative, UNICEF Country Office, Yerevan

AVAGYAN, Armenuhi, Dean of Special Education Faculty, Armenian State Pedagogical University

after Kh. Abovyan, Yerevan

BAKHSHYAN, Anahit, Board Member, Education Expert, Armenian Centre for Democratic Education

CIVITAS, Yerevan

GRIGORYANTS, Grigori, Policy and Quality Assurance Expert, Child Protection and Education,

World Vision Armenia, Yerevan

HAKOBYAN, Anna, Chief Expert, Ministry of Labour and Social Issues, Unit on Issues of Disabled

Persons, Yerevan

HARUTYUNYAN, Anna, Senior Manager, Programme Development & Quality & Member Services,

Child Protection Focal Point, Save The Children, Yerevan

KARAPETYAN, Sona, Evaluation and Monitoring Officer, UNICEF Country Office, Yerevan

KATINYAN, Vardush, Director, Yerevan Medical-Psychological-Pedagogical Assessment Centre,

Yerevan

KIRAKOSYAN, Artak, Chairman of the Board, Civil Society Institute Armenia, Yerevan

MURADYAN, Ruzanna, Deputy of the National Assembly, Republic of Armenia; President of the

NGO “Education without Boundaries”, Yerevan

NAZARYAN, Lilit, Civil Society Programme Coordinator, Open Society Foundations Armenia,

Yerevan

POGHOSYAN, Meri, Education Officer, UNICEF Country Office, Yerevan

SAGHOYAN, Irina, Country Director, Save The Children, Yerevan



SHAHVERDYAN, Anush, Education Consultant, The World Bank, Yerevan

STEPANYAN, Robert, Head of Division for Development Programmes & Monitoring, Ministry of

Education, Yerevan

TER-GHEVONDYAN, Armine, Education and M&E Assistant, UNICEF Country Office, Yerevan

Kosovo (UNSCR 1244):

ARIFI, Muhamet, Director, Balkan Sunflowers (Volunteers for Social Reconstruction), Prishtina

BEAUMONT, Sophie, Social Development Task Manager, Operations Section, European Union Office

in Kosovo (UNSCR 1244), European Union Special Representative, Prishtina

BEHLULI, Lulavere Kadri, Head of Department for Special Needs Education, Ministry of Education,

Science and Technology, Prishtina

BËRDYNA, Agim, Director of the Department for Development of pre-University Education, Ministry

of Education, Science and Technology, Prishtina

BERISHA, Remzije, Director, NGO “TEMA” (Trainers for Interactive Teaching), Prishtina

DEMJAHA, Luljeta, Kosovo (UNSCR 1244) Education Centre, Prishtina

DUMOSHI, Vlora, Prishtina Municipality, Prishtina

ENGEL, Rand, Coordinator, Balkan Sunflowers (Volunteers for Social Reconstruction), Prishtina

GAD, Leila Omar, Head of Office, UNICEF Country Office in Kosovo (UNSCR 1244), Prishtina

GOSALCI, Lirije, Consultant, NGO “TEMA” (Trainers for Interactive Teaching), Prishtina

ISTOGU, Alush, Director of Department of pre-University Education Administration, Ministry of

Education, Science and Technology, Prishtina

KADIQ, Enesa, Ministry of Education, Science and Technology, Prishtina

KADRIU, Drita, Political Advisor, Ministry of Education, Science and Technology, Prishtina

KASNECI, Miranda, Head of Division for pre-University Education, Ministry of Education, Science

and Technology, Prishtina

KELMENDI, Flora, World Bank, Prishtina

MUSLIU, Mimoza, Ministry of Education, Science and Technology, Prishtina

PUPOVCI, Dukagjin, Director, Kosovo (UNSCR 1244) Education Centre, Prishtina

VISOKA, Kushtrim, School Mediators Programme Coordinator, Balkan Sunflowers (Volunteers for

Social Reconstruction), Prishtina



The former Yugoslav Republic of Macedonia:

ALEKSOVA, Anica, Projects Manager, Macedonian Civic Education Centre (MCEC), Skopje

BERRADA, Rajae Msefer, Deputy Representative, UNICEF Country Office, Skopje

BULESKA, Natasha, Project Management Specialist, Education & Workforce Development, U.S.

Agency for International Development (USAID), Skopje

CHESHLAROV, Mitko, Head of Sector (Curriculum), Bureau for Development of Education, Ministry

of Education and Science, Skopje

CONTEVA, Zaneta, Advisor for Research, Bureau for Development of Education, Ministry of

Education and Science, Skopje

CUPI, Redzep Ali, Director: Directorate for Promotion and Development of the Languages in

Education for the Ethnic Minorities, Ministry of Education and Science, Skopje

DESMOULINS, Dr. Bertrand, Representative, UNICEF Country Office, Skopje

GEORGIEVA, Loreta, Executive Director, Macedonian Civic Education Centre (MCEC), Skopje

KENIG, Dr Nikolina, Professor, Ss. Cyril and Methodius University, Faculty of Philosophy, Centre for

Human Rights and Conflict Resolution, Skopje

LAMEVA, Beti, National Examinations Centre of the Republic of Macedonia, Skopje

LUBAROVSKA, Biljana, Project Officer: Child Protection, UNICEF Country Office, Skopje

MICEVSKA, Andrijana, Programme Assistant, UNICEF Country Office, Skopje

NACEVA, Bojana, Senior Education Specialist, The World Bank, Skopje

NIKOLOVA, Marija, Advisor: Directorate for Promotion and Development of the Languages in

Education for the Ethnic Minorities, Ministry of Education and Science, Skopje

PETROSKA-BESHKA, Violeta, Professor, Ss. Cyril and Methodius University, Faculty of Philosophy,

Centre for Human Rights and Conflict Resolution, Skopje

POLENAKOVIC, Liljana, Advisor for Mathematics, Bureau for Development of Education, Ministry

of Education and Science, Skopje

SABANI, Nora, Education for Development Specialist, UNICEF Country Office, Skopje

SGOBBA, Stefano, Delegation of the European Union, Skopje

STOJANOV, Zoran, Monitoring & Evaluation Officer, UNICEF Country Office, Skopje

SULEJMANI, Naziktere, State Advisor, Ministry of Education, Sector for primary and secondary

education, Skopje



TALIMDZIOSKA, Danica, Head: Inter-Municipal Department, Advisor for class teaching, Bureau for

Development of Education, Ministry of Education and Science, Skopje

TODOROVSKA, Elizabeta, State Secretary, Ministry of Education and Science, Skopje

TRAJKOVSKA, Julija, Advisor for Psychology, Bureau for Development of Education, Ministry of

Education and Science, Skopje

Serbia:

BAUCAL, Aleksandar, Institute for Psychology, Belgrade

BOGDANOV, Jovanka, Advisor in Pre-school Department, Ministry of Education Science and

Technology Development, Belgrade

ČAPRIĆ, Gordana, Deputy Director, Institute of Education Quality and Evaluation, Belgrade

CUKOVIC, Anna-Maria, Consultant, UNICEF Country Office, Belgrade

CVETKOVIC, Gordana, Head of School Administration, external evaluation of schools, Ministry of

Education Science and Technology Development, Belgrade

GOSOVIC, Radmila, Network of Organisation for Children (MOST), Belgrade

JANKOV, Ratko M., Education Forum, Belgrade

JOVIC, Aleksandra, Social Policy Specialist, UNICEF Country Office, Belgrade

KIJEVCANIN, Slavica, Pestalozzi, Belgrade

KLAŠNJA, Snežana, Assistant Minister, Ministry of Youth and Sports, Belgrade

KOVAČ-CEROVIĆ, Tinde, Faculty of Philosophy, Belgrade

LAZAREVIC, Bogoljub, Head of Pre-school Department, Ministry of Education Science and


LAZAREVIC, Snezana, MDRI, Belgrade

LAJOVIC, Biljana, Coordinator Violence Prevention Unit, Ministry of Education Science and


MACURA, Sunčica, Teacher Faculty, Jagodina

MAKSIMOVIC, Borislava, Coordinator for Inclusive Education, Department for Development of

Education and International Cooperation in Education and Science, Ministry of Education

Science and Technology Development, Belgrade

MAKSIMOVIĆ, Mirjana, Deputy Manager and Social Policy and Roma Inclusion Coordinator, Social

Inclusion and Poverty Reduction Unit, Government of Serbia, Belgrade



MARKOVIC, Jelena, Education and Human Capital Development Coordinator, Social Inclusion and

Poverty Reduction Unit, Government of Serbia, Belgrade

MAROLT, Ljiljana, Advisor in Pre-school Department, Ministry of Education Science and Technology

Development, Belgrade

MIHAJLOVIC, Milena, Centre for Interactive Pedagogue, Belgrade

MILLER, Lesley, Deputy Representative, UNICEF Country Office, Belgrade

MILORADOVIC, Sanja, Social Protection and Education, Belgrade

MITROVIC, Aleksandra, NGO Society for Improvement of Local Roma Communities

MRSE, Snjezana, Special Education Needs evaluator

NIKOLIC, Bozidar, Romanipen, Kragujevac

PAVLOVIC, Dragica, Institute for Psychology, Belgrade

PEŠIKAN, Ana, NGO Education Forum, Belgrade

RANKOVIC, Tanja, Education Specialist, UNICEF Country Office, Belgrade

SIMIC, Ljiljana, Coordinator of the Network for Inclusive Education

SIMONOVIC, Tamara, Open Club, Nis

SKAREP, Angelina, Department for EU Integration, Development and Research Projects in Education

and Science, Ministry of Education Science and Technology Development, Belgrade

STOJANOVIC, Jadranka, Open Society Foundation, Belgrade

STOJANOVIC, Jelica, EU Delegation, Belgrade

STOJIC, Tanja, Open Society Foundation, Belgrade

TMUSIC, Velimir, Head of School Inspectorate, Ministry of Education Science and Technology

Development, Belgrade

VUJICIC, Lidia, Swiss Development Cooperation, Swiss Embassy, Belgrade

ZAVISIC, Vanja, Veliki Mali, Belgrade

Turkey:

ABULABAN, Ayman, UNICEF Turkey Representative, UNICEF Country Office, Ankara

AKPINAR, Aişe, Head of EDUSER Consulting firm, Ankara



ARICAN Atıcı, Hediye, School Counsellor, former MoNE personnel involved in inclusive education,

Ankara

AYDAGÜL, Batuhan, Coordinator of Education Reform Initiative, Istanbul

BALCI, Mustafa, Sector Manager Education and Training, EU Delegation, Ankara

BELELİ, Özsel, Author of Out-of-School Children in Turkey, Ankara

ÇELİK, Salih, Deputy Undersecretary, Ministry of National Education, Ankara

DEMİR, Mehmet Caner, Sector Manager Social Policy and Human Resources Development, EU

Delegation, Ankara

ER, İbrahim, former (2007-2011) Director-General of Primary Education Department, Ministry of

National Education, Ankara

DOMINICIS de, Regina, UNICEF Turkey Deputy Representative, UNICEF, Ankara

GÖKALP, Yadigar, Head of Social Security Institution, Ankara

GÜNDÜZ-HOŞGÖR, Ayşe, Deputy Dean and researcher METU Faculty of Science and Art, Ankara

HATİPOĞLU, Esra, UNESCO, Ankara

KARABIYIK, Ertan, former UNICEF field assistant, Ankara

KARİP, Emin, Head of Board of Education, Ankara

KUSCUL, Hilal, AÇEV, Ankara

LEVENT YENER, Ahment, World Bank Senior Human Development Specialist, WB, Ankara

OTARAN, Nur, Gender Review Study, Ankara

ÖZDEMİR, Servet, former Director-General of Primary Education Department, MoNE, Ankara

ÖZDEMİR ULUÇ, Fatma, UNICEF Education Officer, UNICEF, Ankara

SART, Hande, Researcher at Bosporus University, Istanbul

TOL, Ulaş, Researcher at YA-DA consultation firm, Istanbul

TOPÇU, Kamil, School Counsellor, former MoNE personnel involved in inclusive education, Ankara

VIS, Simone, UNICEF Turkey Education Specialist, UNICEF, Ankara

WISEMAN, William, World Bank, Sector Leader Human Development, WB, Ankara

YORGANCILAR, Nilgün, Doğan Gazetecilik, Ankara

1

APPENDIX 1:

Terms of Reference

1

MULTI-COUNTRY EVALUATION OF REGIONAL KNOWLEDGE AND LEADERSHIP AREAS

INCLUDING ALL CHILDREN IN QUALITY LEARNING IN CEE/CIS

2002 - 2012

Terms of Reference BACKGROUND For about 10 years now, UNICEF has adopted in the CEE/CIS region a ‘system’ approach to programming with the view to accompany reforms and contribute to concrete changes at institutional, societal and individual level. The 2009 and 2010 ‘Mid-Term Review’ submissions to the Executive Board reported a number of outcome and impact results achieved at country level in the CEE/CIS Region. This represented a clear evidence that UNICEF’s upstream work in a significant number of countries, territories and sub-regions yields tangible results not only in terms of system changes (outcome results) but also in terms of changes in the life of children (impact results). In April 2012, the Regional Office initiated a participatory process involving all Country Offices in CEE/CIS with the view to identify few strategic result areas where UNICEF has the capacity to deliver high-quality and relevant results that contribute to address child rights violations and close equity gaps. A Regional Knowledge and Leadership Agenda (RKLA) emerged from this process, with a double purpose: (1) Inform and guide UNICEF’s future work in CEE/CIS, with a strategic focus on few areas where results

for children can be achieved over the next two-three years across a number of sectors, countries and territories (prospective approach); and

(2) Generate evidence and document - in an aggregated picture - how UNICEF contributed over the past

decade to outcome and impact results for children in a significant number of countries and territories, (retrospective approach). This process is expected to contribute to a regional knowledge exchange

agenda and generate mutual learning among countries on practices and strategies in order to inform the prospective approach.

Philippe TESTOT-FERRY, 14 March 2013 The result areas that were selected for the retrospective approach share the following features:

2

a) They address key violations of child rights in terms of magnitude and/or severity; b) They explicitly contribute to closing equity gaps (except for results in health and HIV/AIDS, which aim

at universal coverage or overall elimination); c) They are being achieved in a significant number of countries and territories; d) And they can be documented and evaluated. The strategic intent is to explore not only the link between system reforms and reduction in child rights violations but also to track the reduction or equity gap and assess the extent to which the indicators related to the most disadvantaged children are catching up with national averages. For each of the following areas, multi-country evaluations will be undertaken to demonstrate how reduction of equity gaps and impact results (in terms of changes in the life of children) were made possible through changes in the national (regional/local) systems and document UNICEF’s contribution to these changes. (1) Children’s right to be raised in a family environment (2012) (2) Juvenile Justice: Children’s right to support to re-integration into society (2013) (3) Children’s right to early learning / school readiness (2013) (4) Inclusion of all Out of School Children in Quality Learning (2012) (5) Children’s right to health: infant and under 5 and mortality (2013) For each of these areas, Reference Groups (composed of representatives of the Regional Office and concerned Country Offices) were set up in order to identify common elements and differences in the country approaches and policy options that led to the achievement of various impact and outcome results. The present Terms of Reference concerns the multi-country evaluation planned for the fourth key result area: Inclusion of all Out of School Children in Quality Learning. CONTEXT National average primary school enrolment rates in the CEECIS region are all over 90-95%; thus, nearly every child enrolls in school at some point in their life. However, the national picture belies sub-national disparities that leave certain groups of children completely excluded from basic education. There are 3.7 million children of primary and lower-secondary school age and 1.6 million children of pre-primary school age out of school in CEE/CIS (UNESCO, 2010 GMR); an additional 12 million adolescents are estimated to be out of school. While on the whole, there are no big gender disparities in access and participation in basic education, with a few exceptions, notably in Tajikistan and few years ago in Turkey, gender disparities in learning outcomes (at the disadvantage of boys) particularly in reading but also in sciences and mathematics, become increasingly reported. Children and adolescents out of school are those from the most socially, culturally and economically marginalized communities and thus are the hardest to reach. Major equity gaps in both access to and outcomes of education exist between these children and their peers from majority populations. The profiles of children that are most likely to be out of school and thus represent the most urgent challenges for governments and partners in the region are: (a) Children from ethnic minorities, especially the Roma; (b) Children with disabilities; (c) girls in few countries and territories, and boys in others; (d)

3

Children from the poorest households; (e) Working children; (f) Children performing below academic standards; (g) Children of pre-primary age - from all the above groups; (h) Adolescents - from all the above groups; and (i) Children with multiple disadvantages (for instance, being a girl, from an ethnic minority and living in a poor household and therefore not being enrolled in pre-primary education). APPROACH TO INCLUSIVE EDUCATION IN CEE/CIS AND KEY RESULTS ACHIEVED For the past 10 years, the overarching result pursued by UNICEF in CEE/CIS is that “Innovative and effective approaches for addressing exclusion in education are scaled-up by governments”. Based on UNICEF’s Education Strategy and MTSP, the vision that has guided UNICEF’s work in education in CEE/CIS for the past 10 years is that ‘Every child in the region will access and complete basic education of good quality’. Toward this vision, UNICEF has focused its education interventions in the region on four long-term (2015) goals, adapted to the specific context of CEE/CIS, in particular the two following ones: (1) reaching the last 10-15% of children who are out of school; and (2) improving the quality and relevance of basic education in order to reduce school drop-outs and increase completion/achievement rates. These two objectives are clearly mentioned in the 2007 Regional Education Strategy Note. In order to support the long-term vision and goals for education in CEE/CIS, the Regional Office set for itself the following objective: ‘Establish a programmatic environment that will enable Country Offices to better position their education programmes, achieve large scale results and remain relevant’. More specifically, four operational strategies to be supported at regional, sub-regional and country levels were formulated, as follows: 1. Critical knowledge will be generated, analyzed, packaged and disseminated in order to:

Create a body of evidences on key issues in basic education (exclusion, disparities, quality, relevance…) that will allow evidence-based advocacy among policy and decision-makers, and influencing sector reforms;

Create a critical mass of best practices or successful projects within countries and territories that have the potential to influence and steer changes within education systems.

2. Strategic partnerships with key actors in education at the regional and international levels will be built

or strengthened in order to: (1) mobilize funding and leverage influence towards sector reforms; and (2) develop programmatic synergies for greater impact of partners’ interventions.

3. Quality assurance and oversight will be provided by the Regional Office in order to improve the quality,

effectiveness, and sustainability of UNICEF country programming approaches in education.

4. Technical assistance will be made available to Country Offices through maintaining and nurturing high calibre technical expertise at regional and international levels in relevant areas of education.

Significant results in terms of changes in children’s life (impact results) have been achieved during this period in 5 countries and territories: Armenia, Kosovo(UNSCR 1244), Serbia, the Former Yugoslav Republic of Macedonia and Turkey.

4

In Armenia: (1) The number of children in special schools decreased by 60% in 2011-12 compared to 2007-2008 (National Centre for Education Technologies); and (2) The percentage of children with certified disabilities attending regular schools is 70% (Source: UNICEF Armenia Survey on the access of children with disabilities to education, health and social protection services); in regions with higher concentration of inclusive schools (Yerevan and Tavush) where UNICEF programs have been implemented, the proportion is higher - 75 and 80 % respectively. In Kosovo (UNSCR 1244), the proportion of drop-outs at primary and lower secondary education levels was divided by three in 8 years, from 1.67% in 2002/03 to 0.5% 2010/11. In addition, the gender parity index in primary school increased from 0.79 in 2004 to 0.93 in 2011, hence implying that more girls are now participating in primary education. In Serbia, the primary school enrolment of Roma children increased from 66% in 2005 (Source: MICS 3) to 91% in 2010 (Source: MICS 4). In the Former Yugoslav Republic of Macedonia , the primary school enrolment of Roma children increased from 61.1% in 2006 (Source: MICS 3) to 85.6% in 2011 (Source: MICS 4).In addition to that: (1) Fourth grade student knowledge in numeracy increased by at least 10 per cent against the baseline; (2) Early grade teachers (1-3 grades) knowledge in numeracy increased by at least 10 per cent against the baseline; (3) Fourth grade student knowledge in literacy (reading and writing) increased by at least 10 per cent against the baseline; and (4) Early grade teachers (1-3 grades) knowledge in literacy (reading and writing) increased by at least 10 per cent against the baseline (Source: assessments in progress of student outcomes and early grade teacher knowledge). In Turkey: (1) It is estimated that about 350,000 children (250.000 girls and 100.000 boys) were enrolled in school as a result of the 2003-2007 Girls Education Campaign (Source: Ministry of National Education); and (2) Gender disparity in primary education has been reduced from 7.15% in 2003 to 0.37% in 2010-2011 (Source: Ministry of National Education).

The results listed above – which translate into concrete changes in the life of thousands of children -were made possible because of the reforms in education systems achieved over the past 10 years by governments with the support of their partners, including UNICEF, both at country and regional level. These concrete changes in the life of children are measured by the following impact indicators: Primary, secondary and pre-primary school enrolment rates, disaggregated by gender, location,

personal characteristics (disability) and ethnicity (Roma, Ashkali and Egyptians) ; Number/percentage of children out of primary, secondary and pre-primary school, disaggregated by

gender, location, personal characteristics (disability) and ethnicity (Roma, Ashkali and Egyptians) ; Gender party index; Percentage of children with special needs attending regular schools versus special schools; Percentage of increase in grade 4 student outcomes in literacy reading and writing) and numeracy; Participation in preparatory pre-school programme. RATIONALE The following section is meant to explain the regional theory of change that led to these impact results, and to which UNICEF made a contribution. A Theory of change is defined as a “Blueprint of building blocks

5

needed to achieve the long-term goals of a social change initiative1”. In this case, it refers to the reduction of equity gaps and impact results achieved through changes in the national (regional/local) systems, partly due to the contribution of UNICEF. Theory of Change The reality is that 10 years ago, there was no explicit theory of change in the CEE/CIS region, neither in the Regional Office, nor in Country Offices. However, what has guided the work of UNICEF in this region over the past decade (regional story line or pathway) is a consensus that the progressive realization of child rights and reduction of equity gaps is best achieved through changes in systems at national/sub-national levels, and that sustained UNICEF engagement through its core roles contributes to these system changes (see also the Generic Theory of Change underlying UNICEF’s approach in the CEE/CIS region). So, this TOR attempts to re-construct a regional theory of change which has been implicit (regional pathway), is based on the mainstreaming the UNICEF Child-Friendly School framework in education systems at national/regional/local, and which contributed to significant changes in children’s life, as measured by the impact results listed above. UNICEF being a very decentralized organization (the Country Programme is central to its work), this theory of change was led, facilitated and supported by the Regional Office (as described below) and implemented in various ways (but with similar goals: the progressive realization of child rights and reduction of equity gaps in basic education) by Country Offices. Since 1995, UNICEF’s work in education globally has been guided by the Child-Friendly School (CFS) approach. Initially, the CFS concept was presented as an ‘umbrella’ under which the diverse activities and goals of UNICEF’s work on schools might be consolidated and rationalized. By early 2000, UNICEF expanded the definition of quality for key elements of child-friendly schools. By the end of 2001, UNICEF had conceptualized a comprehensive and complex CFS approach which is a framework for rights-based and child-friendly educational systems and schools, which are characterized as inclusive of all children, effective for learning, healthy and protective of children, gender-sensitive, and promoting the participation of children, families and communities. The CFS concept was designed as a tool for fulfilling the right of children to have access to an education of good quality. At the national level, for ministries, development agencies, and civil society organizations, the CFS approach and principles were meant to be used as a human rights-based normative framework for education policies and programmes, leading to child-friendly education systems and environments. At the community level, for school staff, parents, and other community members, the framework was meant to serve as a tool for quality improvement through localized self-assessment, planning and management, and as a means for mobilizing the community around education. While the number of countries in which UNICEF is promoting and using the CFS approach increased from 33 countries in 2004 to 56 countries in 2007, the CFS package was adapted by UNICEF Country Offices to specific country realities and this has resulted in important variations in its application within UNICEF programmes. The main issue has been a tendency to overprescribe on child-friendly schools (project approach) and to under-emphasize normative and upstream work, policy development and capacity building of duty-bearers (system approach).

1 UNICEF Programme, Policy and Procedure Manual.

6

The Eastern Asia and Pacific region is the part of the world where UNICEF made the fastest and deepest progress in promoting and implementing the CFS approach. In 2005, there were more than 800 schools certified as child-friendly in Thailand, a similar number in the Philippines, and the Regional Office in Bangkok had developed a set of standards and indicators2 for each of the five components of the CFS framework. In the CEE/CIS region, the CFS approach was introduced in the context of the breakup of the Soviet Union and Yugoslavia, and the beginning of the transition from socialism to pluralist societies, with education becoming a key vehicle for managing changes in values, respects for child rights and democratic participation. Since the mid-nineties, the focus in the region has been mainly on the ‘effectiveness’ component of the CFS framework, with the implementation of child-centered approaches in teaching and learning such as the Active Learning (AL) projects in Azerbaijan, Bosnia and Herzegovina, The former Yugoslav Republic of Macedonia and Serbia or the Global Education Project in four Central Asian countries. In some other countries, the focus has been more on the ‘inclusiveness’ component of the CFS framework, with the Girls Education Campaign (project) in Turkey, as well as the inclusion of children with disabilities in Armenia or Roma children in Kosovo (UNSCR 1244) and Serbia, among other countries and territories. The transition period also corresponded to and was followed by a phase of wars and conflicts – between Armenia and Azerbaijan from 1988 to 1994, in countries and territories of the former Yugoslavia (Bosnia & Herzegovina, Croatia and Slovenia) from 1991 to 1995, the civil war in Tajikistan in the 1990s, the Kosovo (UNSCR 1244) war from 1998 to 1999…etc. The impact of these conflicts and the need to restore education services led a number of UNICEF Country Offices to focus on service delivery (infrastructure, basic commodities and supplies) and to adopt a fragmented approach. Indeed, the common feature of education interventions in most of these CEE/CIS countries and territories has been the project approach (pilot projects) in very small numbers of schools (often 10-15), with low potential for replication and expansion at national level, and limited impact on education systems. This fact was confirmed by a number of evaluations commissioned or supported by the Regional Office between 2005 and 2008, such as the multi-country evaluation of Global Education in Central Asia as well as CFS or AL evaluations and assessments in Azerbaijan, Bosnia and Herzegovina, Kosovo (UNSCR 1244), The former Yugoslav Republic of Macedonia and Moldova. In the mid-2000s, the acquisition of the EU Candidate status by some countries in the region (mainly in the Balkans but also previously Turkey) triggered significant education sector reforms and system changes. In 2005, the Regional Office initiated a Regional Analysis of Education in CEE/CIS (‘Education for Some More than Others?’), which laid the foundations for a regional vision for Basic Education which, in turn, were later formalized in the Regional Education Strategy and Regional Education Strategy Note. In 2006, the Regional Office organized a CFS Study Tour in Thailand for 12 countries and territories involved in implementing the CFS approach in the region - Armenia, Albania, Bosnia & Herzegovina, Kosovo (UNSCR 1244), The former Yugoslav Republic of Macedonia, Montenegro, Serbia, Tajikistan, Turkmenistan, Turkey, Uzbekistan. This study tour represents a key milestone (or starting point) for the shift made by most countries and territories in CEE/CIS from small CFS projects to a more systemic approach using the CFS concepts and principles as an overarching conceptual framework for policy development and sector reform rather than for applying them at school level only. Returning from Thailand, a number of countries

2 Attig, G. & Hopkins, J. (2006). Assessing child-friendly schools: A guide for program managers in East Asia and the

Pacific. Bangkok: UNICEF East Asia and Pacific Regional Office

7

undertook evaluations or assessments of their CFS projects (as mentioned above), some decided to scale up their initiatives (Azerbaijan, Uzbekistan), and others initiated a system approach, working at various levels (central, regional, local) and with different components of education sectors (policy and planning, curriculum, pedagogic centres, teachers training, assessment departments …); this has been the case in The former Yugoslav Republic of Macedonia Serbia and Turkey. Between 2006 and 2009, all Regional Education Network Meetings and technical support (country missions, external expertise) provided by the Regional Office have been occasions to emphasize the importance to phase out of project-focused approaches, to mainstream the CFS approach and principles in national education laws and policies, and to work strategically and synergistically with different sub-sectors of education systems in support education reforms. Country Offices have systematically been encouraged by the Regional Office to move away from service delivery-type of interventions towards more upstream strategies such as, evidence-based knowledge generation and advocacy, capacity development, cutting-edge technical assistance and partnerships building. Since 2006, the Regional Office has managed to obtain significant amounts of Education Thematic Funding (around $ 5 million per year on average) to support such a programmatic shift and has systematically prioritized UNICEF Country Offices focusing their programming on system changes. Unlike other sectors in UNICEF Country Programmes, the Education Sector in the region has benefitted from a modest but regular and more predictable funding. This has certainly helped shaping the education programming landscape in a more strategic manner in CEE/CIS. Following these shifts at country level, the Regional Office carried out a Regional Study on the Development of Standards for Quality Basic Education, covering 8 countries and territories (Armenia, Azerbaijan, Bosnia & Herzegovina, Kosovo (UNSCR 1244), The former Yugoslav Republic of Macedonia, Moldova, Turkey and Uzbekistan). This study involved a stock-taking of where these countries and territories were in terms of their basic education standards and proposed recommendations as well as a roadmap to develop new standards or revise existing ones. This study represented an important contribution to the system changes that took place in Armenia, Kosovo, The former Yugoslav Republic of Macedonia and Turkey (see country theories of change in annex), and which contributed to the impact results reported above. Such contribution is well documented in a recent article published by Clair, Miske and Patel in the European Education Journal. The work initiated at country level was supported by regional positions and guidance such as the two Regional Position Papers on Inclusive Education for Roma children and children with disabilities. The Regional Office also initiated a reflection in CEE/CIS on the quality of education and equity in learning outcomes through the production and dissemination of two Regional Analyses of PISA results (2006 and 2009). One of the key findings of the 2006 regional PISA analysis was that Estonia, an ex-CIS country, managed to achieve both quality of education (in terms of learning outcomes) and equity. The finding is the same in 2009). In 2010, the Regional Office organized a study tour in Estonia, during which representatives of various departments of the Ministry of Education (curriculum, teachers training, assessment…) presented the system approach adopted by the government to reform its soviet-inherited education sector over the past two decades. A survey of the 15 country delegations (UNICEF staff, MoE counterparts, and NGO representatives) which participated in the study tour (Albania, Armenia, Azerbaijan, BiH, Kazakhstan, Kosovo (UNSCR 1244), Kyrgyzstan, The former Yugoslav Republic of Macedonia, Moldova, Montenegro,

8

Romania, Serbia, Turkey, Turkmenistan and Uzbekistan) revealed that 30% of them had used or were planning to use the lessons learnt from the Estonian experience. The Background Information formulated by the countries and territories participating in this multi-country evaluation and attached in annex (country pathways) provide more details on how system changes, to which UNICEF made a contribution together with its partners, led to a greater inclusion of out of school children in quality learning. The regional theory of change stems from the combination of regional guidance and leadership, and initiatives with country initiatives. This multi-country evaluation is justified by the willingness of the Region to further assess the impact results reported by Country Offices and validate the regional theory of change (pathway) outlined above (in particular the contribution of UNICEF that led to these results but also the lessons learnt, including the not so good ones). The multi-country evaluation comes at a time of funding scarcity when UNICEF is increasingly requested to demonstrate how the organization achieves tangible results and makes a difference in children’s life in countries (particularly in middle-income countries) where financing and supply transactions are not any more the main development commodities and take a secondary role compared to leveraging alliances and partnership for public sector reforms and supporting civil society engagement to improve social norms and attitudes towards children. While the main user of the multi-country evaluation will be the Regional Office and the five concerned Country Offices, the 16 other Country Offices in CEE/CIS will also benefit from the lessons learned. Utilization: The multi-country evaluation report will be used at the regional level to inform the Regional Knowledge and Leadership Agenda (prospective approach) as well as the forthcoming MTR and Country Programming processes. More specifically, its findings and recommendations will inform the Regional Out of School Children and Adolescents Initiative (which Conceptual Framework is the backbone of the prospective approach for including all children in quality learning) and guide the work of the Reference Group. The multi-country evaluation will also be used at a more global level to showcase the work carried out in the CEE/CIS region to include out of school children in quality learning, in UNICEF Headquarters (notably through the Regional MTR Reports to the Executive Board) as well as among partners and donors. The presentations or launches of flagship regional knowledge products (such as the regional reports on quality education and learning outcomes) as well as other regional political or regional conferences will be used as opportunities to raise the visibility of the results achieved by countries and territories in CEE/CIS with the support of UNICEF and its partners. The primary audience for the multi-country evaluation will be UNICEF Senior management at Headquarters, members of UNICEF’s Executive Board, strategic partners and donors; among donors, the government of Norway which is the main provider of Education Thematic Funding not only for UNICEF globally but also for the CEE/CIS region, will be a key recipient and beneficiary of the evaluation. A secondary audience will be governments and partners at country level. The Reference Group will be responsible for ensuring that best use of the findings of the multi-country evaluation will be made. The Regional Leadership Agenda has adopted the global UNICEF Monitoring of Results for Equity (MoRES) framework. MoRES and its determinant analytical framework (Annex 2) will help linking UNICEF’s support to the strengthening of policies and systems to concrete changes in the lives of children. The MoRES

9

determinant analysis will be explicitly used to identify which bottleneck were removed and how. In addition, by using the MoRES framework, the multi-country evaluation will document impact results (L4 indicators); assess and demonstrate how outcome results in terms of system changes (L3 – MoRES determinants) contributed to the impact results; and document how UNICEF had an influence on changes in the children's lives through its contribution (L2) to system changes. Such contributions would belong to the following core roles: Policy advise and technical assistance – through well-designed UNICEF positions (based on local,

regional, international best practices) on key issues, supporting the development of the normative frameworks related to specific national legislation, policy or programme as well as private sector standards that can improve equity;

Modeling – through well designed demonstration models meant to provide the required evidence for policy advice and advocacy;

Facilitating national dialogue towards child friendly social norms – bringing together government, private sector and civil society, as well as convening stakeholders with different interests and perspectives to enhance public debate, participation and action around equity and child rights;

Enabling knowledge exchange – fostering horizontal cooperation and exchange of experience among countries, territories and regions on ‘what works’ for enhancing child well-being and equity;

Monitoring and evaluation – assisting independent assessments and analyses of the realization of child rights and the reduction in equity gaps in child well-being;

Leveraging resources from the public and private sectors – accompanying and re-directing reforms, including those supported by the EU, IFIs, bilateral organizations and national/multi-national corporations.

PURPOSE AND OBJECTIVES OF THE EVALUATION In view of the key results achieved by some countries and territories in CEE/CIS in addressing child rights violations and reducing equity gaps in basic education, and based on the Regional Theory of Change described above, the purpose of the multi-country evaluation is two-fold: 1. Demonstrate and assess the extent to which UNICEF's contribution to programme interventions

which, at both regional and country level, address major child rights violations in education and reduce equity gaps, led to impact results for children – in terms of reduction in the number of children out of school and improved quality of education/learning outcomes; and

2. Generate learning on practices, innovations and models across a critical mass of countries and

territories addressing similar issues - to be used during various UNICEF country programming processes (MTRs, CPDs...etc).

More specifically the multi-country evaluation has the following objectives: a) Document and report on impact results (in terms of changes in children's life) and reduction of equity

gaps, as demonstrated by surveys, administrative data or other sources of information;

b) Assess and demonstrate - through an in-depth review of government and partners' interventions - how such results were made possible through systems changes (removal of system bottlenecks) at national and/or local levels; and

10

c) Document the contribution of UNICEF to these system changes. In view of the diversity of results and programme interventions being reported and documented (inclusion of children with disabilities in Armenia, Roma children in Kosovo (UNSCR 1244) and Serbia, and girls in Tajikistan…), the multi-country evaluation will also be a good opportunity to assess UNICEF’s success in tailoring its programming approaches to the qualitative differences between inequities, with the goal to improve future programming by adapting it to the specific groups of children out of school. SCOPE AND FOCUS Timeline The impact and outcome results that will be analyzed and documented, as well as the reforms and system changes that led to them and UNICEF’s contribution have been achieved during a period of about 10 years (2002-2012). So, 10 years will represent the timeline of the multi-country evaluation. There is no relation between this timeline and UNICEF’s Country Programme cycles. Geographic scope The planned evaluation will have a multi-country dimension under a regional umbrella. It will geographically focus on five countries and territories: Armenia, Kosovo (UNSCR 1244), The former Yugoslav Republic of Macedonia, Serbia and Turkey. The selection of these countries and territories is a result of a thorough consultative process between the concerned Country Offices and the Regional Office. The countries and territories that were eventually selected are those for which the Regional Office and Country Office agreed that it is possible to demonstrate and document retrospectively that UNICEF contributed to system changes that in turn resulted in a reduction of child rights violations and equity gaps in education. Focus of the evaluation The multi-country evaluation will focus on the inclusion of out-of-school children, primarily in terms of equity in access to basic education but will also consider quality learning – and particularly equity in learning outcomes – as a key determinant of inclusion. ‘Basic Education’ will be understood as primary and lower-secondary education. A multi-country evaluation of early learning and school readiness will take place separately. The intent of the multi-country evaluation is not to undertake country level evaluations that would be compiled and summarized at regional level. It is neither to evaluate each specific intervention within one key result area in the way a project or a programme evaluation would be carried out. The idea is to have a strategic multi-country approach to validate the regional theory of change outlined above and document UNICEF's contribution to similar results (inclusion with child rights violations and equity gaps reduction) in a group of five countries and territories addressing issues of exclusion from education.

11

The focus of this formative evaluation covers issues at the core of UNICEF engagement in CEE/CIS and its conclusions and recommendations will be used to inform UNICEF engagement in this area not only in countries and territories covered by this exercises but also the other countries in the Region and beyond where this will be relevant. The evaluation will have a clear focus on impact. Limitations to the evaluation In course of preparing the TOR, an evaluability assessment was conducted with all countries and territories included in the evaluation. All concluded that despite some data gaps (mainly related to level of disaggregation) there are enough data available to conduct adequately the multi-country evaluation (see annex 1). There are reliable data to inform the baseline as well as the situation of children in the most recent years covered by the evaluation. It has to be noted that data sources being different from country to country, trends analysis will be preferred over comparison. In some cases results of MICS surveys will constitute a major source of data while in others national statistics will be the main source data. The Evaluation will also take into consideration that depending the country situation not all impact indicators identified will be observed in all countries and territories. EVALUATION QUESTIONS The multi-country evaluation will, in priority, assess UNICEF’s contribution to impact and outcome results for children (good/bad practices, innovations and models as well as strategies that work and can be scaled up or replicated) in terms of their relevance to the child rights and equity agenda, effectiveness, efficiency and sustainability (as defined by OECD/DAC). In addressing the evaluation questions below, the multi-country evaluation will have to establish clear linkages between impact results for children, system changes and UNICEF’s contribution. Impact Do the impact results reported by countries and territories in terms of greater inclusion and improved

quality of education really represent a change in the lives of children, in terms of the realization of their right to quality basic education and/or reduction of equity gaps?

Are these results supported by reliable and validated primary data (surveys, administrative data, evaluations, assessments)?

Have the strategies used by countries and territories led to impact results (a greater realization of

children’s right to education and significant reduction of equity gaps)? What has been UNICEF’s contribution to government’s policy formulation, implementation or fine

tuning through its work in terms of knowledge generation, evidence-based advocacy and policy advice, and technical support?

Based on an analysis of UNICEF’s influence on policy at country level – where has UNICEF’s influence

been the strongest and most effective? Is it in: a) supporting policy development; b) building capacity to implement policy; c) setting norms and standards that help monitor and fine tune implementation; evidence-based advocacy and communication for development; or e) design and pilot innovative models that can be replicated and incorporated in national or sub-national programme design?

12

Have the regional initiatives initiated by the Regional Office influenced and impacted the strategies

and interventions implemented at country level to reduce equity gaps in education?

More specifically, what has been the influence and/or impact of the Regional Office’s support and advice to Country Offices’ education strategies and interventions in terms of knowledge generation (regional position papers, technical guidance, regional studies and evaluations, study tours …), capacity development (study tours, regional education network meetings …), technical assistance (country visits, provision of external expertise…), partnerships building (regional events, leveraging influence and funding) and Communication for Development (C4D campaigns)?

How can UNICEF improve its policy influencing impact? Relevance Have UNICEF’s interventions at country level that were designed to influence inclusive policies and

system changes, been specifically targeted the most marginalized children, those children whose right to education is violated, in particular children with disabilities, Roma children, girls, children from poor rural areas, children performing below academic standards, and children with multiple disadvantages?

Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights

violation in education been relevant to prevailing education sector policies and international standards?

Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights

violation in education been relevant to the work and programmes of national partners? Effectiveness How effective have been government’s interventions in removing system bottlenecks that determined

or contributed to the exclusion of marginalized children from education (use of the UNICEF MoRES Determinant Analytical Framework)?

How effective have been UNICEF’s strategies – both at regional and country level - in contributing to

the removal of such bottlenecks? Have there been opportunities for programmatic synergies between UNICEF’s interventions and those

of its development partners that contributed to increase the effectiveness of government’s efforts to remove system bottlenecks to inclusive education?

Which are the system bottlenecks that have not been addressed by either the government or its

partners (including UNICEF), or which the government and its partners have not been able to remove? And what are the reasons for that?

More specifically, which strategies – among UNICEF’s core roles – have been the most effective in

contributing to remove system bottlenecks to inclusive education: policy advice and technical assistance, modeling, facilitating national dialogue towards child-friendly social norms, enabling

13

knowledge exchange, monitoring and evaluation, leveraging resources from the public and private sectors?

In particular, how effective has UNICEF been in influencing the formulation, adoption and enforcement

of inclusive education policies, strategies and action plans (at both national and local level) targeting the most marginalized and excluded children?

How effective has UNICEF been in monitoring and evaluating the enforcement and implementation of

such policies, strategies and action plans (particularly at local level)? How effective has UNICEF been in modeling and piloting (together with government counterparts and

other partners) inclusive education interventions that have a potential to be replicated, expanded and scaled up?

Efficiency Have UNICEF’s resources invested in support of more inclusive education policies and plans been used

in a strategic and cost-effective manner? More specifically, has UNICEF been successful in playing a catalytic role and using its meager core

resources strategically to leverage partners’ funding for inclusive education reforms? Would there have been a more cost-effective way to obtain the expected results?

How efficient has UNICEF been in modeling and piloting (together with government counterparts and


Is UNICEF properly staffed and equipped to play an effective role at country level and influence the

development and support the implementation of meaningful inclusive policies and reforms in the education sector?

Have there been gaps – and if yes, how large are they – between the design of UNICEF’s interventions

at country level and the realities of implementation? For instance, have UNICEF’s recommended strategies such as child-centered approaches resulted in a greater inclusion of marginalized children in education?

Have there been overlaps between UNICEF’s interventions and those of its development partners that have undermined the effectiveness of government’s efforts to achieve a greater inclusion of marginalized children in education?

What have been the coordination mechanisms between the government and UNICEF and its partners

– if any – and how have they contributed to the removal of system bottlenecks to inclusive education? Has UNICEF’s equity agenda and targeting of marginalized and excluded children, been aligned to

governments’ policies/reform agendas and priorities of other duty bearers and partners?

14

What is the level of awareness of the major partners with regards to UNICEF’s programme goals and strategies at country level?

If not, has UNICEF been successful in leveraging governments’ political will and financial resources to

address child rights violations and equity issues in terms of access to and outcomes of education, and focus on the inclusion of the most vulnerable groups?

Has UNICEF been successful in creating alliances, mobilizing partners, donors and other duty bearers,

and triggering national partnerships towards greater equity and inclusiveness in education? In other words, has UNICEF been successful in making governments’ reforms and partners’ priorities

more relevant to the realization of the right of all children to be included in quality learning? Sustainability Are there indications that the impact results (greater inclusion of marginalized and vulnerable children

in basic education) will or won’t be sustained over time in countries and territories where they took place?

And have the system changes that led to these impact results been sustained in countries and territories where they took place? And if not, what are the bottlenecks to the sustainability of these results?

In working towards the removal of system bottlenecks to the inclusion of out of school children in

quality learning, has UNICEF acquired a comparative advantage to contribute in a significant way to the sustainability of these results?

Human rights-based approach to programming To which extent a human-rights based approach to programming has been applied in the design and

implementation of education sector reforms and has strengthened the impact of such interventions? The Evaluation Team will be responsible to ensure that the multi-country evaluation reflects UNICEF’s human rights-based approach to programming – principles, policies and standards stated in the UNEG Guidance on Integrating human-rights and gender equality in evaluation (see link below) and complies with the organization’s commitment to gender mainstreaming as expressed in the Policy on Gender Equality and the Empowerment of Girls. http://www.uneval.org/documentdownload?doc_id=980&file_id=1294 Evaluation questions will be further refined and additional ones will be incorporated by the Evaluation Team – if required - during the inception phase. Responses to the evaluation questions will have to be pitched at a strategic and regional level so that the evaluation does not get lost in in-depth analysis of individual project activities. EVALUATION PROCESS AND METHODS

http://www.uneval.org/documentdownload?doc_id=980&file_id=1294

15

The approach followed from the onset of the evaluation will be as participative as possible. Stakeholders will participate to the evaluation through discussions, consultations, provide comments on draft documents and some will reply to the recommendations made by the evaluation in the management response. In gathering data and views from stakeholders, the evaluation team will ensure that it considers a cross-section of stakeholders with potentially diverse views to ensure the evaluation findings are as impartial (or representative) as possible.

The evaluation will employ relevant internationally agreed evaluation criteria of relevance, efficiency, effectiveness, impact, and sustainability.

The methodology should demonstrate impartiality and lack of biases by relying on a cross-section of information sources (e.g. stakeholder groups, including beneficiaries, etc.) and using a mixed methodological (e.g. quantitative, qualitative, participatory) to ensure triangulation of information through a variety of means.

The evaluation will be based on analysis of secondary data and on primary data collection. As much as possible, secondary data will be assessed during the pre-mission phase to start addressing evaluation issues and identifying the information gaps prior to the in-country mission. In view of the information already available at both country and regional levels (see list of key information sources in Annex 1) it is not expected that the Evaluation Team will need to undertake a major survey; it may be however necessary to dig into existing ones (especially the results of the recent MICS survey completed). The evaluation team will also be expected to conduct interviews and focus groups in UNICEF as well as with external stakeholders in countries and territories or at regional level (as relevant).

Inception Phase: The first step of the evaluation process will be the inception phase during which the Evaluation Team will develop an evaluation framework based on the TOR (regional theory of change, UNICEF’s regional core roles and strategies, evaluation questions). For each of the questions and most probably sub-questions, the evaluation team will develop indicators to inform the responses and identify the corresponding means of verification. On that basis, the team will develop a detailed methodology based on the key elements identified above. In addition, the Evaluation Team will assess potential limitations to the evaluation work and in particular the availability and reliability of data. A Desk Review of all possible evidence available at regional and country level in relation to impact and systems results, reduction of equity gaps and theory of change in the area of inclusive education will be undertaken. The desk review will of course not be limited to UNICEF documentation (CPDs, COARs, MTR reports, SITANs, strategy notes, studies, evaluation and survey reports (MICS) but will also cover government and partners’ documents, including external evaluations, surveys (DHS, LLHS…), assessments, studies, policy documents, strategy papers, plans of action national education plans, PRSPs, reports and publications produced by the World Bank and the EU, existing analyses of the education sector, CRC and CEDAW reports and comments/observations, evaluations and documentation of projects implemented by other partners … Survey results, administrative data or other available data sources will be verified and analyzed to confirm the assumption that changes in children’s lives (impact) and reduction of equity gaps have indeed occurred during the past decade ; evaluations and other assessment reports will be used to demonstrate how system changes and UNICEF’s contribution to them led to changes in the lives of children; and other documents will be used to explain and support the theory of change. Country visits will be carried out in the five concerned countries and territories in order to fill knowledge gaps and carry out a rigorous triangulation of the information collected through the desk review: key informant interviews and consultations will take place with education policy makers, planners and

16

administrators (officials from MOE, statistical departments, EMIS …) as well as donors (European Union, GIZ, USAID, World Bank …) and other partners/stakeholders in the field of education (UN, OSI, ISSA …). Country visits are not country level evaluations and there are no plans to have separate country reports. However, the Evaluation Team may be requested to provide a debriefing to the CO at the end of each country mission (the RO could be video-linked) and present specific findings and possible suggestions which are country specific. The Evaluation Team could also be asked to provide at the end of the mission a short memo or a PowerPoint presentation.

The possibility of sequencing the country missions will be explored. The evaluation work may begin with a “pilot” country mission in which the whole external Evaluation Team will be included, and whose results will help consolidating the evaluation framework and overall approach to the country mission. Then, the other four country missions may take place simultaneously with only one international team member, assisted by national expertise. This will help making best use of the external Evaluation Team and reducing the evaluation timeframe. Another advantage will be to validate the evaluation framework based on country realities. General considerations: The methodology of the multi-country evaluation will be in line with the United nations Evaluation Group (UNEG) Norms and Standards. UNEG Norms and Standards. The multi-country evaluation will rely, to the extent possible, on already available data. Plans for use and dissemination of the findings of the multi-country evaluation will be made at a later stage. STAKEHOLDER INVOLVEMENT During country visits, local stakeholders may be involved in the evaluation process, in particular at the planning stage as well as during the validation process. If needed, the UNICEF Country Representatives may wish to establish a Local Reference Group composed of key stakeholders, the Evaluation Team and UNICEF Education Chiefs/Specialists, in order to facilitate and guide the evaluation process. The national experts/research institutes that will be hired by the external consulting firm (with possible support from UNICEF Country Offices) will provide local technical support before and during the field missions. The involvement of national experts/research institutes will contribute to a greater ownership of the concerned countries and territories for the evaluation work. Both the Regional Office and concerned UNICEF Country Offices will be involved in the dissemination of the findings of the multi-country evaluation. National stakeholders will be associated to this process of dissemination of the results of the evaluation. EVALUATION TEAM COMPOSITION AND COMPETENCIES In view of the purpose, scope, focus of the evaluative work, the multi-country evaluation will be conducted by an external institution or consulting firm with expertise in evaluation of education programmes and projects, education statistics, education policies, education planning, quality of education and coordination of research work.

http://www.uneval.org/normsandstandards/index.jsp;jsessionid=BF5B67B9BD7B90E3998ECF1BBCD3CA5A?doc_cat_source_id=%3c%25=UnevalConst.UNEG_NORMS_STD%25%3e

17

The consulting firm/institution will have to put together a multidisciplinary team of experts – led by an Evaluation Expert - in order to cover the different aspects of the multi-country evaluation. The number of members of the Evaluation Team will be discussed based on the technical and financial proposals that will be submitted through the bidding process. The external Evaluation Team will be assisted before and during their field missions by national expertise (national experts/research institutes). The core competencies required from the external experts will be the following:

Advanced degree in Educational Sciences, Programme Planning and Management or related fields; 8-10 years of professional experience at the national and international level. Previous experience of multi-country evaluations of national education programmes, policies,

strategies and plans; experience in developing/analyzing policies to enhance equity (including gender) in education systems an asset;

Ability to work in an international environment; previous experience of working in CEE & CIS countries and territories an asset.

Excellent analytical and report writing skills. Familiarity with UNICEF’s mission and mandate an asset. Excellent mastering of English.

While it is understood that there will be a team of experts with different competencies, the specific nature of the expertise required will be discussed once technical proposals will be submitted. ACCOUNTABILITIES The Multi-Country Evaluation is a regional undertaking led by the Regional Office for CEE/CIS. At the regional level, the Reference Group (chaired by the Turkey Country Office and composed of the Regional Education Advisor and Education Chiefs/Specialists of participating countries and territories) will provide a general oversight on the evaluation work. The Regional Education Advisor and Regional Monitoring and Evaluation Advisors will provide technical advice and supervision to the external evaluation team. The Regional Office Management Committee will ensure that the evaluation process is in line with the regional evaluation guidance. The UNICEF Office of Evaluation, as well as Programme Division/Education Section (HQs), and the Innocenti Research Center (IRC) will play an advisory role at specific points of the evaluation process (review of these TOR, inception and evaluation reports …). At country level, the UNICEF Country Offices will provide the Evaluation Team with the technical assistance and logistical support required in the design, planning, and organization of the evaluation work. The UNICEF Country Offices (and Local Reference Groups, where they exist) will be responsible for organizing the field visits, meetings, consultations and interviews, for providing access to the government counterparts, donors and partners, and for coordinating the work at country level with other stakeholders. The Evaluation Team will be responsible for conducting the desk review of the project, organizing the technical preparation of the field visits, undertaking the country visits and producing the deliverables.

18

While it is not expected that vulnerable children will be requested to participate in the evaluation, the Evaluation Team will ensure that the evaluation process is ethical, in line with UNEG Ethical Guidelines. http://www.uneval.org/search/index.jsp?q=ETHICAL+GUIDELINES Possible responsibilities of the national experts/research institutes – which will be part of and work under the guidance and supervision of the Evaluation Team - could include the following: Assist in the preparatory work of the multi-country evaluation in advance of the arrival of the

international expert; Assist the international experts in the design of the questionnaires for the meetings and interviews; Collect all available documentation for the evaluation; Coordinate and support evaluation activities: field visits, meetings, consultations, interviews; Brief the international experts about key issues relevant to the national education systems; Participate in the analysis of country level data and information; Comment on the intermediate and final evaluation reports and provide inputs/recommendations; Accomplish other tasks to assist the international experts as required. The UNICEF Country Offices and local reference group may wish to organize a debriefing meeting during which the findings of the evaluation work will be presented by the Evaluation Team. The UNICEF Regional Office and concerned Country Offices will approve the final product and arrange its dissemination. PRODUCTS TO BE DELIVERED Deliverables The main deliverables will include: a) The Inception Report b) The main Evaluation Report, c) A brief Advocacy Note summarizing key findings and recommendations from the main report; and d) A Power Point Presentation of the evaluation report. The evaluation report – in both its format and content - will have to comply with the UNICEF Evaluation Report Standards, which will be made available to the Evaluation Team at the beginning of the assignment. UNEG Norms and Standards UNICEF reserves the right to withhold all or a portion of payment if performance is unsatisfactory, if work/outputs are incomplete, not delivered of for failure to meet deadlines. Structure of the Evaluation Report Structure of the Inception Report Response to the TOR Evaluation Framework Methodology Potential limitations of the evaluation according to data availability and reliability Structure of the Evaluation Report (Tentative) Title Page

http://www.uneval.org/search/index.jsp?q=ETHICAL+GUIDELINES

http://www.uneval.org/normsandstandards/index.jsp;jsessionid=BF5B67B9BD7B90E3998ECF1BBCD3CA5A?doc_cat_source_id=%3c%25=UnevalConst.UNEG_NORMS_STD%25%3e

19

Table of content List of Acronyms Acknowledgements Executive Summary Object of the Evaluation Evaluation Purpose, Objectives and Scope Evaluation Methodology Findings Conclusions and Lessons Learned Recommendations Annexes The structure of the final report will be further discussed with the Evaluation Team (during the Inception Phase).

20

Evaluation Process and Tentative Time Frame 1. Preparation Phase Terms of Reference (TOR) 03/09/12 – 05/10/12: Preparation of 1st draft TOR – Evaluation Manager3 with support of Regional

M&E Advisor; 08/10/12 – 21/12/12: Review of draft TOR – Regional/HQs M&E Quality Assurance System, including

the Reference Group; 07/01/13 – 08/02/13: Revision and preparation of 2nd draft TOR (based on comments from

regional/HQs review) - Evaluation Manager; 18/02/13 – 11/03/13: Final review and finalization of the TOR - Evaluation Manager; Regional M&E

Advisor; 05/03/13 : Final approval of the TOR – Evaluation Committee Chair; Budgeting and Evaluation Team Contracting 05/11/12 – 21/11/12: Identification of potential consulting firms/institutions (regional and global

rosters, informal networks …) – Evaluation Manager; 18/03/13 : Sending Call for Proposals to potential consulting firms/ institutions with a 3

week timeframe to respond – Evaluation Manager; 08/04/13 – 26/04/13: Review of technical and financial proposals; checking of references; review of

previous evaluation reports; phone interviews; budget preparation and review; preparation of a recommendation to the Evaluation management Committee - Evaluation Manager with support of Regional M&E Advisor;

29/04/13 – 03/05/13: Meeting of the Evaluation Management Committee; Final approval of the Budget

and Selection of the Evaluation Team – Committee Chair; 06/05/13 – 17/05/13: Contract approval and Issuance of contract to selected Evaluation Team –

Contract Review Committee and Regional HR Team.

3 Regional Education Advisor

21

2. Evaluation Phase Evaluation Launch 2-3 days : Briefing of the Evaluation Team; handing over of key documents, information

sources and contacts at both regional and country levels – Evaluation Manager and Reference Group;

: Discussion of the TOR (in particular the evaluation approach and methodology,

as well as research criteria and questions, and the format and quality of the report – Evaluation Manager and Regional M&E Advisor.

Inception Phase 1 month : Review of key documents; clarification of the theory of change provided in the

TOR; development of an Evaluation Framework and Evaluation Methodology (based on the theory of change and evaluation questions) – Evaluation Team;

: Preparation of a 1st draft Inception Report – Evaluation Team; : Review of the draft Inception Report and sharing of comments with the

Evaluation Team – Evaluation Manager, Regional M&E Advisor, Reference Group and Regional M&E Quality Assurance System;

: Revision and preparation of Final Inception Report (based on comments

received) – Evaluation Team; : Approval of Final Inception Report before next steps can take place – Evaluation

Manager, Regional M&E Advisor and Reference Group. Desk Review 11/2 month : Review of the relevant documentation (as described in the Section above on

Evaluation Process and Methods, p.11) – Evaluation Team. Country Visits 11/2 month : One week-visit to the five participating countries and territories (need to explore

the possibility of sequencing country missions as described in the Section above on Evaluation Process and Methods, p.13) – Evaluation Team.

22

Report Writing 2 months : Preparation of 1st draft Evaluation Report – Evaluation Team; : Review of 1st draft Evaluation Report – Evaluation Manager, Regional M&E

Advisor, Reference Group and Regional M&E Quality Assurance System; : Revision and preparation of 2nd draft Evaluation Report (based on the first

round of comments – Evaluation Team; : Review of 2nd draft Evaluation Report – Internal stakeholders and expected

users; : Consultation/Workshop with key internal /external stakeholders, expected

users, and Evaluation Team in order to present the main findings and discuss/validate the recommendations of the multi-country evaluation – Evaluation Manager, Regional M&E Advisor, Reference Group;

: Revision and preparation of final draft Evaluation Report (based on the second

round of comments and workshop results – Evaluation Team; : Final review/finalization of the Evaluation Report and presentation to the

Evaluation Management Committee – Evaluation Manager and Regional M&E Advisor;

: Meeting of the Evaluation Management Committee; Approval of the Final Multi-

Country Evaluation Report – Committee Chair. 3. Management Response and Communication Strategy (to be discussed with Regional Office

management) ESTIMATED RESOURCE REQUIREMENT Funding requirement can only be estimated once financial proposals will be received from potential consulting firms and institutions, which will take place in April 2013.

APPENDIX 2:

Work Plan

Evaluation phases The assignment is divided into five phases as follows (working days are presented as the sum of days of all experts per phase):

Phase Outputs

Working days

Inter-national

National

Phase 1:

Inception Phase (pre-pilot) 17 June-09 August 2013

1st Draft Inception Report (submission: 12 July 2013) 2nd Draft Inception Report

(submission: 02 August 2013)

64 8

Phase 2:

Extended document review and preparation of country visits 12 August-30 August 2013

35 13

Phase 3:

Inception Phase (finalisation of pilot) 01-13 September 2013 Country visits 01 September-05 October 2013

Debriefings to Country Offices, Memos, PowerPoint presentations

Final Inception Report (submission: 16 September 2013)

55 59

Phase 4:

Reporting Phase 07 October-09 December 2013

1st Draft Evaluation Report (submission: 18 October 2013) 2nd Draft Evaluation Report

(submission: 09 December 2013)

41 5

Phase 5:

Review, Presentation and Validation (before submitting final report) December 2013-January 2014

Consultation Workshop (16-17 January 2014)

Final Evaluation Report (submission: 22 January 2014)

25 5

TOTAL DAYS 220 90

The total of working days for this assignment is 220 international expert days + 90 national expert days. The overall period of execution of the consultancy is 7 months. Phase 1: Inception phase The Inception Phase commenced with a briefing of the evaluation team at UNICEF CEE/CIS office in Geneva, and then continued with a first document review in preparation for the further refinement of the evaluation framework as shown in Section 3 above. It continued with the submission of the 1st draft Inception Report, comprising an updated methodology (including indicators related to the evaluation questions) in light of (i) the outcomes of the briefing in Geneva, and (ii) additional findings and conclusions from the first document review. After submission of the 1st draft of the Inception Report, UNICEF reviewed the report and submitted comments to the evaluation team on 26 July for incorporation into the 2nd draft of the Inception Report. Following the submission of the 2nd draft Inception Report, UNICEF gave an official go-ahead to continue with the document review and the preparation of the country visits even though a formal approval of the Inception Report could only be expected after submission of the Final Inception Report on 16 September 2013, following the conclusion of the Kosovo (UNSCR 1244) pilot phase. The Inception Report was officially approved on 25 September. Phase 2: Extended document review and preparation of country visits From 12-30 August, the evaluation team prepared for the country visits. Taking into account the large volume of documentation to be studied (from UNICEF, partner governments, and donors), a total of 20 working days was distributed over three experts in order to adequately digest the information contained in the broad range of documents suggested for the review. The team leader designed a matrix with specific thematic areas assigned to the team members who then prepared synopses for distribution among the team. The overall focus was on the identification (and further analysis) of “all possible evidence at regional and country level in relation to impact and systems results, reduction of equity gaps and theory of change in the area of inclusive education” (cf ToR, p.15). On the basis of the extended document review, instruments (such as interview sheets and guidelines for focus group discussions) were prepared “in order to fill knowledge gaps and carry out a rigorous triangulation

of the information collected through the desk review” (cf ToR, p.16). The design of the country visits (including the accompanying consultation instruments) took into account the consultative nature of such visits which do not constitute country-level evaluations. Regional consultants made use of a similar reporting matrix with specific thematic areas related to the evaluation questions and indicators. Phase 3: Country visits Country visits were undertaken during the period 01 September until 05 October to the five countries and territories mentioned in the ToR, i.e.

Armenia;

Kosovo (UNSCR 1244);

The Former Yugoslav Republic of Macedonia;

Serbia; and

Turkey. Visits to countries and territories had a duration of 1½ weeks each, with the participation of one international education expert plus at least one regional expert from the respective country/territory/region, as further described in Section 4.2 below. We followed the suggestion provided in the ToR to begin with a “pilot” country mission in order to validate the evaluation framework and the overall approach to the country missions as such. As discussed and agreed during the briefing session in Geneva, we started with Kosovo (UNSCR 1244), being a smaller territory and also showing two specific key thematic areas (i.e. issues related to drop-out and gender parity). Due to the involvement of both international experts, they can share responsibilities, which resulted in a reduced duration of this “joint” territory visit of one week instead of one and a half. Immediately after the first territory visit, and in order to utilise the generated experiences to the utmost benefit for the remaining visits, a four-day workshop was conducted for all regional experts under the guidance of the two international education experts. The workshop familiarised the whole team with the approach and the methodology, gave sufficient room to conduct hands-on exercises regarding the consultation procedures, and allowed for a thorough reflexion on and validation of the evaluation procedures. We considered it absolutely crucial for the evaluation team – regarding both national and international experts – to be on exactly the same line, particularly in the interest of future comparability of data from the five countries and territories. After the completion of the Kosovo (UNSCR 1244) pilot visit and on the basis of validated evaluation tools (including the validated evaluation matrix), subsequent country visits were carried out sequentially to TFYRoM and Armenia (Education Expert I) and Turkey and Serbia (Education Expert II), whereby every expert spent 3 weeks (i.e. 1½ weeks – or 7 working days – per country) in the field. The countries and territories assigned for each expert ensured a similarity in regional distribution and also regarding the key thematic areas. Regional consultants accompanied the international consultants and provided the necessary local and thematic expertise according to their experience. Although no separate country reports were foreseen at the end of the country visits (cf ToR p.16), the team gave debriefings to the country offices with a focus on country-specific issues emerging from the consultations. This was supported by PowerPoint presentations, which were also shared with the Regional Office and can be made available to other relevant stakeholders upon request. Phase 4: Reporting phase, presenting/validating report and submitting final version The reporting phase commenced with a one-week team workshop of the international team members immediately after the field visits. At the workshop, experiences from the country visit were discussed, further analysed and harmonised for presentation in the draft evaluation report. Again, the team workshop agreed on the distribution of topics and responsibilities in the evaluation report in line with UNICEF Evaluation Report Standards. After submission of the first draft of the Evaluation Report, UNICEF reviewed the report and submitted comments to the evaluation team on 22 November 2013 for incorporation into the second draft of the Evaluation Report which has been submitted on 09 December 2013. After submitting consolidated comments to the evaluation team by 09 January 2014, the team will prepare a response to the comments which will then be further discussed during the presentation of the 2nd Draft Evaluation Report at a consultation workshop with key internal and external stakeholders schedule to take place on 16 January 2014 in Geneva. The Final Evaluation Report will take into account the outcomes of the Consultation Workshop and will be submitted on 23 January 2014. Phase 5: Presenting/validating report and submitting final version

The purpose of the Consultation Workshop is to present the main findings (as presented in the 2nd Draft Evaluation Report) to key stakeholders and expected users of the report, and to discuss the recommendations of the evaluation. In close collaboration with UNICEF, the workshop will be designed in such a way that the discussions culminate in a validation of the evaluation process and its findings, in order to enhance overall ownership by stakeholders and prospective users. Following the workshop, there will be an additional phase for a final review by UNICEF. Based on the outcomes of the workshop and the additional comments received from UNICEF, the evaluation team will prepare a Final Evaluation Report. After submission, UNICEF will review the report and eventually approve it at a meeting of the Evaluation Management Committee comprising the relevant officers as nominated by UNICEF. Roles and responsibilities of team members The international core team was responsible for the overall design and implementation of the evaluation. It comprises two education experts (including the team leader) and one evaluation scientist sub-team (two members) who collaborated with the team leader during the development of the overall analysis grid (together with development of the related data collection instruments), and also during the data analysis. He furthermore took part in the document analysis and advised the other team members in the application of the required methodological steps. Finally, he was responsible for the quantitative data analyses. The team leader took responsibility for the overall quality assurance of the evaluation. Field visits were carried out by the two education experts who were supported by a regional evaluation team with experts from the targeted countries and territories. The regional experts worked under the guidance and supervision of the international core team. They assisted in preparing the country visits, validated the school selections, and provided valuable inputs regarding the respective local contexts and their specific key thematic areas. In particular, they supported the evaluation activities in the field (meetings, consultations, interviews, focus group discussions) and participated in the analysis of country data and information. The expertise of the regional team reflects the key thematic areas of the evaluation. Logistical issues The following services were part of the backstopping package provided by PROMAN:

Contractual relations with the Client;

Organisation of the evaluation activities, field trips, presentations and (de)briefings;

Quality control of the services and reporting through PROMAN’s highly qualified in-house expertise;

Timely, discretely and diplomatic handling of any friction that may arise between the different participants to the evaluation;

Financial management. In addition, the team was in touch with UNICEF field offices for support during the time of the survey. Such support included the preparation and arrangement of meetings with key stakeholders, and assistance with transport where possible. Support provided by UNICEF Offices extended to translation and interpretation services. In the former Yugoslav Republic of Macedonia, an independent translator was contracted through PROMAN for the school visits.

Detailed Work Plan – UNICEF CEE/CIS, RKLA 4: Inclusive Education

Date Activity Place

Experts Core Team National/Regional

TL Eval. Exp.

Educ. Exp.

KV RS/MK

AM TR

JUN

E 2

013

15/06/13

16/06/13

17/06/13 Contract approval / Contract issuance

18/06/13

19/06/13 Travel to UNICEF CEE/CIS office, Geneva - Briefing UNICEF, Geneva 1 1 1

20/06/13 Briefing of evaluation team UNICEF, Geneva 1 1 1

21/06/13 Team meeting / Return travel to home base UNICEF, Geneva 1 1 1

22/06/13

23/06/13

24/06/13 Document Review I Home bases 1 1 1 1 1 1 1

25/06/13 Document Review I Home bases 1 1 1 1 1 1 1

26/06/13 Document Review I Home bases 1 1 1



29/06/13

30/06/13

JUL

Y 2

013


02/07/13 Team workshop I: Evaluation Framework Köln-Troisdorf 1 1 1



05/07/13 Drafting Inception Report Home bases 1 1 1

06/07/13

07/07/13





12/07/13 Finalising and submitting Draft Inception Report 1 1 1

13/07/13

14/07/13

15/07/13 Review of draft Inception Report





20/07/13

21/07/13





26/07/13 Submission of comments to Evaluation Team

27/07/13

28/07/13

29/07/13 Drafting Final Inception Report Home bases 1 1 1

30/07/13 Drafting Final Inception Report Home bases 1 1 1

31/07/13 Drafting Final Inception Report Home bases 1 1

AU

GU

ST

201

3

01/08/13 Drafting Final Inception Report Home bases 1

02/08/13 Finalising and submitting Final Inception Report 1

03/08/13

04/08/13

05/08/13 Review of final Inception Report





10/08/13

11/08/13

12/08/13 Document Review II 1 1 1 1 1 1 1

13/08/13 Document Review II 1 1 1 1 1 1 1

14/08/13 Document Review II 1 1 1 1

Date Activity Place


TL Eval. Exp.

Educ. Exp.

KV RS/MK

AM TR

15/08/13 Document Review II 1 1 1

16/08/13 Document Review II 1 1 1

17/08/13

18/08/13

19/08/13 Document Review II 1 1

20/08/13 Document Review II 1 1

21/08/13 Document Review II 1

22/08/13 Document Review II 1

23/08/13 Approval of final Inception Report

24/08/13

25/08/13

26/08/13 Preparation of Territory Visit - Kosovo (UNSCR 1244) Home bases 1 1 1 1





31/08/13

SE

PT

EM

BE

R 2

013

01/09/13 Travel to Territory Kosovo (UNSCR 1244) 1 1

02/09/13 Territory visit - Kosovo (UNSCR 1244) Kosovo (UNSCR 1244)

1 1 1


1 1 1


1 1 1


1 1 1


1 1 1

07/09/13

08/09/13 1 1

09/09/13 Team Training workshop Kosovo (UNSCR 1244)

1 1 1 1 1 1


1 1 1 1 1 1


1 1 1 1 1 1


1 1 1 1 1 1

13/09/13 Wrap-up of pilot Territory visit Kosovo (UNSCR 1244)

1 1 1 1 1 1

Preparation of updated evaluation matrix

14/09/13 Travel Kosovo (UNSCR 1244)

1 1 1

15/09/13 FYROM/Turkey

16/09/13 Country visit - FYROM and Turkey FYROM/Turkey 1 1 1 1









25/09/13 Travel to next country Travel 1 1 1 1

26/09/13 Country visit - Armenia and Serbia Armenia/Serbia 1 1 1 1


28/09/13 Armenia/Serbia

29/09/13 Armenia/Serbia


OC

TO

BE

R 2

013





05/10/13 1 1

06/10/13

07/10/13 Team workshop II: 1st draft Evaluation Report Frankfurt 1 1 1





12/10/13

13/10/13

14/10/13 Finalising 1st draft Evaluation Report Home bases 1 1 1 1 1 1

15/10/13 Finalising 1st draft Evaluation Report Home bases 1 1 1

Date Activity Place


TL Eval. Exp.

Educ. Exp.

KV RS/MK

AM TR

16/10/13 Finalising 1st draft Evaluation Report Home bases 1 1

17/10/13 Finalising 1st draft Evaluation Report Home bases 1 1

18/10/13 Finalising and submitting 1st draft Evaluation Report Home bases 1 1

19/10/13

20/10/13

21/10/13 Review of 1st draft Evaluation Report





26/10/13

27/10/13





NO

VE

MB

ER

201

3


02/11/13

03/11/13






09/11/13

10/11/13






16/11/13

17/11/13






23/11/13

24/11/13

25/11/13 Drafting 2nd draft Evaluation Report Home bases 1

26/11/13 Drafting 2nd draft Evaluation Report Home bases 1 1

27/11/13 Drafting 2nd draft Evaluation Report Home bases 1 1 1

28/11/13 Drafting 2nd draft Evaluation Report Home bases 1 1 1


30/11/13

DE

CE

MB

ER

201

3

01/12/13






07/12/13

08/12/13

09/12/13 Drafting and submitting 2nd draft Evaluation Report

10/12/13 Review of 2nd draft Evaluation Report




14/12/13

15/12/13



Date Activity Place


TL Eval. Exp.

Educ. Exp.

KV RS/MK

AM TR




21/12/13

22/12/13

23/12/13

24/12/13

25/12/13 Christmas

26/12/13

27/12/13

28/12/13

29/12/13

30/12/13

31/12/13

JAN

UA

RY

201

4

01/01/14 New Year

02/01/14

03/01/14

04/01/14

05/01/14

06/01/14

07/01/14 Orthodox Christmas

08/01/14

09/01/14


11/01/14

12/01/14

13/01/14

14/01/14 Addressing Comments in preparation of Workshop Home bases 1 0.5 1 1 1 1 1

15/01/14 Travel 1 1 1

16/01/14 Presentation/validation of 2nd report Geneva 1 1 1

17/01/14 Presentation/validation of 2nd report Geneva 1 1 1

18/01/14

19/01/14

20/01/14 Reporting Home bases 1 1 1

21/01/14 Reporting Home bases 1 0.5 1

22/01/14 Reporting Home bases 1 1

23/01/14 Reporting Home bases 1 1

24/01/14 Reporting Home bases 1

25/01/14

26/01/14

27/01/14 Reporting Home bases





FE

BR

201

4

01/02/14

02/02/14

03/02/14 04/02/14

0

Reporting Home bases




07/02/14 Submitting 3rd draft Evaluation Report

TOTAL INPUTS 88 55 77 21 28 20 21

220 90

APPENDIX 3:

List of documents consulted

References in this table shown under “Path“ are based on the original file directory submitted to the evaluation team at the onset of the evaluation, in order for UNICEF RO to easily track the document source. The original set of documents was later expanded by the team; these documents are printed in italics. Document types include the following: Guideline: Relevant for evaluation design and reporting, general guidelines, standards, norms, etc., also UNICEF Programming, CfS Evaluation: Project/Programme evaluation reports Policy: Policy, Position or Strategy paper or CPAP, CPD, UNDAF Study: (Quantitative) Study or Assessment Report: Project/Programme reports, e.g. Progress Reports Gov: Government document/publication Other: Other types of documents apart from the 5 mentioned above (Matrix, List of Indicators, etc.)

No. Title Path Doc Type

1 2013 GEROS Draft eval Reports template \Regional Office\Guidelines & Programme Docs\Evaluation Templates Guideline

2 2013 Regional M&E facility draft inception report review template \Regional Office\Guidelines & Programme Docs\Evaluation Templates Guideline

3 UNEG 2005 Norms EN \Regional Office\Guidelines & Programme Docs\Monitoring & Evaluation Guideline

4 UNEG 2005 Standards EN \Regional Office\Guidelines & Programme Docs\Monitoring & Evaluation Guideline

5 UNEG 2008 Ethics Guidelines \Regional Office\Guidelines & Programme Docs\Monitoring & Evaluation Guideline

6 UNEG 2011 HRGE Handbook \Regional Office\Guidelines & Programme Docs\Monitoring & Evaluation Guideline

7 UNEG Norms for Evaluation \Regional Office\Guidelines & Programme Docs\Monitoring & Evaluation Guideline

8 UNEG Standards for Evaluation \Regional Office\Guidelines & Programme Docs\Monitoring & Evaluation Guideline

9 Pilot Project Criteria \Regional Office\Guidelines & Programme Docs\Programming in UNICEF Guideline

10 Reaching the Marginalized \Regional Office\Guidelines & Programme Docs\Programming in UNICEF Guideline

11 Report on Pilot Projects \Regional Office\Guidelines & Programme Docs\Programming in UNICEF Guideline

12 Scaling Up-Key Points \Regional Office\Guidelines & Programme Docs\Programming in UNICEF Guideline

13 Short Guide PPPM \Regional Office\Guidelines & Programme Docs\Programming in UNICEF Guideline

14 Determinant Analysis Framework \Regional Office\MoRES Guideline

15 Developing CFS Standards_Final Report_12.06.10 \Quality Education standards Guideline

16 Regional CfS Compilation Study - Final Report \Regional Office\Regional Evaluations\CFS Evaluation

17 GE Final Evaluation Report_English only_FINAL VERSION_back-to-back_Volume1 \Regional Office\Regional Evaluations\Global Education Evaluation

18 GE Final Evaluation Report_English only_FINAL VERSION_back-to-back_Volume2 \Regional Office\Regional Evaluations\Global Education Evaluation

19 Regional CfS Compilation Study - Final Report \Regional Office\Regional Work in Education\CFS Evaluation

20 OOSCI CEE-CIS Regional Report final draft 17_June2012_wcomments ET \Regional Office\Regional Work in Education\Out of School Children Report

21 UNICEF_Right Children Disabilities_En_Web \Regional Office\Regional Work in Education\Out of School Children\Children with Disabilities Policy


22 Roma Education Position Paper - Final Draft \Regional Office\Regional Work in Education\Out of School Children\Roma Children Policy

23 UNICEF_PISA_WEB - 18 June 09 \Regional Office\Regional Work in Education\Quality of Education\Learning Outcomes\2006 Study

24 UNICEF Equity in Leaning_2009 PISA Results \Regional Office\Regional Work in Education\Quality of Education\Learning Outcomes\2009 Study

25 Developing CFS Standards_Final Report_12.06.10 \Regional Office\Regional Work in Education\Quality of Education\Standards Policy

26 Education Strategy Note - 21 January \Regional Office\Regional Work in Education\Regional Education Strategy Policy

27 Regional Position Paper - Final Draft - 19th December 2008 \Regional Office\Regional Work in Education\Regional Education Strategy Policy

28 1301 Generic TOC for RKLA \Regional Office\RKLA Guideline

29 120405 CEE_CIS Key Equity Results For Children TECHNICAL INTERNAL ZERO draft COMPLETE SET \Regional Office\RKLA Guideline

30 130608 CEECIS RKLA Table \Regional Office\RKLA Guideline

31 RKLA4_Prospective Approach_Draft Concept Note - v.7May2013 \Regional Office\RKLA\RKLA 4 Guideline

32 2003 AR Armenia \Countries\Armenia\Annual Reports\2003 Report

33 Armenia 2004 Annual Report \Countries\Armenia\Annual Reports\2004 Report

34 Armenia Annual Report 05 \Countries\Armenia\Annual Reports\2005 Report

35 Armenia_ annex A_AR 2006 \Countries\Armenia\Annual Reports\2006 Report

36 Armenia_ AR 2006 - final \Countries\Armenia\Annual Reports\2006 Report

37 Armenia annual report 2007 final \Countries\Armenia\Annual Reports\2007 Report

38 Armenia MTSP indicators 2007 \Countries\Armenia\Annual Reports\2007 Other

39 Armenia Annual Report 2008 \Countries\Armenia\Annual Reports\2008 Report

40 Armenia Baseline 2008 \Countries\Armenia\Annual Reports\2008 Study

41 Armenia 2009 Annual Report \Countries\Armenia\Annual Reports\2009 Report

42 Armenia AR 2010 \Countries\Armenia\Annual Reports\2010 Report

43 Armenia \Countries\Armenia\Annual Reports\2011 Report

44 Armenia \Countries\Armenia\Annual Reports\2012 Report

45 Armenia CPAP matrix 2010-2015 \Countries\Armenia\CPAP Other

46 Armenia CPAP narrative 2010-2015 \Countries\Armenia\CPAP Policy

47 Armenia CPD 2005-2009 \Countries\Armenia\CPD Policy

48 Armenia CPD 2010-2015 \Countries\Armenia\CPD Policy

49 Armenia CPD results matrix 2010-2015 \Countries\Armenia\CPD Other

50 Armenia CPD results matrix 2005-2009 \Countries\Armenia\CPD Other

51 Armenia 2005 DHS \Countries\Armenia\Data\DHS Study

52 Armenia DHS 2010 \Countries\Armenia\Data\DHS Study

53 CFS Assessment \Countries\Armenia\Evaluations Study


54 Evaluation IE policies programmes \Countries\Armenia\Evaluations Evaluation

55 School Wastage Study Focusing on Student Absenteeism in Armenia \Countries\Armenia\Studies Study

56 UNICEF-Report ENGfinal \Countries\Armenia\Studies Study

57 Armenia UNDAF Matrix 2010-2015 \Countries\Armenia\UNDAF Policy

58 Kosovo [UNSCR 1244] Annual Report 2003 \Countries\Kosovo\Annual Reports\2003 Report

59 Kosovo [UNSCR 1244] 2004 ANNUAL REPORT \Countries\Kosovo\Annual Reports\2004 Report

60 Kosovo [UNSCR 1244] 2005 ANNUAL REPORT Final \Countries\Kosovo\Annual Reports\2005 Report

61 Kosovo [UNSCR 1244]_AR 2006 \Countries\Kosovo\Annual Reports\2006 Report

62 Kosovo [UNSCR 1244] 2007 Annual Report \Countries\Kosovo\Annual Reports\2007 Report

63 Kosovo [UNSCR 1244] Baseline 2007 \Countries\Kosovo\Annual Reports\2007 Study

64 Kosovo [UNSCR 1244] Annual report 2008 final_Kosovo [UNSCR 1244] \Countries\Kosovo\Annual Reports\2008 Report

65 Kosovo [UNSCR 1244] Baseline 2008_Annex A_final \Countries\Kosovo\Annual Reports\2008 Other

66 Kosovo [UNSCR 1244] - Annual Report 2009 \Countries\Kosovo\Annual Reports\2009 Report

67 Kosovo [UNSCR 1244] Annual Report 2011_FINAL_31Jan2011 \Countries\Kosovo\Annual Reports\2011 Report

68 Kosovo [UNSCR 1244] \Countries\Kosovo\Annual Reports\2012 Report

69 Kosovo [UNSCR 1244] KPAP narrative 2011-2015 \Countries\Kosovo\CPAP Policy

70 Kosovo [UNSCR 1244] KPAP Results Matrix 2011-2015 \Countries\Kosovo\CPAP Other

71 Kosovo [UNSCR 1244] CPD 2011-2015 \Countries\Kosovo\CPD Policy

72 Final Evaluation Report Nov04 without Pictures \Countries\Kosovo\Evaluations Evaluation

73 ELDS_report_ENG_for WEB \Countries\Kosovo\Other Publications Report

74 Eng - Justice \Countries\Kosovo\Studies Study

75 FINAL Kosovo [UNSCR 1244] Education Strategic Plan (EN) \Countries\Kosovo\ESP & MTPEF Gov

76 K.Kurrikules_ENG_web \Countries\Kosovo\ESP & MTPEF Gov

77 UNICEF FINAL QUALITY EDUCATION MINORITY REPORT 2006 \Countries\Kosovo\Studies Study

78 RAE Strategy (English)[1] \Countries\Kosovo\ESP & MTPEF Gov

79 OOSC-Kosovo [UNSCR 1244]-Combined-Deliverable \Countries\Kosovo\Out of School Children Study

80 ROOSCAI Case study Kosovo [UNSCR 1244] 2013-04-23 \Countries\Kosovo\Out of School Children Study

81 Annex 02 - Baseline Study (Final Draft) \Countries\Kosovo\Studies Study

82 Joined Hands_ENG_web \Countries\Kosovo\Studies Study

83 2003 AR FYROM \Countries\Macedonia\Annual Reports\2003 Report

84 [The former Yugoslav Republic of] Macedonia COAR 2004 \Countries\Macedonia\Annual Reports\2004 Report

85 [The former Yugoslav Republic of] Macedonia Annual Report 05 \Countries\Macedonia\Annual Reports\2005 Report


86 [The former Yugoslav Republic of] Macedonia Annex A COAR 2006 [The former Yugoslav Republic of] Macedonia \Countries\Macedonia\Annual Reports\2006 Other

87 [The former Yugoslav Republic of] Macedonia MTSP baseline questionnaire 28 December \Countries\Macedonia\Annual Reports\2006 Other

88 [The former Yugoslav Republic of] Macedonia_AR 2006 28 December \Countries\Macedonia\Annual Reports\2006 Report

89 [The former Yugoslav Republic of] Macedonia_ Annual Report 07 \Countries\Macedonia\Annual Reports\2007 Report

90 [The former Yugoslav Republic of] Macedonia_AnnRep 07 Questionnaire \Countries\Macedonia\Annual Reports\2007 Other

91 [The former Yugoslav Republic of] Macedonia Annex A- MTSP Monitoring checklist_[The former Yugoslav Republic of] Macedonia 2008 \Countries\Macedonia\Annual Reports\2008 Other

92 [The former Yugoslav Republic of] Macedonia Final Annual report 2008 \Countries\Macedonia\Annual Reports\2008 Report

93 [The former Yugoslav Republic of] Macedonia Annual Report 2009 Final \Countries\Macedonia\Annual Reports\2009 Report

94 [The former Yugoslav Republic of] Macedonia AR 2010 \Countries\Macedonia\Annual Reports\2010 Report

95 [The former Yugoslav Republic of] Macedonia \Countries\Macedonia\Annual Reports\2011 Report

96 [The former Yugoslav Republic of] Macedonia \Countries\Macedonia\Annual Reports\2012 Report

97 [The former Yugoslav Republic of] Macedonia CPAP Matrix 2010-2015 \Countries\Macedonia\CPAP Other

98 [The former Yugoslav Republic of] Macedonia CPAP narrative 2010-2015_final APPROVED \Countries\Macedonia\CPAP Policy

99 [The former Yugoslav Republic of] Macedonia CPD 2005-2009 \Countries\Macedonia\CPD Policy

100 [The former Yugoslav Republic of] Macedonia CPD 2010-2015 \Countries\Macedonia\CPD Policy

101 [The former Yugoslav Republic of] Macedonia CPD result matrix 2010-2015 \Countries\Macedonia\CPD Other

102 IAL Evaluation 2003 \Countries\Macedonia\Evaluations Evaluation

103 INTERETHNIC BASELINE 2010 \Countries\Macedonia\Evaluations Study

104 INTERETHNIC Final Evaluation 2012 \Countries\Macedonia\Evaluations Evaluation

105 INTERETHNIC Mid-term Evaluation2011 \Countries\Macedonia\Evaluations Evaluation

106 PREVENTION OF VIOLENCE Evaluation 2012 \Countries\Macedonia\Evaluations Evaluation

107 ROMA EDUCATION CEB DEP FINAL EVALUATION 2008 \Countries\Macedonia\Evaluations Evaluation

108 CFS BASELINE_STUDY 2007 \Countries\Macedonia\Studies Study

109 CFS Case Study 2009 \Countries\Macedonia\Studies Study

110 INCLUSIVE STUDY 2010 \Countries\Macedonia\Studies Study

111 LITERACY BASELINE 2009 \Countries\Macedonia\Studies Study

112 LITERACY EFFECTIVENESS OF TRAINING IN [THE FORMER YUGOSLAV REPUBLIC OF] MACEDONIA European Education [The former Yugoslav Republic of] Macedonia Article \Countries\Macedonia\Studies Study

113 NUMERACY Mathematics BASELINE 2009 \Countries\Macedonia\Studies Study

114 NUMERACY Mathematics PROGRESS ASSESSMENT 2012 \Countries\Macedonia\Studies Study

115 [The former Yugoslav Republic of] Macedonia UNDAF matrix 2010-2015 \Countries\Macedonia\UNDAF Policy

116 2003 AR Serbia and Montenegro \Countries\Serbia\Annual Reports\2003 Report


117 Serbia and Montenegro - 2004 Annual Report \Countries\Serbia\Annual Reports\2004 Report

118 Serbia and Montenegro Annual Report 05 \Countries\Serbia\Annual Reports\2005 Report

119 S&M_AR 2006 final Belgrade \Countries\Serbia\Annual Reports\2006 Report

120 Serbia-2007 Annual Report \Countries\Serbia\Annual Reports\2007 Report

121 AR 2008 Serbia, final version \Countries\Serbia\Annual Reports\2008 Report

122 AR_Annex A, final version \Countries\Serbia\Annual Reports\2008 Report

123 AR 2009 Serbia_final \Countries\Serbia\Annual Reports\2009 Report

124 Serbia AR 2010 \Countries\Serbia\Annual Reports\2010 Report

125 Serbia \Countries\Serbia\Annual Reports\2011 Report

126 Serbia \Countries\Serbia\Annual Reports\2012 Report

127 Serbia CPAP narrative 2011-2015 \Countries\Serbia\CPAP Policy

128 Serbia CPAP result matrix 2011-2015 \Countries\Serbia\CPAP Other

129 Serbia CPD 2011-2015 \Countries\Serbia\CPD Policy

130 Serbia CPD result matrix 2011-2015 \Countries\Serbia\CPD Other

131 Serbia_MICS_2005(1) \Countries\Serbia\Data\MICS 3 Study

132 MICS4_Key_Highlights \Countries\Serbia\Data\MICS 4 Study

133 MICS4_Serbia_FinalReport_Eng \Countries\Serbia\Data\MICS 4 Study

134 Serbia_infographic(single3) \Countries\Serbia\Data\MICS 4 Study

135 Active learning in Serbia and Montenegro \Countries\Serbia\Evaluations Evaluation

136 AL - Summary for Reporting on MTRs & Evaluations \Countries\Serbia\Evaluations Evaluation

137 UNICEF DEC Evaluation Final Report \Countries\Serbia\Evaluations Evaluation

138 Serbia UNDAF matrix 2011-2015 \Countries\Serbia\UNDAF Policy

139 2003 AR Turkey \Countries\Turkey\Annual Reports\2003 Report

140 Turkey Annual Report 2004 \Countries\Turkey\Annual Reports\2004 Report

141 Turkey Annual Report 05 \Countries\Turkey\Annual Reports\2005 Report

142 Turkey_ AR 2006 \Countries\Turkey\Annual Reports\2006 Report

143 Turkey_Office MTSP baseline questionnaire AR 2006 \Countries\Turkey\Annual Reports\2006 Other

144 Turkey ANNUAL REPORT 2007 ANNEX A - MTSP QUESTIONNAIRE \Countries\Turkey\Annual Reports\2007 Other

145 Turkey ANNUAL REPORT 2007 TEXT \Countries\Turkey\Annual Reports\2007 Report

146 TURKEY Annex A 2008 \Countries\Turkey\Annual Reports\2008 Report

147 TURKEY ANNUAL REPORT 2008 FINAL \Countries\Turkey\Annual Reports\2008 Report

148 2009 Annual Report FINAL \Countries\Turkey\Annual Reports\2009 Report


149 Turkey AR 2010 \Countries\Turkey\Annual Reports\2010 Report

150 Turkey \Countries\Turkey\Annual Reports\2011 Report

151 Turkey \Countries\Turkey\Annual Reports\2012 Report

152 Turkey CPAP Narrative 2011-2015 \Countries\Turkey\CPAP Report

153 Turkey CPAP Result Matrix 2011-2015 \Countries\Turkey\CPAP Other

154 CPD 2010-PL.6-Turkey-ODS-English \Countries\Turkey\CPD Report

155 Turkey CPD 2006-2010 \Countries\Turkey\CPD Report

156 Turkey CPD 2011-2015 \Countries\Turkey\CPD Report

157 Turkey DHS 2003 \Countries\Turkey\Data\DHS Study

158 GE_2005_Midterm_Report \Countries\Turkey\Evaluations Evaluation

159 GE_Evaluation-2004 \Countries\Turkey\Evaluations Evaluation

160 GEC_Final_Evaluation_Report_FINAL_VERSION \Countries\Turkey\Evaluations Evaluation

161 BasicPolicypaper_for_Edu.Tech.Dep \Countries\Turkey\Other Publications Policy

162 EducationMonitoringReport2008_ExecSum \Countries\Turkey\Other Publications Study

163 e-school \Countries\Turkey\Other Publications Policy

164 GirlsEducationPractices \Countries\Turkey\Other Publications Study

165 M&E_documentation-Education \Countries\Turkey\Other Publications Policy

166 OOSCTurkey_Draft6_Sept12_V01 \Countries\Turkey\Out of School Children Study

167 Provincial_Analysis-2005 \Countries\Turkey\Out of School Children Other

168 Devamsizlik_ArastirmaRaporu_IngilizceOzet_Final \Countries\Turkey\Studies Study

169 EIR2009_ExecSum_English_0 \Countries\Turkey\Studies Study

170 ERI.civilmonitoring \Countries\Turkey\Studies Study

171 ERI.educationofgirls \Countries\Turkey\Studies Study

172 ERI.vocationaleducation \Countries\Turkey\Studies Study

173 ERI_Rights_MUFR_ExecSum \Countries\Turkey\Studies Study

174 IlkogretimdenOrtaogretimeGecis_ArastirmaRaporu_IngilizceOzet_Final \Countries\Turkey\Studies Study

175 IlkogretimKurumlarininMaliYonetimi_ArastirmaRaporu_IngilizceOzet_Final \Countries\Turkey\Studies Study

176 RESEARCH_ON_LATE_ENROLMENT \Countries\Turkey\Studies Study

177 Turkey UNDAF matrix 2011-2015 \Countries\Turkey\UNDAF Policy

178 MICS 2011 \Countries\Macedonia\Data\MICS 4 Study

179 Learning Achievement in the CEE CIS_Eng \Regional Office\Regional Work in Education\Quality of Education\Learning Outcomes Other

180 Clare.Miske.Patel 2012_Child Rights and Quality Education \Regional Office\Regional Work in Education\CFS Other


181 Poghosyan_Inclusive Education In Armenia_2009 \Countries\Armenia\Other Publications Other

182 Armenia Educational national plan 01-05 \Countries\Armenia\ESP & MTPEF Gov

183 MTEPF Armenia 12-14 final version \Countries\Armenia\ESP & MTPEF Gov

184 MTPEF Armenia 04-06 \Countries\Armenia\ESP & MTPEF Gov



187 National Plan of Action Armenia for the Protection of the Rights of the Child 04-15 \Countries\Armenia\ESP & MTPEF Gov

188 Aide memoire from Joint Annual Review 2013 of the Education Sector in Kosovo [UNSCR 1244]_eng \Countries\Kosovo\ESP & MTPEF Gov

189 Kosovo [UNSCR 1244] Annual Report on Donor Activities 2009 \Countries\Kosovo\ESP & MTPEF Gov

190 Kosovo [UNSCR 1244] Education Strategic Plan 11-16 \Countries\Kosovo\ESP & MTPEF Gov

191 Kosovo [UNSCR 1244] Government Programme 11-14 \Countries\Kosovo\ESP & MTPEF Gov

192 Kosovo [UNSCR 1244] MTEF 09-11 \Countries\Kosovo\ESP & MTPEF Gov



195 Kosovo [UNSCR 1244] Public Expenditure and Financial Accountability (PEFA) 2009 \Countries\Kosovo\ESP & MTPEF Gov

196 Kosovo [UNSCR 1244] Public Expenditure and Financial Accountability (PEFA) 2011 \Countries\Kosovo\ESP & MTPEF Gov

197 Kosovo [UNSCR 1244]_Education & Science Sectoral Progress Report 2012_eng \Countries\Kosovo\ESP & MTPEF Gov

198 WB_Kosovo [UNSCR 1244] Public Expenditure Review 2010 \Countries\Kosovo\ESP & MTPEF Gov

199 WB_Kosovo [UNSCR 1244]_MTPEP 2002 \Countries\Kosovo\ESP & MTPEF Gov

200 [The former Yugoslav Republic of] Macedonia Fiscal Strategy 10_12 \Countries\Macedonia\ESF & MTPEF Gov

201 [The former Yugoslav Republic of] Macedonia National Report on the Implementation of the UN Action Plan – “World Fit for Children” 2006 \Countries\Macedonia\ESF & MTPEF Gov

202 [The former Yugoslav Republic of] Macedonia_National Report on the Follow-up of the World Summit for children \Countries\Macedonia\ESF & MTPEF Gov

203 [The former Yugoslav Republic of] Macedonia_National_Strategy_Dev_Education 05-15 \Countries\Macedonia\ESF & MTPEF Gov

204 OECD_[The former Yugoslav Republic of] Macedonia Public Expenditure Management System 2008 \Countries\Macedonia\ESF & MTPEF Gov

205 OECD_[The former Yugoslav Republic of] Macedonia Public Expenditure Management System 2009 \Countries\Macedonia\ESF & MTPEF Gov

206 Economic and fiscal programme of the Republic of Serbia 12-14 \Countries\Serbia\ESP & MTPEF Gov

207 Serbia Common Action Plan for Advancement of Roma Education \Countries\Serbia\ESP & MTPEF Gov

208 Serbia Fiscal Strategy 13-15 \Countries\Serbia\ESP & MTPEF Gov

209 Serbia Plan of Action for Children 2004 \Countries\Serbia\ESP & MTPEF Gov

210 Serbia Public Expenditure and Financial Accountability 2010 \Countries\Serbia\ESP & MTPEF Gov

211 Serbia Strategy for Improvement of Roma Status 2010 \Countries\Serbia\ESP & MTPEF Gov

212 2011 National Report of Turkey (to EC) \Countries\Turkey\ESP & MTPEF Gov


213 Basic Education in Turkey 2005 \Countries\Turkey\ESP & MTPEF Gov

214 Turkey Medium Term Programme 11-13 \Countries\Turkey\ESP & MTPEF Gov

215 Turkey Ninth Development Plan 07-13 \Countries\Turkey\ESP & MTPEF Gov

216 Turkey Short_Term_Action_Plan 03-04 \Countries\Turkey\ESP & MTPEF Gov

217 List of interventions EDU [the former Yugoslav Republic of] Macedonia final \Countries\Macedonia\Other Publications Other

218 Schools Out_English \Countries\Kosovo\Studies Study

219 Final Report IE Evaluation \Countries\Armenia\Evaluations Evaluation

220 TEFA eval.21-11 \Countries\Armenia\Evaluations Evaluation

221 TEFA III Final evaluation report 20130102 \Countries\Armenia\Evaluations Evaluation

222 L4 Indicators_Education and budget \Countries\Kosovo\Other Publications Other

223 UNICEF KAOR 2012_Annex A \Countries\Kosovo\Other Publications Other

224 Edu RWP 2012-13 \Countries\Macedonia\UNDAF Other

225 EDU RWP 2010-2011 \Countries\Macedonia\UNDAF Other

226 CPAP_finalENG _2 Feb 2010 \Countries\Macedonia\CPAP Other

227 Local cofinancing \Countries\Macedonia\Other Publications Other

228 Social Budget Analysis [The former Yugoslav Republic of] Macedonia \Countries\Macedonia\Other Publications Other

229 VIOLENCE POLICY ENG \Countries\Macedonia\ESF & MTPEF Gov

230 90932_OSCE_2012 Budget Development Process \Countries\Kosovo\Other Publications Other

231 MTEF-2010-2012_ENGpdf[1] \Countries\Kosovo\ESP & MTPEF Gov

232 Donor Coordination \Countries\Kosovo\Other Publications Other

233 12 03 23 JAR Background Paper, final \Countries\Kosovo\ESP & MTPEF Gov

234 Consolidated report_FINAL _Kosovo [UNSCR 1244] \Countries\Kosovo\Other Publications Other

235 Donor report_Dutch_FINAL_ Kosovo [UNSCR 1244] \Countries\Kosovo\Other Publications Other

236 Donor Report_ Kosovo [UNSCR 1244]_final Ellen edited \Countries\Kosovo\Other Publications Other

237 Dutch Report Kosovo [UNSCR 1244] Final \Countries\Kosovo\Other Publications Other

238 Kosovo [UNSCR 1244]_netherland-final \Countries\Kosovo\Other Publications Other

239 Netherland report Kosovo [UNSCR 1244] Final \Countries\Kosovo\Other Publications Other

240 Progress report Dutch funding for Kosovo [UNSCR 1244] \Countries\Kosovo\Other Publications Other

241 UNTFHS FINAL DOC 13082008 \Countries\Kosovo\Other Publications Other

242 Life Skills Based Education Evaluation Report \Countries\Kosovo\Evaluations Evaluation

243 LSBE Assessment Report- September 2006 - final - English \Countries\Kosovo\Evaluations Evaluation

244 WL&ECE Midterm Evaluation Report FINAL LAST VERSION \Countries\Kosovo\Evaluations Evaluation


245 Progresi i Raportit_ENG_01(press) \Countries\Kosovo\ESP & MTPEF Gov

246 Progress Report 2011 \Countries\Kosovo\ESP & MTPEF Gov

247 EEPCT Youth Participatory Research Case Study Dec 2011 \Countries\Kosovo\Studies Study

248 English_Report_Final \Countries\Kosovo\Studies Study

249 Kosovo [UNSCR 1244] Case Study_Eng_Web \Countries\Kosovo\Studies Study

250 Annex 4_ Indicators Framework \Countries\Kosovo\ESP & MTPEF Gov

251 Annex 5_Fact and Figures_brochure \Countries\Kosovo\ESP & MTPEF Gov

252 95112 \Countries\Kosovo\Studies Study

253 Annex 7_CFS Case Study Kosovo [UNSCR 1244] \Countries\Kosovo\Studies Study

254 2002 Annual Report \Countries\Kosovo\Annual Reports\2002 Report

255 2010 UNICEF Kosovo [UNSCR 1244] Annual Report \Countries\Kosovo\Annual Reports\2010 Report

256 2012 UNICEF Kosovo [UNSCR 1244] Annual Report FINAL_14 January 2013 \Countries\Kosovo\Annual Reports\2012 Report

257 Government Computer for Every Child Project \Countries\Macedonia\Other Publications Other

258 Government Programme for Education 2011-2015 \Countries\Macedonia\Other Publications Other

259 National Programme for Development of Education 2005-2015 \Countries\Macedonia\ESF & MTPEF Gov

260 UNICEF Child focused Public Expenditure Review_p29 section on Education \Countries\Macedonia\ESF & MTPEF Gov

261 World Bank FYROM Public Expenditure Review_p19-35 on education \Countries\Macedonia\ESF & MTPEF Gov

262 National Strategy - Steps Towards Integrated Education \Countries\Macedonia\ESF & MTPEF Gov

263 Macroeconomic data \Countries\Macedonia\ESF & MTPEF Gov

264 Pre-accession economic programme 2013-2015_with MTEF equivalent data \Countries\Macedonia\ESF & MTPEF Gov

265 Public Investment Programme 2011-2013 \Countries\Macedonia\ESF & MTPEF Gov

266 World Bank report Demand for Skills in FYROM \Countries\Macedonia\Other Publications Other

267 World Bank report FYROM \Countries\Macedonia\Other Publications Other

268 MICS 2005-2006 \Countries\Macedonia\Data\MICS 3 Study

269 2002 \Countries\Serbia\Annual Reports\2003 Report

270 2007_Equal Chances in Secondary Schooling_evaluation_report \Countries\Serbia\Studies Study

271 Analysis of Drop Out from Compulsory Education_2012 \Countries\Serbia\Other Publications Other

272 AnnexB v.7 \Countries\Serbia\Annual Reports\2006 Report

273 Belgrade_Conference_outcome_document \Countries\Serbia\Other Publications Other

274 CIP - engleski \Countries\Serbia\Studies Study

275 Comprehensive Analysis of Primary Education in FRY \Countries\Serbia\Studies Study

276 Costing of Inclusive Education_2011 \Countries\Serbia\Studies Study


277 DURN - Izvestaj o radu - ROC 2009-2012 \Countries\Serbia\Other Publications Other

278 E2008-2010 Equal Chances xternal evaluation \Countries\Serbia\Studies Study

279 Final Narrative REPORT IE Network - MOST (050813) \Countries\Serbia\Other Publications Other

280 Rapid Assessment_Inclusive Education_2010 \Countries\Serbia\Evaluations Evaluation

281 Integrative Report, Evaluation of programme School without violence, Engl \Countries\Serbia\Evaluations Evaluation

282 Progress Narrative Report Aug 2103 Open Club Nis \Countries\Serbia\Other Publications Other

283 REF UNICEF campaign 2012 \Countries\Serbia\Other Publications Policy

284 Review of Human Resource Development_2010 \Countries\Serbia\Studies Study

285 State_of_Children_in_Serbia_2006_ANNEXES(1) \Countries\Serbia\Studies Study

286 State_of_Children_in_Serbia_2006_1-56(1) \Countries\Serbia\Studies Study

287 Status projekta DURN (23 Feb 2012) \Countries\Serbia\Other Publications Other

288 Strategija razvoja obrazovanja u Srbiji do 2020 \Countries\Serbia\Other Publications Other

289 List of interventions EDU Turkey means of verif \Countries\Turkey\Other Publications Other

290 List of interventions EDU Kosovo [UNSCR 1244] final \Countries\Kosovo\Other Publications Other

291 List of interventions EDU Armenia \Countries\Armenia\Other Publications Other

292 List of interventions EDU Serbia means of verif \Countries\Serbia\Other Publications Other

293 Theory of Change 1 \Countries\Kosovo\Other Publications Other

APPENDIX 4:

Evaluation Matrix

Final version following the validation at the team workshop

during the pilot phase in Kosovo (UNSCR 1244), 09-12 September 2013

No. Evaluation question Indicator Formulated (old/new)

Data type Availability Data Source Method/Instrument MoRES

Determinant Annotation

IM.1a Do the impact results reported by countries in terms of greater inclusion and improved quality of education really represent a change in the lives of children, in terms of (a) the realisation of their right to quality basic education and/or reduction of equity gaps? In principle, this question will be approached in three steps: 1. What do the figures say about the

degree of change in education outcomes?

2. To which degree do government

and any other relevant authority accept the figures as accurate?

3. Do teachers‘, parents‘, and

children‘s perceptions of improvements match up with the level of change the figures imply?

All Indicators represent different minorities, depending on availability of data. Overall, all minorities are represented through different indicators

School readiness over time old (MICS 7.2) administrative data, survey

data partly

D171 (TFYRoM); D131, D132, D133, D134 (Serbia)

Document and statistical analysis

MORES Determinant Framework only relates to Outcome Results (in terms of system changes), not to Impact Results. MORES determinants have only been related to evaluation questions under the Relevance, Effectiveness, Efficiency and Sustainability categories of indicators.

Net intake rate in primary education over time

old (MICS 7.3) administrative data, survey

data partly



Primary school net attendance ratio (adjusted) over time


data partly



Secondary school net attendance ratio (adjusted) over time


data partly



Children reaching last grade of primary over time


data partly



Primary completion rate over time old (MICS 7.7) administrative data, survey

data partly



PISA scores on science, mathematics and reading

adapted survey partly D023, D024 Document and statistical

analysis

Transition rate to secondary school over time


data partly



Literacy rate among young women over time


data partly



Gender parity index (primary school) over time


data partly

D171 (TFYRoM); D131, D132, D133, D134

(Serbia); D020


Gender parity index (secondary school) over time

old (MICS 7.10)

administrative data, survey

data partly

D171 (TFYRoM); D131, D132, D133, D134

(Serbia); D020


Roma Children reaching last grade of primary over time

newly formulated


data partly



Primary completion rate for Roma Children over time

newly formulated


data partly



Transition rate to secondary school for Roma children over time

newly formulated


data partly



PISA score on science scale for girls over time

newly formulated

survey partly D023, D024 Document and statistical

analysis




PISA score on reading scale for girls over time

newly formulated


analysis


PISA score on mathematics scale for girls over time

newly formulated


analysis

Degree of Roma children receiving an education in mainstream schools

new field data not yet

Interviewees FGD participants

Classroom observation

Interview form FGD form

to be done during country visits

Degree of SEN children receiving an education in mainstream schools along with specialised instruction appropriate to their abilities

new field data not yet


Classroom observation

Interview form FGD form


Degree to which active learning principles are reflected in teaching practice

new field data Classroom observation Classroom observation

form to be done during country visits

new field data Classroom observation

Percentage of out of school children - comparison of survey and administrative data (Fig. 11+12+19)

new survey data partly D020 Statistical analysis

IM.2 Are these results supported by reliable and validated primary data (surveys, administrative data, evaluations, assessments)?

Coherence of results across multiple sources

new surveys

documents field data

yes (secondary data)

not yet (field data)

All secondary data

Interviewees - heads of TTCs, administrators, teacher union, country

office staff

Review

new surveys

documents field data

yes (secondary data)

not yet (field data)

All secondary data

Interviewees - heads of TTCs, administrators, teacher union, country

office staff

IM.3 Have the strategies used by countries led to impact results in terms of (a) greater realization of children’s right to education and significant reduction of equity gaps?

Change in discriminatory attitudes new field data not yet Interviewees

FGD participants Interview form

FGD form to be done during country visits

Changes in perception and self-perception of children and other stakeholders, with a specific focus on minorities

new field data not yet Interviewees



Degree of realisation of CfS standards adapted documents partly

D016; D053;D072; D102; D160

Interviewees - principals,

FGD participants - teachers, students, parents

Meta analysis of studies


Responsiveness of teachers towards inclusiveness, participation and equity


Principals, teachers, Interview form





students, parents, classroom observations

Degree of comprehensiveness of policies (including legislation)

new documents field data

not yet

Policy and Legislative documents

Interviewees, FGD participants

Document analysis against assessment

framework


Quantitative and qualitative involvement of multiple stakeholders




IM.4 What has been UNICEF’s contribution to government’s policy formulation, implementation or fine tuning through its work in terms of knowledge generation, evidence-based advocacy and policy advice, and technical support?

Occurrence of UNICEF concepts (CfS, GE, AL) in governmental publications and/or statements


not yet



Document analysis, Interview form

FGD form


UNICEF Budget and expenditure by activities

new financial data probably UNICEF regional and

country budgets Financial analysis

to be prepared by COs before country visits

IM.5

IM.6 Have the regional initiatives initiated by the Regional Office influenced and impacted the strategies and interventions implemented at country level to reduce equity gaps in education?

Degree of countries' replication of regional initiatives focussing on the reduction of equity gaps in education at national level


not yet


Interviewees - Administrators, country

office staff, topic 5; Round Table participants

Document analysis Interview form

FGD form






IM.7a More specifically, what has been the influence and/or impact of the Regional Office’s support and advice to Country Offices’ education strategies and interventions in terms of (a) knowledge generation (regional position papers, technical guidance, regional studies and evaluations, study tours …),

Occurrence of Regional Office knowledge generation content in Country Office documents (CPD, CPAP)

new documents not yet Country office documents

CPD, CPAP Document analysis


IM.7b (b) capacity development (study tours, regional education network meetings …),

Degree of application/usage of capacity development content in Country offices and/or counterpart institutions at central and decentralised levels where applicable

new field data not yet Interviewees Interview form to be done during country visits

IM.7c (c) technical assistance (country visits, provision of external expertise…),

Degree of application/usage of technical assistance content in Country offices and/or counterpart institutions at central and decentralised levels where applicable





IM.7d (d) partnerships building (regional events, leveraging influence and funding)

Added value arising out of established partnerships, e.g. in terms of money and/or capacities


IM.7e (e) and Communication for Development (C4D campaigns)?

Degree of application/usage of C4D content in Country offices and/or counterpart institutions at central and decentralised levels where applicable


IM.8 How can UNICEF improve its policy influencing impact?

R.1a Have UNICEF’s interventions at country level that were designed to influence inclusive policies and system changes, been specifically targeted the most marginalized children, those children whose right to education is violated, in particular (a) children with disabilities,

Degree to which activities address access to and enrolment in basic education programmes

new Documents Field data

not yet Planning documents

Interviews Document analysis

against Scoring scheme S2; D1; D2; Q1




Interviews

R.1b (b) Roma children, Degree to which activities address the completion of primary and secondary education by Roma children




against Scoring scheme E1; D1; D2


R.1c (c) girls, Degree to which activities address the equal participation of girls in education




against Scoring scheme E1; D2


Gender stereotypes eliminated from the curricula and teaching materials

old (CFS17) Documents Field data

partly Textbooks

D016; D053; D072, D102; D160

Analysis of illustrations E1 to be done during country visits

Degree to which activities address the improvement of learning outcomes for girls




against Scoring scheme E1; S2


R.1d (d) children from poor rural areas, Degree to which activities address the improved participation in education of children from poor rural areas




against Scoring scheme E1; D1


R.1e (e) children performing below academic standards,

Degree to which activities address the improvement of learning outcomes for children performing below academic standards




against Scoring scheme E1; S2


R.1f (f) and children with multiple disadvantages?

Degree to which activities address the improved participation in education of children with multiple disadvantages




against Scoring scheme E1; S2; D1; D2


R.2 Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights violation in education been relevant to prevailing education sector

Degree to which country issues are reflected/ matched in/by UNICEF programming




against Scoring scheme E1


Degree of relevance of UNICEF practices as assessed by policy makers and DPs




against Scoring scheme Q1





policies and international standards?

R.3 Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights violation in education been relevant to the work and programmes of national partners?

Degree to which national partner issues are reflected/ matched in/by UNICEF programming




against Scoring scheme Q1


Degree of relevance of UNICEF practices as assessed by national implementing partners




against Scoring scheme E2


ES.1 How effective have been government’s interventions in removing system bottlenecks that determined or contributed to the exclusion of marginalised children from education (use of the UNICEF MoRES Determinant Analytical Framework)? Preparatory work: Identification and operationalisation of key government strategies (how they are translated into activities)

Degree to which marginalised children benefit from key Government educational interventions


not yet

Education Sector Plan, MTEF, JRM reports,

M&E reports Interviewees

RT and FGD participants


E2; S2; D3

Example for 2 possible interventions: Degree to which marginalised children benefit from expansion of infrastructure; Degree to which marginalised children benefit from expansion of ICT provision at schools Could be prepared ahead of field visits if ESPs (or similar) are provided to Evaluation Team ; then follow-up during field visits

Degree of change in allocation and disbursement of required resources for educational reforms with regard to inclusion of out-of-school children

adapted MoRES indicator

administrative data, financial

data Field data

not yet Government budget, MTEF, Interviewees


Document analysis Financial analysis

Interview form E3

Degree to which the government adheres to specific standards related to marginalised children (CRC, CEDAW)


administrative data, single case data, survey data

not yet

Education Sector Plan, MTEF, JRM reports, M&E

reports, Interviewees (administrators)


Q1

ES.2 How effective have been UNICEF’s strategies – both at regional and at country level - in contributing to the removal of such bottlenecks?

Occurrence of UNICEF concepts (CfS, GE, AL) in governmental interventions


not yet

Education Sector Plan

MTEF JRM reports M&E reports Interviewees


E1; E2;

Number and types of UNICEF support activities to key Government educational interventions focussing on marginalised children


not yet

UNICEF planning documents

Policy papers Strategy papers

Interviewees


E1; E2; S1; S2;




Value of change as a consequence of UNICEF support in relation to key Government educational interventions focussing on marginalised children


not yet

UNICEF planning documents

Policy papers Strategy papers

Interviewees


E1; E2; S1; S2; D2;

Existence of partnerships (e.g. donor coordination group) between the stakeholders in the educational sector


Documents Field data

Partnership minutes

Interviewees Document analysis

Interview form E4

Degree of influence of partnerships on policy making towards inclusive education


not yet

Partnership minutes UNICEF planning

documents Policy papers

Strategy papers Interviewees


E4

ES.3 Have there been opportunities for programmatic synergies between UNICEF’s interventions and those of its development partners that contributed to increase the effectiveness of government’s efforts to remove system bottlenecks to inclusive education?

Number and types of utilised opportunities for programmatic synergies


not yet

Country programmes Policy papers

Strategic papers Interviewees


E1; E2; E4

Value of change as a consequence of programmatic synergies


not yet




E1; E2; E4

ES.4 Which are the system bottlenecks that have not been addressed by either the government or its partners (including UNICEF), or which the government and its partners have not been able to remove? And what are the reasons for that?

Degree to which marginalised children benefit from educational interventions


not yet





E1; E2; D1; D2; D3; Q1

ES.5a More specifically, which strategies – among UNICEF’s core roles – have been the most effective in contributing to remove system bottlenecks to inclusive education: (a) policy advice and technical assistance,

Value of change as a consequence of policy advice and technical assistance


not yet




E1; E2: S1; S2; D1; Q1


not yet



ES.5b (b) modelling,

Value of change as a consequence of modelling


not yet





E1; E2: S1; S2; D1; Q1

Clear definition for modelling needed

ES.5c (c) facilitating national dialogue towards child-friendly social norms,

Value of change as a consequence of facilitating national dialogue towards Cf social norms


not yet Country programmes

Policy papers Document analysis

Interview form E1; E2: S1; S2; D1; Q1





ES.5d (d) enabling knowledge exchange, Value of change as a consequence of enabling knowledge exchange


not yet




E1; E2: S1; S2; D1; Q1

ES.5e (e) monitoring and evaluation,

Value of change as a consequence of M&E


not yet





E1; E2: S1; S2; D1; Q1

ES.5f (f) leveraging resources from the public and private sectors? Value of change as a consequence of

leveraging resources new


not yet




E3; D1; D3

IM.5a Based on an analysis of UNICEF’s influence on policy at country level – where has UNICEF’s influence been the strongest and most effective? Is it in: (a) supporting policy development;

Value of change as a consequence of supporting policy development


not yet




E1; E2: S1; S2; Q1

IM.5b (b) building capacity to implement policy; Value of change as a consequence of

building capacity to implement policy new


not yet




S2, S1; D2

IM.5c (c) setting norms and standards that help monitor and fine tune implementation;

Value of change as a consequence of setting norms and standards that help monitor and fine-tune implementation


not yet




Q1

IM.5d (d) evidence-based advocacy and Communication for Development;

Value of change as a consequence of evidence-based advocacy and C4D


not yet





E1; E2: D2

IM.5e (e) or design and pilot innovative models that can be replicated and incorporated in national or sub-national programme design?

Value of change as a consequence of designing and piloting innovative models that can be replicated and incorporated in national or sub-national programme design


not yet





E2: E3; S1; S2; Q1

ES.6a In particular, how effective has UNICEF been in influencing (a) the formulation,

Value of change as a consequence of influencing the formulation of inclusive education policies, strategies and action plans targeting the most marginalised and excluded children


not yet




E2: D3




ES.6b (b) influencing adoption Value of change as a consequence of influencing the adoption of inclusive education policies, strategies and action plans targeting the most marginalised and excluded children


not yet




E1; S1; S2

ES.6c (c) and influencing enforcement of inclusive education policies, strategies and action plans (at both national and local level) targeting the most marginalized and excluded children?

Value of change as a consequence of influencing the enforcement of inclusive education policies, strategies and action plans targeting the most marginalised and excluded children


not yet





E1; E2: E3;S1; S2; D1; D3; Q1

ES.7 How effective has UNICEF been in monitoring and evaluating the enforcement and implementation of such policies, strategies and action plans (particularly at local level)?

Value of change as a consequence of monitoring and evaluating the enforcement and implementation of inclusive education policies, strategies and action plans targeting the most marginalised and excluded children


not yet





E2: E3; Q1

ES.8 How effective has UNICEF been in modelling and piloting (together with government counterparts and other partners) inclusive education interventions that have a potential to be replicated, expanded and scaled up?

Value of change as a consequence of modelling and piloting inclusive education interventions that have a potential to be replicated, expanded and scaled up


not yet





E4, S1; S2; Q1

EY.1 Have UNICEF’s resources invested in support of more inclusive education policies and plans been used in a strategic and cost-effective manner?

Number of pilot projects having been replicated, expanded and/or scaled up


not yet Policy papers Project reports Interviewees


S1; E3; E4 to be done during country visits

Relation of investments made by UNICEF compared to other international benchmarks (with regard to, inter alia, MOE staff capacity building, teacher and school principals training, curriculum development or revision).

new Financial data

Field data not yet

UNICEF regional and country budgets

Interviewees


Interview form E3; E4


EY.2 More specifically, has UNICEF been successful in playing a catalytic role and using its meagre core resources strategically to leverage partners’ funding for inclusive education reforms?

Funds invested by partners for inclusive education reforms due to UNICEF involvement in the education sector

new Financial data

Field data not yet

Partner budgets Interviewees

Financial analysis Interview form

E3; E4; S1; S2

Depending on accessibility of detailed partner data

EY.3 Would there have been a more cost-effective way to obtain the expected results?


new Financial data

Field data not yet

UNICEF regional and country budgets

Other budgets Interviewees


Financial analysis Interview form

E4 to be done during country visits




EY.4 How efficient has UNICEF been in modelling and piloting (together with government counterparts and other partners) inclusive education interventions that have a potential to be replicated, expanded and scaled up?

Relation/ratio of cost of inclusive pilot projects before and after integration into government system (replication, expansion, scaling up)


not yet

Policy papers Project reports Interviewees



S1 to be done during country visits

EY.5 Is UNICEF properly staffed and equipped to play an effective role at country level and influence the development and support the implementation of meaningful inclusive policies and reforms in the education sector?

Degree of perceptional confidence by UNICEF staff

new Field data not yet UNICEF country office staff Self-assessments S2; E1 to be done during country visits

EY.6 Have there been gaps between the design of UNICEF’s interventions at country level and the realities of implementation? For instance, have UNICEF’s recommended strategies such as child-centred approaches resulted in a greater inclusion of marginalized children in education? If yes, how large are the gaps?

Degree of severity of problems (including frequency of their occurrence) encountered during practical implementation


not yet


Country Office staff


E2; E4 to be done during country visits

Degree of necessary adjustments to intervention design during implementation


not yet


Country office staff


D3 to be done during country visits

EY.7 Have there been overlaps between UNICEF’s interventions and those of its development partners that have undermined the effectiveness of government’s efforts to achieve a greater inclusion of marginalized children in education?

Degree of observed substitution effects new

Administrative data

Financial data Field data

not yet Government budgets

Interviewees Financial analysis

Interview form E1; E4


Degree of use of common arrangements and procedures for delivery of education for marginalised children


not yet MoU

DP procedural guidelines Interviewees


E4 to be done during country visits

EY.8 What have been the coordination mechanisms between the government and UNICEF and its partners – if any? If there have been coordination mechanisms, how have they contributed to the removal of system bottlenecks to inclusive education?

Degree of use of common arrangements and procedures for delivery of inclusive education


not yet MoU




EY.9 Has UNICEF’s equity agenda and targeting of marginalised and excluded children, been aligned to governments’ policies/reform agendas and priorities of other duty bearers and partners?

Degree to which policy issues of government and other partners are reflected/matched in/by UNICEF programming




E1; E2; E4; Q1 to be done during country visits




EY.10 What is the level of awareness of the major partners with regards to UNICEF’s programme goals and strategies at country level?

Degree of awareness by policy makers and DPs regarding UNICEF programme goals and strategies




E1; E4; Q1 to be done during country visits

EY.11 If not, has UNICEF been successful in leveraging governments’ political will and financial resources to address child rights violations and equity issues in terms of access to and outcomes of education, and focus on the inclusion of the most vulnerable groups?

Degree of progress in terms of expected system changes

new Documents

Financial data Field data

not yet Government budgets

Interviewees Financial analysis

Interview form E3; D1

We assume this evaluation question refers to question 9 rather than 10

EY.12 Has UNICEF been successful in creating alliances, mobilising partners, donors and other duty bearers, and triggering national partnerships towards greater equity and inclusiveness in education?

Degree of successful mobilisation of partners, DPs and other duty bearers towards inclusiveness


not yet MoU



E3; E4; to be done during country visits

EY.13 In other words, has UNICEF been successful in making governments’ reforms and partners’ priorities more relevant to the realization of the right of all children to be included in quality learning?



not yet

MoU DP procedural guidelines

Interviewees Administrators, RT

participants


E1; E2; E4; D3 to be done during country visits

S.1 Are there indications that the impact results (greater inclusion of marginalized and vulnerable children in basic education) will or won’t be sustained over time in countries where they took place?

Degree of country’s political stability over time

new Administrative

data Survey data

not yet Country fact sheets Travel advisories

Document analysis to be done during field visits

Degree of country's macro-economic stability

new Macro-

economic data not yet Macro-economic forecasts Document analysis

to be done during field visits

Degree of successful mainstreaming of impact results into government systems


not yet Government documents

Document analysis to be done during field visits

Recurrent costs reflected in national budgets

new Financial data not yet Government

budgets Budget reports



S.2 Have the system changes that led to these impact results been sustained in countries where they took place? And if not, what are the bottlenecks to the sustainability of these results?

Degree of persistence of once introduced system changes which led to desired impact results (greater inclusion of marginalised and vulnerable children in basic education)


not yet Policy papers

Legislative documents Document analysis D3, E2


S.3 In working towards the removal of system bottlenecks to the inclusion of out of school children in quality learning, has UNICEF acquired a

Subjective perception of UNICEF as a partner by the Government, particularly when compared to other partners in the field


not yet Interviewees Interview form E4 to be done during field visits




comparative advantage to contribute in a significant way to the sustainability of these results?

H.1 To which extent a human-rights based approach to programming has been applied in the design and implementation of education sector reforms and has strengthened the impact of such interventions?

Degree to which implementation of education sector reforms is in line with Human Rights and Human Rights standards


not yet

Policy papers Legislative documents

Interviewees, RT participants

Document analysis Interview Form

E2; D3; Q1 to be done during country visits


not yet




not yet



Extent to which the design of education sector reforms embraces Human Rights and Human Rights standards


not yet




E1; E2; D3; Q1 to be done during country visits


not yet




not yet



Extent to which UNICEF ensures that policy makers and community representatives will encourage and facilitate the meaningful participation of children and young people in their communities


not yet Policy papers

Legislative documents Interviewees


E1; D2; D3; Q1 to be done during country visits

APPENDIX 4a:

Assessment of general answerability of evaluation questions

(as contained in Inception Report)

Assessment of answerability of evaluation questions

No. Evaluation questions Assessment of answerability

Impact

IM.1

Do the impact results reported by countries in terms of greater inclusion and improved quality of education really represent a change in the lives of children, in terms of the realisation of their right to quality basic education and/or reduction of equity gaps?87

Statistical data is available for the former Yugoslav Republic of Macedonia and Serbia in MICS database. For the other countries it will be searched for regional/national statistics and alternative data sources. The availability and quality of the data will be assessed accordingly.


This question should be answerable as far the information from the databases is accessible.


Question will be answerable mainly by qualitative interview data from the field. Furthermore, documents which provide secondary data are also available.

IM.4

What has been UNICEF’s contribution to government’s policy formulation, implementation or fine tuning through its work in terms of knowledge generation, evidence-based advocacy and policy advice, and technical support?

Answerability will depend mainly on the availability of UNICEF financial data.

IM.6

Have the regional initiatives initiated by the Regional Office influenced and impacted the strategies and interventions implemented at country level to reduce equity gaps in education?

Question should be answerable on the basis of interview data. Additional country-specific documents could provide further evidence.

IM.7a

More specifically, what has been the influence and/or impact of the Regional Office’s support and advice to Country Offices’ education strategies and interventions in terms of (a) knowledge generation (regional position papers, technical guidance, regional studies and evaluations, study tours …),


IM.7b (b) capacity development (study tours, regional education network meetings, etc.)




IM.8 How can UNICEF improve its policy influencing impact? To be addressed under “Recommendations”

Relevance

R.1a

Have UNICEF’s interventions at country level that were designed to influence inclusive policies and system changes, been specifically targeted the most marginalized children, those children whose right to education is violated, in particular (a) children with disabilities

Question should be answerable on the basis of interview data. Additional country-specific documents could provide further evidence. R.1b (b) Roma children,

R.1c (c) girls,

R.1d (d) children from poor rural areas,



R.2

Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights violation in education been relevant to prevailing education sector policies and international standards?


R.3

Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights violation in education been relevant to the work and programmes of national partners?


87 Indicators represent different minorities, depending on availability of data. Overall, all minorities are

represented through different indicators.



Effectiveness

ES.1

How effective have been government’s interventions in removing system bottlenecks that determined or contributed to the exclusion of marginalised children from education (use of the UNICEF MoRES Determinant Analytical Framework)? Preparatory work: Identification and operationalisation of key government strategies (how they are translated into activities)

Various data sources could be identified so far. Further information about government budgets and interview data will be necessary and should be available.

ES.2

How effective have been UNICEF’s strategies – both at regional and at country level - in contributing to the removal of such bottlenecks?


ES.3

Have there been opportunities for programmatic synergies between UNICEF’s interventions and those of its development partners that contributed to increase the effectiveness of government’s efforts to remove system bottlenecks to inclusive education?

Question should mainly be answerable on the basis of interview data.

ES.4

Which are the system bottlenecks that have not been addressed by either the government or its partners (including UNICEF), or which the government and its partners have not been able to remove? And what are the reasons for that?


ES.5a

More specifically, which strategies – among UNICEF’s core roles – have been the most effective in contributing to remove system bottlenecks to inclusive education: (a) policy advice and technical assistance,


ES.5b (b) modelling,


ES.5d (d) enabling knowledge exchange,

ES.5e (e) monitoring and evaluation,

ES.5f (f) leveraging resources from the public and private sectors?

IM.5a88

Based on an analysis of UNICEF’s influence on policy at country level – where has UNICEF’s influence been the strongest and most effective? Is it in: (a) supporting policy development;

IM.5b (b) building capacity to implement policy;




ES.6a In particular, how effective has UNICEF been in influencing (a) the formulation

Question should be answerable on the basis of interview data. Additional documents could provide further evidence.

ES.6b (b) influencing adoption

ES.6c

(c) and influencing enforcement of inclusive education policies, strategies and action plans (at both national and local level) targeting the most marginalized and excluded children?

88 These evaluation questions were originally grouped under “Impact“ in the ToR. We however feel they are better

placed under “Effectiveness“. Nevertheless, we maintain the coding for “Impact“ (IM) in order to show the

original source in the ToR.



ES.7

How effective has UNICEF been in monitoring and evaluating the enforcement and implementation of such policies, strategies and action plans (particularly at local level)?

Answerability will depend mostly on the availability of further country documents.

ES.8

How effective has UNICEF been in modelling and piloting (together with government counterparts and other partners) inclusive ed. interventions that have a potential to be replicated, expanded and scaled up?

Answerability will depend mostly on the availability of further country documents.

Efficiency

EY.1 Have UNICEF’s resources invested in support of more inclusive education policies and plans been used in a strategic and cost-effective manner? All efficiency questions should be answerable on the basis of interview

data and, as far as available, programme documentation. EY.2

More specifically, has UNICEF been successful in playing a catalytic role and using its meagre core resources strategically to leverage partners’ funding for inclusive education reforms?


All efficiency questions should be answerable on the basis of interview data and, as far as available, programme documentation. Full details are provided in the matrix Overview of Interview/FGD/Round Table items against indicators and MoRES Determinants, presented at the onset of Appendix 5

EY.4

How efficient has UNICEF been in modelling/piloting (together with government counterparts and other partners) inclusive education interventions that have a potential to be replicated, expanded and scaled up?

EY.5

Is UNICEF properly staffed and equipped to play an effective role at country level and influence the development and support the implementation of meaningful inclusive policies and reforms in the education sector?

EY.6

Have there been gaps between the design of UNICEF’s interventions at country level and the realities of implementation? For instance, have UNICEF’s recommended strategies such as child-centred approaches resulted in a greater inclusion of marginalized children in education? If yes, how large are the gaps?

EY.7

Have there been overlaps between UNICEF’s interventions and those of its development partners that have undermined the effectiveness of government’s efforts to achieve a greater inclusion of marginalized children in education?

EY.8

What have been the coordination mechanisms between the government and UNICEF and its partners – if any? If there have been coordination mechanisms, how have they contributed to the removal of system bottlenecks to inclusive ed.?

EY.9

Has UNICEF’s equity agenda and targeting of marginalised and excluded children, been aligned to governments’ policies/reform agendas and priorities of other duty bearers and partners?


EY.11

If not, has UNICEF been successful in leveraging governments’ political will and financial resources to address child rights violations and equity issues in terms of access to and outcomes of education, and focus on the inclusion of the most vulnerable groups?

EY.12

Has UNICEF been successful in creating alliances, mobilising partners, donors and other duty bearers, and triggering national partnerships towards greater equity and inclusiveness in education?

EY.13

In other words, has UNICEF been successful in making governments’ reforms and partners’ priorities more relevant to the realization of the right of all children to be included in quality learning?



Sustainability

S.1

Are there indications that the impact results (greater inclusion of marginalized and vulnerable children in basic education) will or won’t be sustained over time in countries where they took place?

Probably only an indication can be given on the basis of interview and statistical data.

S.2

Have the system changes that led to these impact results been sustained in countries where they took place? And if not, what are the bottlenecks to the sustainability of these results?

S.3

In working towards the removal of system bottlenecks to the inclusion of out of school children in quality learning, has UNICEF acquired a comparative advantage to contribute in a significant way to the sustainability of these results?

Human rights-based approach to programming

HR.1

To which extent a human-rights based approach to programming has been applied in the design and implementation of education sector reforms and has strengthened the impact of such interventions?


APPENDIX 4b:

Assessment of achievement of individual indicators contained in the Evaluation Matrix

The following table shows the overall rating assigned to the indicators of the final evaluation matrix, scored on the basis of a 3-level “traffic light” system, as agreed following the pilot phase in Kosovo (UNSCR 1244). The scores were assigned jointly by all evaluators on the basis of their own judgement, referring to information and data collected through the completed evaluation matrices and the interviews conducted in the field. The same scoring was applied to the quantitative data.

IM.1 Do the impact results reported by countries in terms of greater

inclusion and improved quality of education really represent a change in the lives of children, in terms of (a) the realisation of their right to quality basic education and/or reduction of equity gaps?

School attendance rate

Percentage of repeaters in primary education

Pupil:Teacher ratio in primary education

PISA scores on science, mathematics and reading

PISA score on science scale for girls over time

PISA score on mathematics scale for girls over time

PISA score on reading scale for girls over time

School attendance over wealth quintiles

Youth unemployment rate

Young women literacy rate

Degree of SEN and other previously excluded children receiving an education in mainstream schools along with specialised instruction appropriate to their abilities





Change in discriminatory attitudes


Degree of realisation of CfS standards






UNICEF Budget and expenditure by activities



IM.5b (b) building capacity to implement policy; Value of change as a consequence of building capacity to implement policy










IM.7a More specifically, what has been the influence and/or impact of the Regional Office’s support and advice to Country Offices’ education strategies and interventions in terms of (a) knowledge generation (regional position papers, technical guidance, regional studies and evaluations, study tours …),

Occurrence of Regional Office knowledge generation content in Country Office documents (CPD, CPAP)


Degree of application/usage of capacity development content in Country offices and/or counterpart institutions at central and decentralised levels where applicable


Degree of application/usage of technical assistance content in Country offices and/or counterpart institutions at central and decentralised levels where applicable



IM.7e (e) and Communication for Development (C4D campaigns)? Degree of application/usage of C4D content in Country offices and/or counterpart institutions at central and decentralised levels where applicable

R.1a Have UNICEF’s interventions at country level that were

designed to influence inclusive policies and system changes, been specifically targeted the most marginalized children, those children whose right to education is violated, in particular (a) children with disabilities,


(a) Physical access

(b) Intellectual access (degree of equity in learning opportunity and benefits)






R.1e (e) children performing below academic standards, Degree to which activities address the improvement of learning outcomes for children performing below academic standards

R.1f (f) and children with multiple disadvantages? Degree to which activities address the improved participation in education of children with multiple disadvantages

R.2 Have UNICEF’s interventions and approaches that were designed to reduce equity gaps and child rights violation in education been relevant to prevailing education sector policies and international standards?

Degree to which country issues are reflected/matched in/by UNICEF programming



Degree to which national partner issues are reflected/matched in/by UNICEF programming


ES.1 How effective have been government’s interventions in

removing system bottlenecks that determined or contributed to the exclusion of marginalised children from education (use of the UNICEF MoRES Determinant Analytical Framework)? Preparatory work: Identification and operationalisation of key government strategies (how they are translated into activities)

















ES.5b (b) modelling, Value of change as a consequence of modelling


Value of change as a consequence of facilitating national dialogue towards child-friendly social norms


ES.5e (e) monitoring and evaluation, Value of change as a consequence of M&E

ES.5f (f) leveraging resources from the public and private sectors? Value of change as a consequence of leveraging resources










EY.1 Have UNICEF’s resources invested in support of more

inclusive education policies and plans been used in a strategic and cost-effective manner?


Relation of investments made by UNICEF compared to other international benchmarks (with regard to, inter alia, MOE staff capacity building, teacher and school principals training, curriculum development or revision)






Relation/ratio of cost of inclusive pilot projects before and after integration into government system (replication, expansion, scaling up)







Degree of observed substitution effects














S.1 Are there indications that the impact results (greater inclusion

of marginalized and vulnerable children in basic education) will or won’t be sustained over time in countries where they took place?

Degree of country’s political stability over time

Degree of country's macro-economic stability

Degree of successful mainstreaming of impact results into government systems

Recurrent costs reflected in national budgets

S.2 Have the system changes that led to these impact results been sustained in countries where they took place? And if not, what are the bottlenecks to the sustainability of these results?

Degree of persistence of once introduced system changes which led to desired impact results (greater inclusion of marginalised and vulnerable children in basic education)

S.3 In working towards the removal of system bottlenecks to the inclusion of out of school children in quality learning, has UNICEF acquired a comparative advantage to contribute in a significant way to the sustainability of these results?


H.1 To which extent a human-rights based approach to

programming has been applied in the design and implementation of education sector reforms and has strengthened the impact of such interventions?

Degree to which implementation of education sector reforms is in line with Human Rights and Human Rights standards

Extent to which the design of education sector reforms embraces Human Rights and Human Rights standards


MoRES

Code Determinant Description


E1 Social norms Widely followed social rules of behaviour

E2 Legislation/Policy Adequacy of laws and policies

E3 Budget/Expenditure Allocation and disbursement of required resources

E4 Management/Coordination Roles and accountability/coordination/partnerships


Supply

S1 Availability of essential commodities/inputs Essential commodities/inputs required to deliver a service or adopt a practice

S2 Access to adequately staffed services, facilities and information

Physical access (services, facilities, information)


Demand

D1 Financial access Direct and indirect cost of services/practices

D2 Social and cultural practices and beliefs Individual/community beliefs, awareness, behaviours, practices, attitudes

D3 Timing and continuity of use Completion/continuity in service, practice


Quality

Q1 Quality of care Adherence to required quality standards (national or international norms)

APPENDIX 5:

Data collection tools:

Evaluation Matrix template for completion after Interviews/FGDs

Overview of Interview/FGD/Round Table items

against indicators and MoRES Determinants

Interview Sheets INT-1a, INT-1b, INT-2, INT-3

Round Table Discussion Sheet RT-1

Focus Group Discussion Sheets FGD-1, FGD-2, FGD-2

Recording Sheet for Interviews

Recording Sheet for Round Table Discussion

Recording Sheet for Focus Group Discussions

Classroom Observation Sheet CO-1

Evaluation Matrix template for completion after Interviews/FGDs Country: (please fill in before completion)

No. Evaluation question Indicator Specific Data Source Summary INT, FGD, classroom observations

IM.1a Do the impact results reported by countries in terms of greater inclusion and improved quality of education really represent a change in the lives of children, in terms of (a) the realisation of their right to quality basic education and/or reduction of equity gaps?


Teacher and students in classroom

Degree of Roma children receiving an education in mainstream schools

Interviewees – principals, school administration

Degree of SEN children receiving an education in mainstream schools along with specialised instruction appropriate to their abilities

Interviewees – principals, school administration



Interviewees - principals, heads of TTCs, administrators, teacher union, country office staff; FGD participants - teachers, students, parents;



FGD participants - teachers, students, parents;


Degree of institutionalisation of CfS standards

Interviewees - principals, FGD participants - teachers, students, parents


Interviewees - principals;


Policy and Legislative documents Interviewees, FGD participants

IM.4 What has been UNICEF’s contribution to government’s policy formulation, implementation or fine

Occurrence of UNICEF concepts (CfS, GE, AL) in


Country: (please fill in before completion)


tuning through its work in terms of knowledge generation, evidence-based advocacy and policy advice, and technical support?

governmental publications and/or statements




Interviewees - Administrators, country office staff, topic 5; Round Table participants


Degree of application/usage of capacity development content in Country offices and/or counterpart institutions

Interviewees


Degree of application/usage of technical assistance content in Country offices and/or counterpart institutions

Interviewees



Interviewees


Degree of application/usage of C4D content in Country offices and/or counterpart institutions

Interviewees

R.1a Have UNICEF’s interventions at country level that were designed to influence inclusive policies and system changes, been specifically targeted the most marginalized children, those children whose right to education is violated, in particular (a) children with disabilities,


Planning documents Interviews






Curriculum framework (national level), school syllabi and teaching materials















Interviewees - principals, heads of TTCs, administrators, teacher union, country office staff

Degree of relevance of UNICEF procedures as assessed by policy makers and DPs



Round Table participants

Degree of relevance of UNICEF procedures as assessed by national implementing partners


ES.1 How effective have been government’s interventions in removing system bottlenecks that determined or contributed to the exclusion of marginalised children from education (use of the UNICEF MoRES Determinant Analytical Framework)?

Degree to which marginalised children benefit from key Government educational interventions (individual indicator for every intervention)

Interviewees, RT and FGD participants



Preparatory work: Identification and operationalisation of key government strategies (how they are translated into activities)




Interviewees (administrators)




Number and types of UNICEF support activities to key Government educational interventions focussing on marginalised children (individual indicator for every intervention)


Value of change as a consequence of UNICEF support in relation to key Government educational interventions focussing on marginalised children (individual indicator for every intervention)






ES.3 Have there been opportunities for programmatic synergies between





UNICEF’s interventions and those of its development partners that contributed to increase the effectiveness of government’s efforts to remove system bottlenecks to inclusive education?




Degree to which marginalised children benefit from educational interventions (individual indicator for every intervention)















Value of change as a consequence of leveraging resources






Value of change as a consequence of building capacity to implement policy
























ES.8 How effective has UNICEF been in modelling and piloting (together with government counterparts and

Value of change as a consequence of modelling and piloting inclusive education





interventions that have a potential to be replicated, expanded and scaled up
















Degree of perceptional confidence by UNICEF staff (regarding, inter alia, technical/financial resources, staff qualifications, language competencies, cultural background)

Interviewees - country office staff

EY.6 Degree of severity of problems (including frequency of their




Have there been gaps between the design of UNICEF’s interventions at country level and the realities of implementation? For instance, have UNICEF’s recommended strategies such as child-centred approaches resulted in a greater inclusion of marginalized children in education? If yes, how large are the gaps?

occurrence) encountered during practical implementation




Degree of observed substitution effects

Interviewees - administrators


Interviewees - administrators; RT participants











Resources invested by government for addressing child rights violations and equity issues due to UNICEF involvement in the education sector

Interviewees











Interviewees


Extent to which UNICEF advocates for effective policies to benefit children


Extent to which UNICEF supports innovative programmes to care for and protect children




In this matrix, evaluation questions and indicators are related to methods of information generation and analysis. The use of interviews, round table and focus group discussions is briefly explained indicating which interview, round table, and focus group discussion topic is most relevant to what evaluation question and indicator. Furthermore, the matrix shows the linkage between the information generated through interviews, discussions, and document analysis and MoRES determinants.

No. Evaluation question Indicator Specific Data Source Specific Method/Instrument MoRES Determinant

Impact

IM.1a Do the impact results reported by countries in terms of greater inclusion and improved quality of education really represent a change in the lives of children, in terms of (a) the realisation of their right to quality basic education and/or reduction of equity gaps?


Teacher and students in classroom

Analysis and cross-reference of information from classroom observation forms; Cross-check information from FGD and interviewees and information from classroom observation forms




Interviewees - principals, heads of TTCs, administrators, teacher union, country office staff; FGD participants - teachers, students, parents;

Analysis and cross-reference of information from interview sheets on: Topic 2 - Expectations of the “inclusive quality education for all” concept; Topic 3: Most important challenges of inclusive quality education for all; Topic 5 - Cooperation with students, parents and communities/ Cooperation, collaboration and partnerships; Analysis and cross-reference of information from FGD sheets on: Question 1 (teachers) - Experience with and involvement in inclusive education; Topic 2 - Characteristics of education services in relation to the “inclusive quality education for all” concept; Topic 3 (parents) - Particular factors impacting on children’s access to and successful participation in education; Topic 4 - Inclusive quality education in practice – classroom experiences Cross-reference of information from interviewees, and from FGD participants; Cross-reference of information from FGD and interviewees.



FGD participants - teachers, students, parents;

Analysis and cross-reference of information from FGD sheets on: Topic 1 (students) - Family characteristics and possible barriers for children’s schooling surrounding the family context; Topic 3 (parents) - Particular factors impacting on children’s access to and successful participation in education; Topic 4 - Inclusive quality education in practice – classroom experiences; Topic 5 - Cooperation with students, parents, community members, education institutes and/or education care specialists Cross-check information from FGD and information from classroom observation forms




Interviewees - principals; Analysis and cross-reference of information from interview and FGD sheets on: Interview topic 4 - Inclusive education in practice; FGD topic 4 (teachers) - Inclusive quality education in practice – classroom experiences;




Document analysis against assessment framework; Analysis and cross-check of information from interview sheets (MoE officials, teacher union) and results of document analyses;




Interview forms FGD forms Round Table forms





Cross-check information from interview and RT recording sheets with results document analysis; Analysis and cross-reference of information from interviews and RT sheets on: Topic 2 - Expectations of the “inclusive quality education for all” concept; Topic 3: Most important challenges of inclusive quality education for all; Topic 4: Inclusive quality education in practice; Topic 5: Cooperation, collaboration and partnerships.



Interviewees - Administrators, country office staff, topic 5; Round Table participants

Cross-reference information on interview topics; cross-check findings with document analysis; Interview sheets RT sheets


Degree of application/usage of capacity development content in Country offices and/or counterpart institutions

Interviewees Analysis and cross-reference of information of interview sheets, topic 1 - introduction, topic 4 inclusive education in practice; topic 5 cooperation, collaboration, and partnerships;


Degree of application/usage of technical assistance content in Country offices and/or counterpart institutions

Interviewees Analysis and cross-reference of information of interview sheets, topic 1 - introduction, topic 4 inclusive education in practice; topic 5 cooperation, collaboration, and partnerships





Degree of application/usage of C4D content in Country offices and/or counterpart institutions


Relevance

R.1a Have UNICEF’s interventions at country level that were designed to influence inclusive policies and system changes, been specifically targeted the most marginalized



Document analysis against Scoring scheme Cross-check information from interview and RT recording sheets with results document analysis; Analysis and cross-reference of information from interview sheets (all topics);

S2; D1; D2; Q1


children, those children whose right to education is violated, in particular (a) children with disabilities,

Analysis and cross-reference of information from FGD sheets (all topics Cross-reference of information from interviewees, and from FGD participants; Cross-reference of information from FGD and interviewees.



D1; D2



D2



S2



D1




S2




S2; D1; D2



Interviewees - principals, heads of TTCs,

administrators, teacher union, country office staff

Document analysis against Scoring scheme Cross-check information from interview recording sheets with results document analysis Analysis and cross-reference of information from interview and RT sheets (all topics)




Round Table participants Document analysis against Scoring scheme Cross-check information from RT recording sheets with results document analysis; Analysis and cross-reference of information from interview and RT sheets (all topics).



Effectiveness

ES.1 How effective have been government’s interventions in removing system bottlenecks that determined or contributed to the exclusion of marginalised children from education (use of the UNICEF MoRES Determinant Analytical Framework)?



Analysis and cross-check of information from interview, RT and FGD recording sheets against results of data and document analyses;

S2; D2; D3


Preparatory work: Identification and operationalisation of key government strategies (how they are translated into activities)



Analysis and cross-check of information from interview and RT recording sheets against results of financial data and document analyses

E3


Interviewees (administrators)

Analysis and cross-check of information from interview recording sheets against results of data and document analyses

Q1




Analysis and cross-check of information from interview, RT and FGD recording sheets against results of document analyses

E1; E2



Analysis and cross-check of information from interview, RT and FGD recording sheets against results of data and document analyses

E1; E2; S1; S2




E1; E2; S1; S2; D2



Analysis and cross-check of information from interview and RT recording sheets against results of partnership minutes and document analyses

E



Analysis and cross-check of information from interview and RT recording sheets against results of partnership minutes and document analyses

E1; E2; E4




Analysis and cross-check of information from interview, RT and FGD recording sheets against results of data and document analyses E1; E2; E4




E1; E2; E4;

ES.4 Which are the system bottlenecks that have not been addressed by either the government or its partners (including UNICEF), or which the government and its partners have




E1; E2; D1; D2; D3; Q1


not been able to remove? And what are the reasons for that?





E1; E2: S1; S2; D1; Q1




E1; E2: S1; S2; D1; Q1




Analysis and cross-check of information from interview, RT and FGD recording sheets against results of data and document analyses E1; E2: S1; S2; D1; Q1




E1; E2: S1; S2; D1; Q1




E1; E2: S1; S2; D1; Q1


Value of change as a consequence of leveraging resources



E3; D1; D3





E1; E2: S1; S2; Q1


Value of change as a consequence of building capacity to implement policy



S1; S2;D2;




Analysis and cross-check of information from interview, RT and FGD recording sheets against results of data and document analyses Q1





E1; E2: D2

IM.5d (e) or design and pilot innovative models that can be replicated and incorporated in national or sub-national programme design?




E2: E3; S1; S2; Q1





E2: D3

ES.6b (b) influencing adoption Value of change as a consequence of influencing the adoption of inclusive education policies, strategies and action


Analysis and cross-check of information from interview, RT and FGD recording sheets against results of data and document analyses E1; S1; S2


plans targeting the most marginalised and excluded children





E1; E2: E3;S1; S2; D1; D3; Q1





E2: E3; Q1





S1; S2;Q1

Efficiency





E3; E4




E3; E4





E3; E4; S1; S2





EY.4 How efficient has UNICEF been in modelling and piloting (together with government counterparts and other partners) inclusive education



Analysis and cross-check of information from interview, RT recording sheets against results of data and document analyses


interventions that have a potential to be replicated, expanded and scaled up?




Analysis of self-assessments; Cross-check of information from interview recording sheets against results of self-assessment analysis

E1




Analysis and cross-check of information from interview recording sheets against results of self-assessment and document analyses

E2; E4



Analysis and cross-check of information from interview recording sheets against results of self-assessment and document analyses

D3


Degree of observed substitution effects Interviewees - administrators

Cross-check of information from interview sheets and financial data and document analyses

E4



Cross-check of information from interview and RT sheets and data and document analyses

E1; E





E4





E1; E2; E4; Q1





E1; E4; Q1




Interviewees Cross-check of information from interview sheets and financial data and document analyses

E3; D1





E3; E4





E1; E2; E4; D3

Sustainability



Interviewees

Analysis and cross-reference of information from interview sheets administrators; Cross-check information from interview sheets (administrators, country staff) and RT sheets; Cross-check information from self-assessment forms and interview sheets administrators.

E4

Human rights-based approach to programming


Extent to which UNICEF advocates for effective policies to benefit children


Cross-check information from interview, RT (topic 5), and FGD sheets (topic 5) with document analysis

E2; D3; Q1

Extent to which UNICEF supports innovative programmes to care for and protect children


Cross-check information from interview, RT, and FGD sheets (all topics) with document analysis E1; E2; D3; Q1



Cross-check information from interview, RT, and FGD sheets (all topics) with document analysis

E1; D2; D3; Q1

Evaluation – Including all children in quality learning in CEE/CIS Interview Sheets

INT-1a, INT-1b, INT-2, INT-3

The interview sheets are designed in such a way that five similar core topics will be covered by all four

target groups interviewed, in order to easily compare and cross-reference the answers during analysis

and interpretation.

Contrarily to Focus Group and Round Table Discussions where the discussion focus will be narrower,

such design allows a certain degree of flexibility when conducting the interview. The interviewer should

focus on the main areas of expertise on the side of the interviewee and thus generate the most rewarding

outcome of the interview.

The five core topics covered are

Topic 1: Introduction

Topic 2: Expectations of the “inclusive quality education for all” concept

Topic 3: Most important challenges of inclusive quality education for all

Topic 4: Inclusive quality education in practice

Topic 5: Cooperation, collaboration, and partnerships89

Within every core topic, about five guiding questions are given. It will not be necessary to cover all

guiding questions at every interview; their purpose is merely to guide the discussion. Issues not

covered at one specific interview can be covered at another interview. Looking at the total of the

interviews conducted, all guiding questions should have been covered.

89 With schools/education institutions, communities, Ministries, Development Partner organisations, NGOs,

education care specialists (social worker, psychologist, speech therapist, behavioural coach, motoric skills

therapist, etc.)

Evaluation – Including all children in quality learning in CEE/CIS INT-1a: Principals PRI-IE: Principals (inclusive education IE) PRI-NOIE: Principals not involved in IE (Control)


Have you or your school been actively involved in inclusive quality education/CfS activities (for example, in programmes for school development, leadership training, implementation of inclusive education, awareness campaigns, or any other initiatives facilitating quality education for all)?

How, what was your role, what were your responsibilities?

What is your involvement in education now?

YES PRI-IE

NO PRI-NOIE (Control Group)


In what way do disability, family background and income, gender and/or ethnicity impact on enrolment and student performance in your school?

How would you describe the social environment for disadvantaged students in the school’s catchment area?

What is the ambition of your school/ in terms of educational development? Where do you expect your school to be in 20 years time?

What do you see as strengths or opportunities of the inclusive quality education approach? What are weaknesses, or threats?

What would you suggest as a strategy for overcoming barriers to students’ enrolment and learning related to disability, family background and income, gender and/or ethnicity?

What kind of qualifications (including attitude, self-esteem, behaviour) do you expect students to have when leaving school?


Based on your experience, what are the most important challenges that “schools for all” are facing? (To be followed up from the challenges mentioned. Ask for concrete examples to clarify statements and/or verify understanding. If no challenges are mentioned, additional questions should be asked as listed below)

One could argue that teaching a diverse group of students is not equally beneficial to all students. What is your opinion on that?

One could say that inclusive education, new teaching methodologies, and assessment requirements are too time-consuming. What is your opinion on that?

We heard that the inclusive quality education approach is duplicating what the “standard” policy (and related training) is already doing. Based on your experiences, what would you say to that?

What characteristics (infrastructure, school organisation, school culture including language, etc.) influence enrolment and learning achievements, particularly the enrolment and participation of minority students when compared to the enrolment and participation of majority students?

Despite primary education being compulsory, recent studies claim that a considerable number of children, particularly from disadvantaged backgrounds, are not being enrolled or leave school early. What is your opinion on that, and how could the situation be addressed?

How would you assess the resources (human, financial, time) required to implement inclusive quality education versus resources available?


In what way are you involved in the inclusive quality education implementation process?

What kinds of external assistance have you or your school received over the last 5 years?

How often do you (or your school) interact with other educational partners on inclusive quality education? What are the benefits of these interactions?

Personally as a principal, what is the most important challenge you are faced with professionally when it comes to the implementation of inclusive quality education, in particularly ensuring quality of learning?

In your opinion, how does the current curriculum address issues causing exclusion? How is this reflected in the textbooks?

Against what standards is the quality of teaching and learning assessed or assured in your school? Based in the information you have, how do you feel about the inclusive quality education concept?

How do you know about the performance of your school? How does it compare to the performance of the “traditional” education system?

What specific information do you receive from teachers regarding factors impacting on children’s performance? How to ensure that all children develop to their full potential?

Can you think of any changes in the responsibilities, function, or set-up of your school since the introduction of inclusive quality education? What kind of changes?

Topic 5: Cooperation with students, parents and communities

Have roles, functions, and/or responsibilities of your (and other) educational institutes changed since the inclusive quality education concept was introduced?

Can you give us an example that shows how cooperation with other educational partners/specialists is different from what used to happen in more “traditional” settings?

Can you give us an example of how inclusive education facilitated discussions and cooperation between yourself, school communities and other stakeholders?

In your role as educational administrator, how can you contribute to maintaining and improving contacts with the local school communities?

Some recent studies claim that parents are rather indifferent towards school, and that the State would need to do everything. What is your opinion on that? How could this issue be addressed? What would be the special relevance for minority children?

In cooperating with parents, i.e. both fathers and mothers, what is the role of language, i.e. the capacity of both fathers and mothers to speak and understand the national language?

Evaluation – Including all children in quality learning in CEE/CIS INT-1b: TTCs TTC-IE: Heads of TTCs (inclusive education IE) TTC-NOIE: Heads of TTCs not involved in IE (Control)


Have you or your institute been actively involved in inclusive quality education/CfS activities (for example, in programmes for school development, leadership training, implementation of inclusive education, awareness campaigns, or any other initiatives facilitating quality education for all)?

How, what was your role, what were your responsibilities?


YES TTC-IE

NO TTC-NOIE (Control Group)


What is the ambition of your institute in terms of educational development? Where do you expect your institute to be in 20 years time?

What do you see as advantages and disadvantages of the inclusive quality education concept and approach?

What would you suggest as a strategy for overcoming barriers to children’s learning, related to disability, family background and income, gender and/or ethnicity?

What kind of qualifications do you expect students to have when leaving school? (including attitude, self-esteem, behaviour)

In what way do disability, family background and income, gender and/or ethnicity impact on enrolment and student performance in your institute?






In your opinion, what are the key challenges in expanding, or scaling up CfS/inclusive quality education initiatives?

What education system characteristics (infrastructure, school organisation, and school culture including language, etc.) influence enrolment and learning achievements particularly the enrolment and participation of minority students when compared to the enrolment and participation of majority students?



In what way are you involved in the inclusive quality education implementation process?

What kinds of external assistance have you or your institute received over the last 5 years?

How often do you (or your institute) interact with other educational partners on inclusive quality education? What are the benefits of these interactions?

What is the most important challenge, teachers are faced with professionally when it comes to the implementation of inclusive quality education, in particularly ensuring quality of learning?

Against what standards is the quality of education assessed or assured in your institute? Based in the information you have, how do you feel about the inclusive quality education concept?

In your opinion, how does the current curriculum address issues causing exclusion? How is this reflected in the textbooks?

Can you think of any changes in the set-up or functioning of your institute since the introduction of inclusive quality education? What kind of changes?

Topic 5: Cooperation with students, parents and communities

Have roles, functions, and/or responsibilities of your (and other) educational institutes changed since the inclusive quality education concept was introduced?

Can you give us an example that shows how cooperation with other educational partners/specialists is different from what used to happen in more “traditional” settings?

Can you give us an example of how inclusive education facilitated discussions and cooperation between yourself, school communities and other stakeholders?

In cooperating with community members and other partners, what is the role of language, i.e. the capacity of people to write, speak and understand the national language?

Evaluation – Including all children in quality learning in CEE/CIS INT-2: MoE Officials; Teacher Union Mentioning of “inclusiveness“ as it relates to gender, linguistic backgrounds, or those historically or otherwise marginalised from full participation in learning.


Have you or your Department/institute been involved in and/or affected by inclusive quality education/CfS activities? (for example, changes in policy-making/strategic planning, education standards, implementation of inclusive education, inter-Ministerial coordination, stakeholder involvement, or any other initiatives promoting quality education for all)?

How, what was/is your role? What were/are your responsibilities?


YES ADM-IE

NO ADM-NOIE (Control Group)


What is the ambition of your organisation/MoE/Department in terms of development of education/the education system?

How is the inclusive quality education approach different to the “standard” education policy?

What do you see as advantages and disadvantages of the inclusive quality education approach? (Including strengths and weaknesses related to influencing changes at education system level)

Should the inclusive quality education approach continue to grow, what would you expect to see in ten years time?

What kind of qualifications (including attitude, values, norms) do you expect students to have when leaving school?





Despite primary education being compulsory, recent studies claim that a considerable number of children, particularly from disadvantaged backgrounds, are not being enrolled, under perform, or leave school early. What is your opinion on that, and how could the situation be addressed?

In your opinion, what are the key challenges in expanding, or scaling up CfS/inclusive quality education initiatives?

As an administrator, what is the most important challenge you are faced with professionally when it comes to supporting the implementation of inclusive quality education?



In your opinion, what are the key changes in education/schools since the introduction of the CfS/inclusive quality education concept?

From your experience, what are possible obstacles for schools to become CfS/inclusive (in terms of legislation, regulations, inspection standards, financial, human capacity)?

Against what standards does your Department/MoE assess the quality of education? How are these different from the more “traditional” standards for education performance?

How does your organisation/Department/MoE contribute to facilitating access to quality education for all children?

Can you think of any changes in your organisation/Department/MoE mandate, recruitment practices, responsibilities, tasks, etc. since the introduction of inclusive quality education? What kind of changes?

Topic 5: Cooperation, collaboration and partnerships

Have inclusive quality education concept, approach, or activities made a difference to your Department/MoE programmes, policies, etc. over the last 5-10 years?

How often do you or your organisation/Department interact with other educational partners on inclusive quality education? What are the benefits of these interactions?

Can you give us an example of how the introduction of inclusive, quality education/CfS facilitated discussions and cooperation between your organisation/Department/MoE and other stakeholders?

How is the performance of education assessed in your country? What about the performance of inclusive education? Based on the information you have, how do you feel about the inclusive quality education concept?

Have mandates and/or responsibilities of your Department, and other educational institutes changed since the inclusive quality education concept was introduced?

Can you give us an example that shows how cooperation with other educational partners is different from what used to happen in more “traditional” settings?

Evaluation – Including all children in quality learning in CEE/CIS INT-3: UNICEF Country Office Staff


In what way were you/your country office involved in inclusive quality education/CfS activities (for example, programmatic collaboration, (national level) dialogue, awareness raising/advocacy, inter-organisational coordination, piloting, modelling, capacity building, or any other initiatives promoting quality education for all)?

What was the impact, effect of the different initiatives and activities (make a distinction between school/student and system level)?

What type of intervention/activity would you say made the biggest contribution to achieving country office objectives?

What is your/the office focus on ensuring “inclusion of all children in quality learning” now?


What is the ambition of the office in terms of the further development of quality education for all? (Including ambitions related to changes at education system level)

What do you see as advantages and disadvantages of the inclusive quality education approach? (Including strengths and weaknesses related to influencing changes at education system level)






Despite primary education being compulsory, recent studies claim that a considerable number of children, particularly from disadvantaged backgrounds, are not being enrolled, under-perform, or leave school early. What is your opinion on that, and how could the situation be addressed?

What is the most important challenge you/your office is faced with when it comes to achieving quality education for all? (what system bottlenecks have not been addressed, removed)


In your opinion, what are the key challenges in expanding, replicating, or scaling up CfS/inclusive quality education initiatives?

From your experience, what are possible barriers for schools to become a CfS/inclusive (in terms of legislation, regulations, inspection standards, financial, human capacity)?

Can you give us examples of problems you/the office encountered during the implementation CfS/inclusive education?

How would you assess the resources (human, technical/financial, time) required to implement or scale up inclusive quality education versus resources available?

Can you think of any changes in country office mandate, (recruitment) practices, responsibilities, tasks, etc. since the introduction of inclusive quality education? What kind of changes?


What kinds of assistance have you or the office received over the last 5-10 years? (from Regional Office, differentiate between knowledge generation, capacity development, TA, partnerships building, campaigns/C4D)

Can you give us an example of how assistance received influenced your work, or the work of UNICEF partners?

How often do you (or the office) interact with other educational partners/regional office on inclusive quality education? What are the benefits of these interactions?

Can you give us an example of how the introduction of inclusive, quality education/CfS facilitated discussions and cooperation between the country office and other stakeholders?

Can you give us examples of partner mobilisation or partnership building (including the Government) in relation to inclusive education? What was the effect of the mobilisation of partners/duty bearers?

Evaluation – Including all children in quality learning in CEE/CIS

Round Table Discussion Sheet

RT-1

The design of the Round Table (RT) discussion sheet takes into account that a variety of issues have

already been discussed in-depth during FGDs and interviews conducted with administrators, teachers,

students, and principals.

The Round Table discussion sheet is designed in such a way that core topics will be covered, although

with a different focus.






Topic 5: Cooperation, collaboration, and partnerships90

Five to six guiding questions are given across the selected core topics above. It will be important for

the interviewer to first carefully listen to what the RT participants bring up on their own before

providing additional guidance for the discussion. Specific keywords for discussion are provided on the

following sheets.



therapist, etc.)

Evaluation – Including all children in quality learning in CEE/CIS Sheet RT-1: Development Partners/NGOs


In what way were you involved in inclusive quality education/CfS activities (for example, programmatic collaboration, (national level) dialogue, awareness raising/advocacy, inter-organisational coordination, piloting, capacity building, or any other initiatives promoting quality education for all)?

In what way has the inclusive education concept/approach influenced your organisations programming, budgeting/expenditure, procedures?

What is your organisation’s focus (priority) on ensuring “inclusion of all children in quality learning” now?


What is the ambition of your organisation in terms of development of education/the education system?

How is the inclusive quality education concept/approach different to education concepts and approaches from your organisation?

What do you see as advantages and disadvantages of the inclusive quality education approach?


What kind of qualifications (mentioning of attitude, self-esteem, values, norms) do you expect students to have when leaving school?





Despite primary education being compulsory, recent studies claim that a considerable number of children, particularly from disadvantaged backgrounds, are not being enrolled, under-perform or leave school early. What is your opinion on that, and how could the situation be addressed?

What is the most important challenge your organisation is faced with when it comes to achieving quality education for all? (what system bottlenecks have not been addressed, removed)



In your opinion, what are the key changes in education/schools since the introduction of the CfS/inclusive quality education concept?

In your opinion, what are the key challenges in expanding, replicating, or scaling up CfS/inclusive quality education initiatives?

From your experience, what possible barriers for schools to deliver inclusive, quality education for all still remain (in terms of legislation, regulations, inspection standards, financial, human capacity)?

Can you give us examples of problems your organisation encountered during implementation of CfS/inclusive education?

How would you assess the resources (human, technical/financial, time) required to implement or scale up inclusive quality education versus resources available?

Can you think of any changes in your organisation’s mandate, (recruitment) practices, responsibilities, tasks, etc. since the introduction of inclusive quality education? What kind of changes?


Have inclusive quality education concept, approach, or activities made a difference to your organisation’s programmes, policies, etc. over the last 5-10 years?

What partnerships/coordination mechanisms exit in your country? Who is participating? What are the main achievements?

How often does your organisation interact with other educational partners (including UNICEF) on inclusive quality education? What are the benefits of these interactions?

Can you give us an example of how the introduction of inclusive, quality education/CfS facilitated discussions and cooperation between your organisation and other stakeholders, including the Government?

Have responsibilities of your (and other) educational institutes changed since the inclusive quality education concept was introduced?

Can you give us an example that shows how cooperation with other educational partners is different from what used to happen in more “traditional” settings?

Can you give us examples of partners being mobilised (including the Government) in relation to inclusive education? What was the effect of the mobilisation of partners/duty bearers?

Evaluation – Including all children in quality learning in CEE/CIS Focus Group Discussion Sheets

FGD-1, FGD-2, FGD-3

The design of the Focus Group Discussion (FGD) sheets takes into account that a variety of issues have

already been discussed in-depth during the interviews conducted with administrators, principals and

development partners.

The Focus Group Discussion Sheets are designed in such a way that core topics will be covered, although

with a different focus for every target group.


Topic 1: Family characteristics and possible barriers for children’s schooling surrounding the family

context

Topic 2: Characteristics of education services in relation to the “inclusive quality education for all”

concept

Topic 3: Particular factors impacting on children’s access to and successful participation in education

Topic 4: Inclusive quality education in practice – classroom experiences

Topic 5: Cooperation with school, students, parents, community members, education institutes,

and/or education care specialists91

These topics are spread across the target groups as follows:

Target Group / Sub-Group Topic 1 Topic 2 Topic 3 Topic 4 Topic 5

Teachers

TEA-IE, TEA-NOIE x x x x Students x x x Parents x x x x

Within every Focus Group, five to six guiding questions are given across the selected core topics

above. It will be important for the interviewer to first carefully listen to what the FGD participants

bring up on their own before providing additional guidance for the discussion. Specific keywords for

discussion are provided on the following sheets.

91 Social worker, psychologist, speech therapist, behavioural coach, motoric skills therapist, etc.

Evaluation – Including all children in quality learning in CEE/CIS FGD-1 TEA: Teachers TEA-IE: Teachers exposed to Inclusive Education (IE) TEA-NOIE: Teachers not exposed to IE (control)

First question: “Have you or your school been actively involved in inclusive quality education/CfS activities, for example, in programmes for school development, leadership training, implementation of inclusive education training, awareness campaigns, or any other initiatives promoting quality education for all?” NO TEA-NOIE (Control Group) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - YES Second question: “Can you briefly explain what “inclusive education” and “quality learning for all” mean for you? If at least three issues out of the list below are being mentioned: TEA-IE If less than three issues out of the list below are being mentioned: TEA-NOIE - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Issues: Social mobilisation of marginalized children (ethnicity, poverty/working, gender, disability, academic

performance); reduction of the equity gaps; individualized teaching and learning; integration, inter-agency collaboration, home visits to families who do not send their child to school. Mentioning of “UNICEF” is not an issue that counts here!

Keywords:

Professional vision Social reality; attitude and behavioural change

Relevance of education; employment opportunities

Topic 2: Characteristics of education services in relation to the “inclusive quality education for all” concept


One could argue that teaching a diverse group of children is not equally beneficial to all students. What is your opinion on that?



What are the benefits of the inclusive quality education approach?

How does the infrastructure of your school influence enrolment, particularly the enrolment of minority children when compared to the enrolment of majority children?

How does the curriculum address the social reality of your pupils, also in terms of the design of the new textbooks?


From your experience, what are possible barriers for marginalized children to attend school, and how could these be addressed?

From your experience, what are possible barriers for marginalized children to enjoy being in school and fully engage in learning, and how could these be addressed?

What changes do you observe in the students when comparing to earlier times when you did not apply inclusive education methodology?

How do you facilitate better cooperation and mutual respect and/or resolve conflict in the classroom, and in what way does the inclusive education philosophy help you to achieve that?


Judging from your day-to-day teaching experience, how well were you prepared for teaching a diverse group of children? What was of particular benefit, what would still be needed?

What new skills or methods do you use most? When do you use them and what for?

As teachers, what is the most important challenge you are faced with professionally when it comes to the implementation of inclusive education, in particularly ensuring quality of learning?

In what ways (how) do you assess individual student’s performance and record progress? When, how often? What are the advantages of this way of assessment/testing?

How do you involve the students in the assessment process? Do they have an opportunity to assess themselves? How do they do that?

Topic 5: Cooperation with students, parents, community members, education institutes and/or education care specialists

Inclusive quality education caters to learning needs of all children, regardless their (dis)-ability, gender, cultural/ ethnic or economic background. In what way do you/does the school involve parents and communities to also ensure participation and learning achievements of disadvantaged children?

Judging from your discussions with parents, how do you think they feel about the inclusive, quality education concept?

Can you give us an example that shows how cooperation with students, parents or community members is different from what used to happen in more “traditional” settings?

What professional skills are required to address the learning needs of students in your school?

What professional expertise (other than teaching skills) is available to you to ensure that unique learning needs of students are fulfilled?

Evaluation – Including all children in quality learning in CEE/CIS FGD-2 STU: Students

Keywords:

Your dreams Your expectations

Family context, family roles

Topic 1: Family characteristics and possible barriers for children’s schooling surrounding the family context Tell us about yourselves – where do you come from? What is it like for you in the area/town/village where you live? What kind of tasks do you have (next to going to school)? Imagine yourself 5 years from now. Where will you live, where would you like to live? Think about a younger sibling/cousin/niece/nephew in your family. What would you like to wish for him/her? What are your professional aspirations?

Topic 4: Inclusive quality education in practice – classroom experiences Just think about an average school day. How do you experience your teachers in the classroom? In what way are they

different from other teachers you knew? Thinking of your classmates, has the inclusive quality education approach made them different in any way? How? If you have a problem with someone in class, how do you go about it? Can you give an example? In your personal experience, how do teachers react to you if you don’t really understand the lesson? How do you know how well you are doing in school?

Topic 5: Cooperation with education care specialists, parents and community members For you, who are the most important persons in your family and in your village? Is there somebody else important, maybe

also outside the village? Do these persons cooperate with each other (talk with each other, visit each other)? How? Are your parents or your parents’ friends interested in what is going on at school? If so, what do they do? In school, there are many tests and teachers assessing you. How do you participate in what teachers have to say about

you? For your schoolwork, do you get help from people other than your classroom teacher? What kind of help, how often? When you talk to your parents at home, how do they feel about school? How do you feel about your teachers? How do you think your teachers feel about you and your classmates?

Evaluation – Including all children in quality learning in CEE/CIS FGD-3 PAR: Parents

Keywords:

Value attached to education Expectations of school/education

School reality, roles, attitude; parent-school relations

Topic 1: Family characteristics and possible barriers for children’s schooling surrounding the family context

What is your background?

Tell us a bit about your children – how many, how old, which classes?

What would you like your children to be one day?

What are your own “school memories”?

There is a lot of talk by many people about “inclusive education” and “quality learning”. What do you think of all this?

How can the inclusive quality education concept help your child to become a better student?

In your family, what are your responsibilities as father/mother/spouse (including perceptions on discipline, gender roles, function of “violence”, family honour)?

Where would you like to see your children in 10 years time? Where in 15 years? In your opinion, what would be a good future for your son(s), for your daughter(s)?


In your opinion, what are the biggest obstacles for your child(ren) in attending school? What would help you most to overcome these obstacles?

In your opinion, what are the biggest obstacles for your child(ren) to enjoy going to school and learn well? How could these be addressed?

Thinking of your children, what would you like to see improved in their class, at school, at home?


Just think about an average school day. What do you hear from your children about school? Is it different in any way from what you hear (heard) from other children?

Can you think of an example of a very special lesson that your children might have told you about?

Thinking of your children, has the inclusive quality education approach made them different in any way? How?

Has the inclusive quality education approach changed the behaviour of your children at home? Can you give an example?

How do your children feel about going to school? How do they feel about their teachers?

How do you know how well your child(ren) is (are) doing in school?

Topic 5: Cooperation with school, education care specialists, family and community members

When it comes to sending your children to school, how do you ensure that family values are maintained?

Are you or your friends interested in what is going on at school? If so, what do you do?

In school, there are many tests and teachers assessing your children. How do you participate in what teachers have to say about your children?

How do they feel personally about school?

Can you give us an example that shows how you as a parent contribute to the success of your child being in school and performing well?

How do you feel about the teachers of your children? How do you think the teachers feel about your children and about you as a parent?

Do children/your child get help from people other than their classroom teacher? What kind of help, how often?

If people want to convince you of something they feel would be good for you, what should they do? Whom would you listen to, whose opinion would be valuable for you?


Recording Sheet: Interview

School/Institute/Organisation/Department:

INT-1a: PRI-IE INT-1b: TTC-IE INT-2: ADM INT-3: CO

INT-1a: PRI-NOIE INT-1b: TTC-NOIE

(Indicate Interview Type with a cross!)

– Only note keywords and/or statements below! –


Topic 2: Expectations of the “inclusive, quality for all concept”

Topic 3: Most important challenges of inclusive education for all


Topic 5: Cooperation collaboration, and partnerships92



therapist, etc.)


Recording Sheet: Round Table with DP

Organisation:



Topic 2: Expectations of the “inclusive, quality for all concept”

Topic 3: Most important challenges of inclusive education for all


Topic 5: Cooperation collaboration, and partnerships93



therapist, etc.)


Recording Sheet: Focus group discussion

School/Town

FGD-1: TEA-IE FGD-2: STU FGD-3: PAR

FGD-1: TEA-NOIE

(Indicate FGD Type with a cross!)


Topic 1: Family characteristics and possible barriers for children’s schooling surrounding the family context

Topic 2: Characteristics of education services in relation to the “inclusive quality education for all” concept

Topic 3: Particular factors impacting on children’s’ access to and successful participation in education


Topic 5: Cooperation with school, students, parents, community members, education institutes, and/or education

care specialists94

94 Social worker, psychologist, speech therapist, behavioural coach, motoric skills therapist, etc.

Evaluation – Including all children in quality learning in CEE/CIS CO-1 Classroom Observation – STUDENTS and TEACHERS

Classroom Interaction Sheet (based on adapted categories for Flanders Interaction Analysis, HOPKINS and MOORE 1993)

Category Sub-

Category Observed behaviour Tally Total Count

Tea

cher

No

n-D

irec

tive

(co

nd

uci

ve t

o c

lass

roo

m in

tera

ctio

n)

1. Accepts Feeling. Accepts and clarifies an attitude of the feeling tone of a pupil in a non-threatening manner

2. Praises or encourages. Praises or encourages pupil action or behaviour. Jokes that release tension, but not at the expense of another individual; nodding head, saying “um”, “hmm” or “go on” are included

3. Accepts or uses ideas of pupils. Clarifying, building or developing ideas suggested by a pupil. Teachers’ extensions of pupil ideas are included but as teacher brings more of his own ideas into play, shift to category five

Dir

ecti

ve

(no

n-c

on

du

cive

to

cla

ssro

om

inte

ract

ion

)

4. Asks questions. Asking a question about content or procedures; based on teacher ideas, with the intent that the pupil will answer

5. Lecturing. Giving facts or opinions about content or procedures; expressing his own ideas, giving his own explanation or citing an authority other than a pupil Initiation

6. Giving directions. Directions, commands or orders to which a student is expected to comply

7. Criticizing or justifying authority. Statements intended to change pupil behaviour from non acceptable to acceptable pattern; bawling someone out; stating why the teacher is doing what he is doing; extreme self-reference

Stu

den

t

pro

-act

ive 8. Pupil-talk – initiation.

Talk by pupils that they initiate. Expressing own ideas; initiating a new topic; freedom to develop opinions and a line of thought, like asking thoughtful questions; going beyond the existing structure

reac

tive

9. Pupil-talk – response. Talk by pupils in response to teacher. Teacher initiates the contact or solicits pupil statement or structures the situation. Freedom to express own ideas is limited

OF

F T

AS

K

10. Pupil-talk – OFF TASK Talk by pupils that they initiate, but not related to the subject matter (side talk)

11. Pupil non-talk activity – OFF TASK Activity by pupils (except pupil talk) not related to the subject matter

Including all children in quality learning in CEE/CIS Self-assessment Sheet (SAS-1 MoE) Experiences of national education staff

Name of your Department: Please cross this box if

you are MALE

Please cross this box if

you are FEMALE

Instruction: The purpose of this exercise is to collect information inclusive education/CfS activities, and how it might have influenced you in your day-to-day work and/or the way you think about education. The information provided by you will help us improving equitable access to and quality of education.

We therefore ask you to sincerely and carefully provide the relevant and appropriate information as much as possible. There is no “right” or “wrong” answer. The information given will be treated strictly confidential, and will not be used against anybody. Please help us by answering all the questions by ticking the appropriate box, and by supplying the correct information as necessary.

In the following, please indicate how strongly you agree or disagree with the given statements:

# Statement strongly

agree agree disagree

strongly disagree

1 I am familiar with the aims and objectives of inclusive quality education/CfS

2 Teaching standards need to be adjusted/developed to bring them in line with inclusive education/CfS requirements

3 Examples of pilot projects and best inclusive quality education/CfS models have not resulted in new initiatives and/or changes in my Department’s policy or strategic plans

4 It is important that schools allow access to all children regardless their background and/or learning ability

5 Development partner cooperation and support have been instrumental in influencing strategic planning of the MoE/my Department towards greater inclusion of marginalized children

6 Interaction and dialogue with development partners on inclusive, child-friendly and social norms has influenced the thinking about the functioning of education systems within my Department

7 Increasing access to pre-school education is particularly important in disadvantaged areas

8 Parents and children from disadvantaged backgrounds are not interested in education

9 MoE uses inclusive education/CfS standards to monitor progress in education system performance (access, quality)

10 The difference between learning outcomes of education between children from socially, culturally and economically marginalized communities and their peers from majority populations is diminishing

11 Information from studies and assessments on reasons for education exclusion did not change the way MoE/ my Department thinks about education delivery

12 The allocation of education resources is facilitating access to education for children in underserved areas

13 Teaching a diverse group of students has a negative impact on the quality of education

14 I wish there would be more technical support on inclusive quality education since I would like to learn more

15 It is not important that all children of one grade fulfil similar grade level expectations

16 Because of capacity building received, MoE/ my Department now cooperates with international, civil society organisations and/or minority communities in education reform programmes related to inclusive, quality education

17 The diversity between pupils in terms of background/ability is a big problem in some schools

18 It is easier for a pupil to be taught by a teacher sharing the same religion or background

19 Through capacity building, I understand why and recognize how children are being excluded from education

20 It is not possible to close equity gaps in primary school enrolment with regard to poverty and ethnicity

21 Outside school, every ethnic group should be with their own ethnic group



strongly disagree

22 In general, parents have no problem enrolling their disabled child in a regular school

23 It is not important for me to have regular contact with development partners

24 Inclusive education, new teaching methodologies, and assessment requirements do not fit into the existing curriculum framework, teaching and learning standards, inspection requirements, or examinations

25 The school environment is not a safe place for children from minority ethnic backgrounds

26 Information on access to education for children from poorest households has resulted in a reformulation of my Department’s/MoE’s policy/ programmes

27 Partnerships (inter-ministerial, different levels of education, national-international) have been crucial for reducing social and educational inequalities for children

28 It is more difficult for minority children to perform well at school than for majority children

29 To realise a strong and positive future of my country, maximum education benefits need to be ensured for all children

30 Most teachers in pre-school education need training to bring their teaching skills in line with international practices

31 To develop respect for our multicultural and multi-ethnic society, life-skills and critical thinking are as important as the teaching of subject-matter content

32 According to our education policy and regulations, parents have the right to send their child to regular schools regardless their child’s learning ability

33 Finding a job after school is more related to students’ background, ethnicity or gender than to their qualification

34 Separation along ethnic lines and tension between ethnic groups is common in ethnically-mixed schools

35 The Ministry’s EMISystem provides me/my Department with information required to monitor education participation of out of school/ marginalized children

36 Inclusive education and child friendly learning methodology are more costly than the traditional way of teaching

37 According to laws and regulations, schools have to treat children equally, regardless their ethnicity, ability, gender or religion

38 Children with special education needs cannot be taught together with children without special needs

39 Standards for inclusive quality education/CfS are not included in our national curriculum (pre-school and primary levels)

40 It is widely acknowledged that negative stereotyping and intolerance are still part of our school curricula and textbooks

41 It is important to have regular meetings with development partners on issues of exclusion from education

42 I am often approached by colleagues because they are interested in what inclusive quality education /CfS is all about

43 MoE has developed key inclusive education strategies through inter-ministerial cooperation or other partnerships

44 I could not tell an outsider what the idea of inclusive quality education/CfS is all about

45 New policies, standards and frameworks make our education system more inclusive and less discriminatory

46 Education policy and regulations, hamper parents from minority ethnicity to send their child to regular schools

47 To ensure that children perform well, children are taught in their mother tongue

48 Financial resources made available by the Government are not sufficient to implement inclusive quality education

49 The biggest obstacle to implement inclusive quality education/CfS are social norms and attitude


Thank you for your cooperation!

Including all children in quality learning in CEE/CIS Self-assessment Sheet (SAS-2 CO) Experiences of UNICEF Country Office staff

Name of your Department: Please cross this box if

you are MALE

Please cross this box if

you are FEMALE

Instruction: The purpose of this exercise is to collect information inclusive education/CfS activities, and how it might have influenced you in your day-to-day work and/or the way you think about education. The information provided by you will help us improving equitable access to and quality of education.

We therefore ask you to sincerely and carefully provide the relevant and appropriate information as much as possible. There is no “right” or “wrong” answer. The information given will be treated strictly confidential, and will not be used against anybody. Please help us by answering all the questions by ticking the appropriate box, and by supplying the correct information as necessary.

In the following, please indicate how strongly you agree or disagree with the given statements:



strongly disagree

1 I am familiar with the regional vision for basic education/ regional education strategy

2 Teaching standards need to be adjusted/developed to bring them in line with inclusive education/CfS requirements

3 Regional education network meetings and country missions provided by the Regional Office were most beneficial for me/my work on inclusive education policy development and system change

4 The majority of schools do not allow access to all children regardless their background and/or learning ability

5 Development partner cooperation and support have been instrumental in influencing strategic planning of the Ministry and its Departments towards greater inclusion of marginalized children in education

6 Interaction and dialogue with development partners on inclusive, child-friendly and social norms has influenced the thinking about the functioning of education systems within MoE (national and/or decentralized levels)

7 It is the Ministry’s priority to increase access to pre-school education in disadvantaged areas

8 Parents and children from disadvantaged backgrounds are not interested in education

9 MoE/departments use inclusive education/CfS standards to monitor education system performance (access, quality)

10 The difference between learning outcomes of education between children from socially, culturally and economically marginalized communities and their peers from majority populations is diminishing

11 Information from studies and assessments on reasons for education exclusion did not change the way the MoE thinks about education delivery

12 Government allocation of education resources is facilitating access to education for children in underserved areas

13 Teaching a diverse group of students has a negative impact on the quality of education

14 I wish the Regional Office would provide more external expertise on how to mainstream inclusive quality education since I would like to learn more

15 It is not important that all children of one grade fulfil similar grade level expectations

16 Because of capacity building received, I support and cooperate with MoE + other ministries, private and civil society organisations, and/or minority communities in education reform programmes related to inclusive, quality education

17 The diversity between pupils in terms of background/ability is a big problem in some schools

18 It is easier for a pupil to be taught by a teacher sharing the same religion or background

19 Regional studies have given me the insight in and understanding of how children are being excluded from education

20 It is not possible to close equity gaps in primary school enrolment with regard to poverty and ethnicity

21 Outside school, every ethnic group should be with their own ethnic group



strongly disagree

22 In general, it is no problem for parents to enrol their disabled child in a regular school

23 It is not important for me to have regular contact with development partners

24 Inclusive education, new teaching methodologies, and assessment requirements do not fit into the existing curriculum framework, teaching and learning standards, inspection requirements, or examinations

25 The school environment is not a safe place for children from minority ethnic backgrounds

26 Information on access to education for children from poorest households has resulted in a reformulation of MoE policy/ programmes

27 Partnerships (inter-ministerial, different levels of education, national-international) have been crucial for reducing social and educational inequalities for children

28 It is more difficult for minority children to perform well at school than for majority children

29 To realise a strong and positive future of my country, maximum education benefits need to be ensured for all children

30 Most of the teachers in pre-school education need training to bring their teaching in line with international practices

31 The national curriculum acknowledges that for developing respect for our multicultural and multi-ethnic society, life-skills and critical thinking are as important as the teaching of subject-matter content

32 According to our education policies, laws and regulations, parents have the right to send their child to regular schools regardless their child’s learning ability

33 Find a job after school is more related to students’ background, ethnicity, and gender than their qualification

34 Separation along ethnic lines and tension between ethnic groups is common in ethnically-mixed schools

35 MoE EMISystem provides information required to monitor education participation of out of school/ marginalized children

36 Inclusive education and child friendly learning methodology are more costly than the traditional way of teaching

37 According to our regulations schools have to treat children equally, regardless their ethnicity, ability, gender or religion

38 Children with special education needs cannot be taught together with children without special needs

39 Standards for inclusive quality education/CfS are not included in our national curriculum (pre-school and primary levels)

40 MoE acknowledges that negative stereotyping and intolerance are part of school curricula, textbooks/ education system

41 It is important to have regular meetings with national and development partners on issues of exclusion from education

42 I am often approached by colleagues because they are interested in how to mainstream inclusive quality education / trigger system change

43 MoE has developed key strategies through inter-ministerial cooperation or other partnerships

44 I could not tell an outsider what the regional education strategy is all about

45 MoE’s key policies, standards and frameworks have made the education system more inclusive and less discriminatory

46 Our education laws hamper parents from minority ethnicity to send their child to regular schools

47 To ensure that children perform well, children are taught in their mother tongue

48 Financial resources made available by the Government are not sufficient to implement inclusive quality education

49 The biggest obstacle to implement inclusive quality education/CfS are social norms and attitude


Thank you for your cooperation!

APPENDIX 6:

Logic model for UNICEF’s Generic Theory of Change

For reference purposes, the logic model for UNICEF’s Generic Theory of Change is

presented here, as the foundation for UNICEF’s engagement in CEE/CIS Region:

Source: UNICEF Regional Office (2013): Regional Knowledge and Leadership Agenda. Generic Theory of Change underlying UNICEF’s engagement in CEE/CIS Region. January 2013

Included in List of Documents (cf Appendix 3), Ref. D028.

APPENDIX 7:

Programme for country visits

Multi-Country Evaluation of Regional Knowledge And Leadership Areas Area 4: Inclusion Of All Out Of School Children In Quality Learning In CEE/CIS

Visits of evaluation team to country offices – basic schedule:

W/Day Task Responsible

1 Initial meeting at UNICEF office, introductions, briefings, discussion of schedule Evaluation team

2 International consultant:

Consultations with English-speaking counterparts (please see list below)

Round table (in English language) with other relevant development partners

National consultant:

6 school visits (i.e. 2 per day) with Focus Group Discussions (FGDs) and interviews (please see list below)

International consultant (at the centre)

National consultant

(in the field)

3

4

5 Consultations with non-English speaking counterparts (national consultant also to facilitate translation) Evaluation team

6

7 Wrap-up, debriefings, presentation at Country Office Evaluation team

Stakeholders to be consulted:

Interviews Round table discussion Focus Group Discussions

Ministry of Education:

Planning/EMIS;

Inspectorate basic education;

Departments of ECE, special needs education, inclusive education, out-of-school/non-formal/informal education

Teacher Union representative(s)

Teacher Training College representative(s)

NGO (dealing with inclusion) representative(s)

Development partners (in English)

UNICEF CO staff (in English)

School level:

Teachers

Students (mix of grades, male and female)

Parents

8-12 participants per group, to be nominated by school principal

(duration: 1 school hour)

Country visits are planned for the following periods, as discussed and agreed during the video conference on 20 June 2013:

Country Date Days*)

Visiting consultants

International Regional

Kosovo (UNSCR

1244) 02-13 September

1-7 reduced to 1-5: 02-06 September; Team training workshop: 09-13 September

Dr. Joe Pfaffe

Ans Smulders

Vlera Kastrati (All regional consultants for training workshop)

The former Yugoslav


16-24 September 1-5: 16-20 September; 6-7: 23-24 September

Dr. Joe Pfaffe Tatjana Peric

Turkey 16-24 September 1-5: 16-20 September; 6-7: 23-24 September

Ans Smulders Dr. Cennet Engin-Demir

Armenia 26 September-04 October 1-2: 26-27 September; 3-7: 30 September-04 October

Dr. Joe Pfaffe Susanna Tadevosyan

Serbia 26 September-04 October 1-2: 26-27 September; 3-7: 30 September-04 October

Ans Smulders Tatjana Peric

*) In the column “Days”, the matching calendar days for the scheduled 7 working days (numbered 1-7 on the visiting schedule) are given.