BUILDING

EDUCATIONAL

INDICATORS FOR

INTERNATIONAL

COMPARISONCamila de Moraes - OCDE

Brasília-DF | Junho 2018

• Introduction

• Comparable raw data

• Comparable indicators

• Conclusion

OUTLINE

Introduction

• OECD mission: provide a forum in which governments can work together to

share experiences and seek solutions to common problems. An important part of

this work is building robust, comparable datasets that will help countries share

best practices.

• Education at a Glance: annual flagship publication on the state of education

around the world. Comparability is our main value-added.

• The report covers all 35 OECD countries and a number of partner countries

(Argentina, Brazil, China, Colombia, Costa Rica, India, Indonesia, Lithuania, the

Russian Federation, Saudi Arabia and South Africa).

Building comparable indicators

What do we want to measure?

Which indicator should be used to measure it and how should it be calculated?

What data do we need and how comparable is it?

Building comparable indicators

What do we want to measure?

What data do we need and how comparable is it?

Which indicator should be used to measure it and how should it be calculated?

Building comparable indicators

What do we want to measure?

What data do we need and how comparable is it?

Which indicator should be used to measure it and how should it be calculated?

1. Raw data

2. Indicator

development

Building comparable indicators: our general principles

• Which does not always mean comparability in the raw data collection

Main goal is indicator comparability

• Trade-off between stricter comparability and more data available

• Trade-off between the “most refined” and the “most comparable” data

Comparability implies trade-offs

• Must accept a degree of flexibility

100% comparability does not exist

• Introduction

• Comparable raw data

• Comparable indicators

• Conclusion

OUTLINE

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

ISCED classifications

• Collaboration between internationalorganizations, experts, and country representatives

• Classification must evolve over time toreflect important changes in educationsystems (e.g. ISCED 2011 classification reflects the Bologna Process in tertiary education)

Manuals for data submission

• Definition of concepts (even seemingly“obvious” ones like teachers, students)

• Scope (what should beincluded/excluded)

• Time-frame for data collection

• How to deal with borderline cases

Manuals: example of level of detail

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

Collect metadata

• Sources of data

• Reference year

• How does data reported differ from standard definition?

• How does data reported differ from specified scope?

• Any additional information

Collect metadata

• Sources of data

• Reference year

• How does data reported differ from standard definition?

• How does data reported differ from specified scope?

• Any additional information

+ Control codes and comments which also provide information on reported data

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

Identifying potential comparability problems

• ISCED Mappings

• Metadata

• Discussions with countries

• Priority rating exercise

• Countries are asked to rate indicators based on policy relevance and technical quality

• Helps prioritize the areas of work

Identifying potential comparability problems: Twoexamples

• Comparability in mapping:

• Early childhood education programmes

• Comparability in definitions/reporting:

• Household expenditure on education

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

Comparability of early childhood educationmapping

• OECD conducted an ad-hoc survey to collect more information on early childhood education programmes (ISCED 01 and ISCED 02).

• Survey collected data on which ISCED criteria were met for inclusion of programmes and which were not.

Overview of the different Early Childhood and Care systems across OECD countries

Comparability of reporting household expendituredata

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

Structure

Oversees strategic direction and coherence of OECD’s work on education

Set priorities and standards for data development and provide direction on the dissemination of the programme’s work

NESLI and LSO: technical discussions to develop and refine indicators

Focus on specific subjects and develop proposals that will be discussed at the INES Working Party and INES Networks

Partners:

Ensuring comparability in raw data: continuous process

Set harmonized standards/definitions

and map data

Collect metadata to assess comparability

Identify potential comparability

problems

Collect more information from

countries (e.g. ad-hoc surveys)

Provide a forum for countries and

experts to discuss

Adapt the data collection, indicator

calculation and reporting accordingly

How to deal with the comparability challenges in the raw data

Comparability issues in the raw data

Clarify the data collection or the manual

Document the comparability issues and set up working groups to continue to work on the topic

Build indicators that maximize comparability given the raw data

How to deal with the comparability challenges in the raw data

Comparability issues in the raw data

Clarify the data collection or the manual

Document the comparability issues and set up working groups to continue to work on the topic

Build indicators that maximize comparability given the raw data

Raw data is comparable

• Introduction

• Comparable raw data

• Comparable indicators

• Conclusion

OUTLINE

Ensuring comparability in indicators: continuousprocess

Restrict indicators to what is most comparable

Assess the impact of different data

sources/methodologies on indicator

Assess the comparability in

interpretation of the indicator

If possible, adapt or refine indicators

Agree on what is publishable and how to

document the comparability issues

Ensuring comparability in indicators: continuousprocess

Restrict indicators to what is most comparable

Assess the impact of different data

sources/methodologies on indicator

Assess the comparability in

interpretation of the indicator

If possible, adapt or refine indicators

Agree on what is publishable and how

to document the comparability issues

Restrict indicators to what is comparable: example of household expenditure

• Results from the ad-hoc survey on the comparability of household expenditure data

showed a considerable degree of variation in reporting across countries.

• Comparability is particularly challenging in reporting expenditure outside educational

institutions (e.g. transportation, rent). As a result, the indicator “expenditure per

student” only takes into account expenditure on educational institutions.

• The Secretariat will continue to work with countries to improve their reporting of

household expenditure. Degree of quality and comparability will be continuously

reassessed.

Restrict indicators to what is comparable: example of earlychildhood education

• Given the comparability challenges of reporting data on early childhood education

programmes, indicators for this level of education were removed from the totals in

Education at a Glance, and grouped into one chapter dedicated exclusively to this level

of education.

• For example, expenditure on education only shows the total from primary to tertiary

(ISCED1 to ISCED 8), and does not include ISCED 0. Expenditure on ISCED 0 is

reported separately in a specific chapter that more adequately explains the challenges and limits in reporting data for these programmes.

Ensuring comparability in indicators: continuousprocess

Restrict indicators to what is most comparable

Assess the impact of different data

sources/methodologies on indicator

Assess the comparability in

interpretation of the indicator

If possible, adapt or refine indicators

Agree on what is publishable and how to

document the comparability issues

Assess impact of different data sources/methodologies on indicator (1/2)

Compare methods:

• Simulations

• Data for countries where both methods are available

Conclusion:

• Given the magnitude of differences, agreed to present them separately and note the overestimation of cross cohort

Assess impact of different data sources/methodologies (2/2)

Ensuring comparability in indicators: continuousprocess

Restrict indicators to what is most comparable

Assess the impact of different data

sources/methodologies on indicator

Assess the comparability in

interpretation of the indicator

If possible, adapt or refine indicators

Agree on what is publishable and how

to document the comparability issues

Interpretability: importance of national context even with comparable raw data

• Luxembourg has the lowest first-time graduation rate for

tertiary education among OECD countries. However, that’s

because 75% of secondary school graduates pursue

tertiary studies abroad.

• First-time graduation rates in Australia and New Zealand

are the highest of all OECD countries, but only because of

the high share of international students.

Ensuring comparability in indicators: continuousprocess

Restrict indicators to what is most comparable

Assess the impact of different data

sources/methodologies on indicator

Assess the comparability in

interpretation of the indicator

If possible, adapt or refine indicators

Agree on what is publishable and how to

document the comparability issues

Evolution of indicators to increase comparability in interpretation

Gross graduation rate

Net graduation rate

First-time net graduation rate

First-time net graduation rate excluding international students

First-time net graduation rate excluding international students and below the typical age

Evolution of indicators to increase comparability in interpretation

Ensuring comparability in indicators: continuousprocess

Restrict indicators to what is most comparable

Assess the impact of different data

sources/methodologies on indicator

Assess the comparability in

interpretation of the indicator

If possible, adapt or refine indicators

Agree on what is publishable and how to

document the comparability issues

Metadata in published book

Footnotes

• Reference years

• Type of institution (e.g. only public institutions available)

• Inclusion/exclusion of a different category if not available in the table

Text analysis

• General limits to interpretability

• National context, policies, reforms, that can help interpret the data

• (e.g. Israel’s mandatory military conscription)

Annexes

• Country-specific details on definitions, coverage, methodology, context

• (e.g. non-inclusion of special-need institutions)

Metadata in online database

• The possibility of including metadata, methodology explanations and country-specific information in an online database is more limited.

• Avoid publishing indicators that require more metadata/explanations to be correctly interpreted (e.g. completion rates, financial returns to education)

• Introduction

• Comparable raw data

• Comparable indicators

• Conclusion

OUTLINE

Conclusion

• Ensuring comparability is a continuous process of improvement of both the raw data and the indicator.

• It is not possible to achieve 100% comparability, so there must be a degree of flexibility.

• Must balance between stricter comparability and less available data.

• Our main strategy to ensure comparability is to work closely with countries and provide opportunities for countries and experts to discuss.

Acesse nossas redes sociais Conheça nossas publicações

MUITO OBRIGADA!

TWITTER@OECDEduSkills

YOUTUBE@EduSkills OECD

PORTAL INEP portal.inep.gov.br

Education at a glance: www.oecd.org/education/education-at-a-glance-19991487.htm

Country notes:http://www.oecd.org/education/educationataglance2017-countrynotes.htm