romania the ministry of education, research and innovation the national centre for curriculum and...

ROMANIATHE MINISTRY OF EDUCATION, RESEARCH AND INNOVATION

THE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATIONTHE NATIONAL CENTRE FOR CURRICULUM AND ASSESSMENT IN PRE-UNIVERSITY EDUCATIONTHE PISA 2009 NATIONAL CENTER

ww

w.c

nceip

.ro

The OECD PISA Study in Romania. Elements of Impact on Educational Policy

Input for The Regional Workshop on

EVIDENCE BASED POLICY MAKING

Vienna, 9-10 November 2009

Roxana MIHAIL, PhD. Head of the Evaluation and Research Department ,

The National Center for Curriculum and Assessment in Pre-university Educationand PISA 2009 ROU National Coordinator



ww

w.c

nceip

.ro

ISSUES:ISSUES:

1. What is Romania’s “history” related to OECD-PISA Programme?

2. What are (some of) the PISA facts?

3. What lessons have we learned by participating in OECD-PISA?

4. What elements of impact can be highlighted?



ww

w.c

nceip

.ro

1. What is Romania’s “history” related to OECD-PISA Programme?

The acronym: Programme for International Student Assessment

An international comparative study, created by the governments of OECD countries, assessing how well 15-year-old students use their knowledge and skills to meet real-life challenges

A long term programme of assessment, measuring trends over time in students’ performances:

- first cycle – PISA 2000 Reading, Mathematics, SciencePISA 2003 Mathematics, Reading, SciencePISA 2006 Science, Mathematics, Reading

- second cycle - PISA 2009 Reading, Mathematics, SciencePISA 2012 Mathematics, Reading, SciencePISA 2015 Science, Mathematics, Reading

PISA is operating with an innovative concept of literacy: students’ capacity to apply knowledge and skills in key subject areas and to analyse, reason and communicate effectively as they pose, solve and interpret problems in a variety of situations:

- Reading literacy

- Mathematical literacy

- Scientific literacy

It does NOT measure students’ curricular and cross-curricular competencies!

PISA is relevant for lifelong learning: it asks students to report on their own motivation to learn, their beliefs about themselves and their own learning strategies



ww

w.c

nceip

.ro

1. What is Romania’s “history” related to OECD-PISA Programme? (continued 2)



ww

w.c

nceip

.ro


Is the most comprehensive and rigorous international programme to assess students’ performance and to collect data on the student, family and institutional factors that can help explaining differences in performance

- OECD-PISA is applying strict technical standards and quality assurance mechanisms in:

- Translation and adaptation of testing materials- School and student sampling- Data collection and validation

Result: high degree of validity and reliability of results- credibility- transparency in presenting the outcomes of

education in the world’s economically most developed countries, as well as countries at earlier stages of economic development



ww

w.c

nceip

.ro


A global programme:A global programme:

PISA 2000 – 43 participating countriesPISA 2000 – 43 participating countries




WHY participating? E.g.:WHY participating? E.g.:

- To understand relative strengths and weaknesses of educational - To understand relative strengths and weaknesses of educational systemssystems

- To establish benchmarks for educational improvementTo establish benchmarks for educational improvement- To learn from shared experiences about the capacity of educational To learn from shared experiences about the capacity of educational

systems to provide high levels of equity in educational outcomes systems to provide high levels of equity in educational outcomes and opportunitiesand opportunities



ww

w.c

nceip

.ro


Participated in PISA 2000 as a PISA+ country (administering instruments in 2001-2002)

Average: 34 out of 43 participating countries

Did not participate in PISA 2003Average: ?????

Participated in PISA 2006Average: 47 out of 57 participating countries

Is participating in PISA 2009Average: ?????

- Participation in PISA 2012?????



ww

w.c

nceip

.ro

3. What are (some of) the PISA facts?

ROU has less than 0,5% students performing at level 5 at the Reading scale and more than 50% of students at or below level 1!

Challenge: encourage high performance AND minimising poor performance

International report:International report: “Poor performance is relevant to reading literacy”Reading levels have a significant impact on:

- the welfare of individuals- the state of society- the economic standing of countries on international arena

The inequality: for ROU the performance distribution shows the large gap between the 5th and the 95th percentile



ww

w.c

nceip

.ro

3. What are (some of) the PISA facts?(continued 2)

Between PISA 2000 and PISA 2006 Romanian students performances in Reading literacy decreased significantly:

In PISA 2000 the average performance was close to 450 score points

In PISA 2006 the average performance was of 396 score points

The OECD average is 496 for PISA 2006 Reading literacy!

The same decreasing trend is visible for Romanian Students in IEA-PIRLS and IEA-TIMSS international comparative studies!

For ROU, on the Reading scale, 49,3% of the total variance in students’ performance is due to the between-schools variance! (FIN=6,3% of the variance)



ww

w.c

nceip

.ro


In Mathematics PISA 2006 ROU students had an average of 415 score points on a 6 level scale, with results comparable with countries like Bulgaria, Chile, Mexico a.s.o.

The OECD average is 500 score points!

Mathematical literacy = “ … an individual capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgements and to use and engage with mathematics in ways that meet the needs of that individual’s life as constructive and reflective citizen”

For ROU, on the Mathematics scale, 51,3% of the total variance in students’ performance is due to the between-schools variance! (FIN=7,4% of the variance)

In ROU the National Curriculum is still not focused on students’ needs for the knowledge society, but on acquiring the Mathematical scientific content, more or less adapted to age level!



ww

w.c

nceip

.ro


In Science PISA 2006 ROU students had an average of 418 score points on a 6 level scale, with results comparable with countries like Montenegro, Mexico a.s.o.

Scientific literacy = “… an individual’s scientific knowledge and use of that knowledge to identify questions, to acquire new knowledge, to explain scientific phenomena, to draw evidence-based conclusions about science-related issues and….

Understanding of the characteristic features of science as a form of human knowledge and inquiry;

Awareness of how science and technology shape our material, intellectual and cultural environments

Willingness to engage with science with science-related issues and with ideas of science, as a reflective citizen”

For ROU, on the Science scale, 49,3% of the total variance in students’ performance is due to the between-schools variance! (FIN=6,3% of the variance)

The impact of demographic and socio-economic factors on variance in The impact of demographic and socio-economic factors on variance in student performance in Science is student performance in Science is smaller smaller than expected and than expected and smaller smaller than than the impact the impact of school factorsof school factors::

13%13% is accounted for by demographic + SES factors – cf. 15,8% OECD is accounted for by demographic + SES factors – cf. 15,8% OECD av.av.21,5%21,5% is accounted for by school factors – cf. 14,1% OECD av is accounted for by school factors – cf. 14,1% OECD av.. 26%26% is accounted for by between school variance – cf. 18,8% OECD is accounted for by between school variance – cf. 18,8% OECD aavv..

Causes & factors should be searched for elsewhere:Causes & factors should be searched for elsewhere:

Curriculum level ?Curriculum level ?

Teacher training level?Teacher training level?

School culture level?School culture level?



ww

w.c

nceip

.ro


The effect of the highest level of parents’ education on student The effect of the highest level of parents’ education on student performance in Reading since PISA 2000:performance in Reading since PISA 2000:

in PISA 2000 5% of the variance is explained, cf. 8% of the OECD in PISA 2000 5% of the variance is explained, cf. 8% of the OECD averageaverage in PISA 2006 2% of the variance is explained, cf. 6% OECD averagein PISA 2006 2% of the variance is explained, cf. 6% OECD average

Causes & factors should be searched for elsewhere: Causes & factors should be searched for elsewhere:

Attitude towards reading?Attitude towards reading?

Attitude towards testing?Attitude towards testing?

Family environment related to reading?Family environment related to reading?



ww

w.c

nceip

.ro

3. What are (some of) the PISA facts? (continued 6)

The effect of the highest occupational status of parents on student The effect of the highest occupational status of parents on student performance in Science since PISA 2000:performance in Science since PISA 2000:

in PISA 2000 3% of the variance is explained, cf. 10% of the OECD in PISA 2000 3% of the variance is explained, cf. 10% of the OECD averageaverage in PISA 2006 16% of the variance is explained, cf. 11% OECD averagein PISA 2006 16% of the variance is explained, cf. 11% OECD average

Significant variation over a six year period!Significant variation over a six year period!

Causes & factors ?Causes & factors ?

Structural changes in Romanian society? European mobility?Structural changes in Romanian society? European mobility?

The PISA databases are very complex!The PISA databases are very complex!

To explore them efficiently we need training and To explore them efficiently we need training and resources!resources!

Secondary analyses can answer questions about “causes Secondary analyses can answer questions about “causes and factors”and factors”



ww

w.c

nceip

.ro


3. What lessons have we learned by What lessons have we learned by participating in OECD-PISA?participating in OECD-PISA?

There are no universal “recipes” for There are no universal “recipes” for improvement at educational system improvement at educational system level!level!

Each education system must find its Each education system must find its own way of dealing with specific own way of dealing with specific problems!problems!

There are though, examples of There are though, examples of good, efficient practice!good, efficient practice!



ww

w.c

nceip

.ro

3. What lessons have we learned by participating in What lessons have we learned by participating in OECD-PISA? OECD-PISA? (continued 2)

E.G.: In Poland the education reform of E.G.: In Poland the education reform of schooling system implemented in 1999…schooling system implemented in 1999…

Now provides a more integrated Now provides a more integrated educational structures for compulsory educational structures for compulsory educationeducation

Over two PISA cycles, Poland raised its Over two PISA cycles, Poland raised its average performance through increasing at average performance through increasing at the lower end of the performance the lower end of the performance distribution!distribution!



ww

w.c

nceip

.ro


PISA is measuring, sample-based, the system level trends, PISA is not PISA is measuring, sample-based, the system level trends, PISA is not reporting at individual levelreporting at individual level

YET: YET:

our test administrators were reporting (testimonies):our test administrators were reporting (testimonies):

“…“…. I do not know what is the occupation of my mother because she . I do not know what is the occupation of my mother because she left Romania four years ago, to work in Spain and I do not know left Romania four years ago, to work in Spain and I do not know how she is doing…”how she is doing…”

We do not know and do not have the means for investigating it We do not know and do not have the means for investigating it further on, the real , complex “testing effects”, on ROU students:further on, the real , complex “testing effects”, on ROU students:

Psychological, motivational effectsPsychological, motivational effects T he novelty of this testing situationT he novelty of this testing situation T he effects of the test format on ROU studentsT he effects of the test format on ROU students



ww

w.c

nceip

.ro

3. What lessons have we learned by participating in What lessons have we learned by participating in OECD-PISA?OECD-PISA?(continued 4)

““Countries with higher national Countries with higher national income tend to perform better in income tend to perform better in Science, 28% of the variation Science, 28% of the variation between mean scores can be between mean scores can be predicted on the basis of their GDP predicted on the basis of their GDP per capita.per capita.””

……. But the GDP is not an educational measure!. But the GDP is not an educational measure!

The key remains: The key remains: how to make use of the existing resources, no matter how how to make use of the existing resources, no matter how scarce they are?scarce they are?



ww

w.c

nceip

.ro


““In a global context, the future economic In a global context, the future economic and social prospects of both individuals and social prospects of both individuals and countries continue to be dependent and countries continue to be dependent on the results they actually achieve, not on the results they actually achieve, not on the performance that might result if on the performance that might result if they were to operate under average social they were to operate under average social and economic conditions” and economic conditions”

In ROU The PISA results eventually created the In ROU The PISA results eventually created the pressure needed for changes!!!pressure needed for changes!!!



ww

w.c

nceip

.ro



ww

w.c

nceip

.ro

4. What elements of impact can be highlighted?

In 2009 the politics converged with the educational policies!

1. The political discourse is informed by some PISA results

2. A top-down decision was taken:

Starting with the school year 2009-2010 the National Evaluation at the end of grade VIII will administer tests created following “the model of international standardised tests”

4. What elements of impact can be highlighted? (continued 2)

3. In 2008, for the first time, the PISA 2006 National report published has been referred to by both politicians, and policy makers

4. PISA released items were analysed by the National Commissions on core subjects, as well as by the working groups for the national examinations

5. Some of the recommendations made for revising the National Curriculum for lower secondary education (gymnasium) are based on the PISA frameworks (2000 and 2006)

6. The item writers for the National Evaluation at grade VIII have served as markers in PISA 2009 exercise



ww

w.c

nceip

.ro


Aspects of “the top-down effect”:Aspects of “the top-down effect”:

The new legislative package to be adopted (the “Law The new legislative package to be adopted (the “Law of National Education”) contains references to “ the of National Education”) contains references to “ the model of international tests” model of international tests”

Yet, no medium and long term strategy of Yet, no medium and long term strategy of participation to the international comparative participation to the international comparative studies can be secured during economic crisis !studies can be secured during economic crisis !



ww

w.c

nceip

.ro


Each country / education system is finding its own way of Each country / education system is finding its own way of benefiting from the international comparative studies benefiting from the international comparative studies databases….databases….

YET….YET….

Are there any common ways for building up Are there any common ways for building up cooperation at regional level for exploring these cooperation at regional level for exploring these national databasesnational databases??

What would be What would be the next phasethe next phase, after using PISA , after using PISA results for “lobbying for educational reforms”, in results for “lobbying for educational reforms”, in this region? this region?



ww

w.c

nceip

.ro

InformationInformation regardingregarding PISA ROMANIAPISA ROMANIA is availableis available on: on: www.edu.ro

andand

www.cnceip.ro

THANK YOU!THANK YOU!



ww

w.c

nceip

.ro

romania the ministry of education, research and innovation the national centre for curriculum and...

Documents

science pisa

cycle pisa

cycle pisa

pisa facts

oecdpisa programme

oecd pisa programme

participating countries

oecd pisa study