international studies of pupil achievement
DESCRIPTION
England’s “plummeting” PISA test scores between 2000 and 2009: Is the performance of our secondary school pupils really in relative decline? . International studies of pupil achievement. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/1.jpg)
1
England’s “plummeting” PISA test scores between 2000 and 2009: Is the performance of our secondary school pupils really
in relative decline?
![Page 2: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/2.jpg)
2
International studies of pupil achievement
• One of the major advances in educational research over the last twenty years is the collection of cross-nationally comparable information on pupil achievement
• Three major surveys (PISA, PIRLS and TIMSS)
• Children from around 40 countries sit an achievement test (in science/reading/maths) at the same age
![Page 3: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/3.jpg)
3
International studies of pupil achievement• Results from these studies are highly regarded –
especially policymakers
• Often presented as a “league table” where countries are ranked by mean performance of children who sat the test
• E.g. England is ranked 25th (out of 65) in PISA 2009 for reading
• This is very politically sensitive
![Page 4: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/4.jpg)
4
International studies of pupil achievement
• Another aim of these studies is to track educational performance of countries over time
• It is this that has grabbed all the attention since the PISA 2009 were released in December 2010.
• England has apparently dropped dramatically down the international ranking
![Page 5: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/5.jpg)
5
2000 2001 2002 2003 2004 2005 2006 2007 2008 200990
95
100
105
110
115
120
125Educational expenditurePISA mathsGCSE % A-C
Inde
x va
lue
(bas
e =
100
in 2
000)
![Page 6: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/6.jpg)
6
“This is conclusive proof that Labour’s claim to have improved Britain’s schools during its period in
office is utter nonsense. Spending on education increased by £30 billion under the last
government, yet between 2000-09 British schoolchildren plummeted in the international
league tables” Daily Telegraph (national newspaper)
![Page 7: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/7.jpg)
7
“The truth is, at the moment we are standing still while others race past. In the most recent OECD PISA survey in 2006 we fell from 4th in the world in the 2000 survey to 14th in science, 7th to 17th
in literacy, and 8th to 24th in mathematics”David Cameron (Prime Minister)
![Page 8: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/8.jpg)
8
““I am surprised that the right hon. Gentleman has the brass neck to quote the PISA figures
when they show that on his watch the standard of education which was offered to
young people in this country declined relative to our international competitors. Literacy,
down; numeracy, down; science, down: fail, fail, fail.”
Michael Gove (Secretary of State for Education)
![Page 9: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/9.jpg)
9
But is this true???
Here I consider the robustness of the finding that secondary school children in England are rapidly losing ground relative to those in other countries
Look at this in two international datasets (PISA and TIMSS)
Provide some concerns about the data
![Page 10: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/10.jpg)
10
Data: PISA and TIMSS
![Page 11: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/11.jpg)
11
PISA• Conducted by the OECD in 2000, 2003, 2006 and 2009• Test of 15 year old children’s “functional ability”• Three subjects covered (reading, science, maths)• Two stage sample design:
– Schools selected as PSU (with probability proportional to size)– 35 children then randomly selected from within
• “Replacement schools” used to limit impact of non-response• Survey weights
– help correct for non-response – scale data from sample to size of national population
• Test scores created by item response theory (“plausible values”)
![Page 12: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/12.jpg)
12
PISA – number of countries• In PISA 2000 around 40 countries took part.
• By PISA 2009 this had risen to 65
• Most of the countries added were non-OECD, but does include some with high achievement levels (e.g. Singapore, Shanghai-China)
• Impact – Means England can fall down the international league table, even if performance of children has not changed
• I.E. It is easier to come 5th in a league of 40 than it is in a league of 65
• England’s performance has declined, however, even relative to the other OECD countries (who have taken part in all waves)
![Page 13: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/13.jpg)
13
TIMSS• Conducted by the IEA in 1995, 1999, 2003 and 2007• Test of “8th grade” pupils (13/14 year olds) performance on an
agreed “international curriculum”• Two subject areas covered (maths and science)• Two stage sample design:
– Schools selected as PSU (with probability proportional to size)– 1 or 2 classes then randomly chosen
• “Replacement schools” used to limit impact of non-response• Survey weights
– help correct for non-response – scale data from sample to size of national population
• Test scores created by item response theory (“plausible values”)
![Page 14: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/14.jpg)
14
Comparability of PISA test scores over time
• I focus on maths test scores in this paper (subject covered in both PISA and TIMSS).
• Issue - The PISA survey organisers state that the maths scores from 2000 are not fully comparable between 2000 and later waves (2003, 2006, 2009)
• Robustness checks:(a) Present results for all subject – survey combinations (reading is
comparable across all waves)(b) Check results are consistent when using 2003 as the base year
![Page 15: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/15.jpg)
15
Countries included in this study• Only include countries that took part in all the PISA and TIMSS waves
since 1999.
• Compare change in PISA (2000 to 2009) to change in TIMSS (1999 to 2007)
• Leaves ten countries:- Developed (Australia, England, Italy, US)- Asian Tigers (Hong Kong, Japan, South Korea)- Lower income (Hungary, Indonesia, Russia)
• Robustness – loosen inclusion criteria and add six more countries into analysis:
Norway, Sweden, Czech Republic, Netherlands, Scotland, New Zealand
![Page 16: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/16.jpg)
16
International z-scores• PISA and TIMSS raw test scores are not directly comparable –
based on a different array of countries.
• Convert into international z-scores.
• Each country’s mean test score (for each wave of the survey) is adjusted by subtracting the mean score achieved amongst all children in the ten countries for that particular year and dividing by the standard deviation
• Estimates refer to English pupils’ test performance relative to that of children in the other nine countries
![Page 17: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/17.jpg)
17
Results: Do PISA 2009 and TIMSS 2007 agree on where England currently
stands?
![Page 18: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/18.jpg)
18
PISA 2009 versus TIMSS 2007 (cross-sectional)
AUS
HKG
HUN
IDN
ITA
JPN
KOR
RUS USAENG
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
TIM
SS
Mat
hs 2
007
-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0PISA Maths 2009
![Page 19: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/19.jpg)
19
Robustness – broader array of countries
AUSCZE
HKG
HUN
IDN
ITA
JPN
KOR
NLD
NZL
NOR
RUS
SWE
USAENG
SCO
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
TIM
SS
Mat
hs 2
007
-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0PISA Maths 2009
![Page 20: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/20.jpg)
20
Results: Do PISA and TIMSS agree on change in average test scores over
time?
![Page 21: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/21.jpg)
21
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
-0.1
0
0.1
0.2
0.3
0.4
0.5England PISA Maths
England TIMSS Maths
Mea
n te
st sc
ore
(inte
rnat
iona
l z-s
core
)PISA versus TIMSS in England (change over time)
![Page 22: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/22.jpg)
22
AUS
HKGHUN
IDN
ITA
JPN
KOR
RUS
USA
ENG
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4
0.5
Cha
nge
TIM
SS
-0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5Change PISA
Change in TIMSS (99 -07) versus change in PISA (00 – 09)
![Page 23: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/23.jpg)
23
…using a larger number of countries
AUS
CZE
HKGHUN
IDN
ITA
JPN
KORNLDNZL
NORRUS
SWE
USA
ENG
SCOT
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4
0.5
Cha
nge
TIM
SS
-0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5Change PISA
![Page 24: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/24.jpg)
24
AUS
HKGHUN
IDN
ITA
JPN
KOR
RUS
USA
ENG
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Cha
nge
TIM
SS
(Mat
hs)
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3Change PISA (Maths)
AUS
HKGHUN
IDN
ITA
JPN
KOR
RUS
USA
ENG
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Cha
nge
TIM
SS
(Mat
hs)
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3Change PISA (Read)
AUS
HKGHUN
IDN
ITA
JPN
KOR
RUS
USA
ENG
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Cha
nge
TIM
SS
(Mat
hs)
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3Change PISA (Sci)
AUS
HKG
HUNIDN
ITA
JPN KORRUS
USA
ENG
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Cha
nge
TIM
SS
(Sci
)
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3Change PISA (Maths)
AUS
HKG
HUN IDN
ITA
JPN KORRUS
USAENG
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Cha
nge
TIM
SS
(Sci
)
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3Change PISA (Read)
AUS
HKG
HUNIDN
ITA
JPNKORRUS
USA
ENG
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Cha
nge
TIM
SS
(Sci
)
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3Change PISA (Sci)
TIMSS (MATH)
TIMSS (SCI)
PISA (MATH)
PISA (READ)
PISA (SCI)
Change looking at different PISA/TIMSS combinations
![Page 25: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/25.jpg)
25
PISA 2003-2009 versus TIMSS 2003-2007 instead
AUS
HKG
HUN
IDNITA
JPN
KORRUS USA
ENG
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4
0.5
Cha
nge
TIM
SS
-0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5Change PISA
![Page 26: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/26.jpg)
26
Why might this conflict between PISA and TIMSS occur:
Data issues
![Page 27: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/27.jpg)
27
Target population change 1:WALES
![Page 28: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/28.jpg)
28
Data issues – TARGET POPULATION 1Children from Wales were not included in PISA 2000 (but were from 2003 onwards).
Children from Wales typically perform worse than those from England:
– Average PISA score for England (492)– Average PISA score for Wales (472)
Hence potentially drags down the score for England in later PISA waves……
…… does this have much impact?
![Page 29: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/29.jpg)
29
Trend in PISA test scores when excluding Wales
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 20090.2
0.25
0.3
0.35
0.4
0.45
England PISA Maths (no Wales)
England PISA Maths
Mea
n te
st sc
ore
(inte
rnat
iona
l z-s
core
)
![Page 30: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/30.jpg)
30
Target population change 2:Year 10/Year 11 pupils
![Page 31: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/31.jpg)
31
Data issues – TARGET POPULATION 2PISA 2000 / 2003 are AGE BASED samples (children born in the same calendar year)
- Thus PISA 2000 / 2003 includes both year 10 (a third) and year 11 (two-thirds) pupils
PISA 2006 / 2009 are (for all intents and purposes) GRADE BASED samples
- Thus 99.6% of PISA 2006 / 2009 pupils are year 11 pupils
England had special dispensation to make this change
ImplicationsPotential impact upon average performanceEducational inequality …..
![Page 32: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/32.jpg)
32
PISA 2000 PISA 2003 PISA 2006 PISA 2009
Birth year Grade Birth year Grade Birth year Grade Birth year Grade
Birth month st01q03 % Year 11 st02q03 % Year 11 ST03Q03 % Year 11 ST03Q03 % Year 11
January 1984 98.7 1987 82.3 1991 100.0 1994 99.8
February 1984 99.3 1987 82.0 1991 100.0 1994 100.0
March 1984 98.4 1987 99.4 1991 99.5 1994 99.7
April 1984 98.6 1987 99.0 1991 99.3 1994 99.7
May 1984 99.1 1987 98.8 1991 99.8 1994 99.4
June 1984 98.0 1987 98.7 1991 99.8 1994 99.7
July 1984 98.6 1987 99.0 1991 99.3 1994 99.7
August 1984 96.4 1987 99.7 1991 100.0 1994 98.9
September 1984 2.2 1987 0.9 1990 99.5 1993 99.4
October 1984 1.0 1987 0.6 1990 99.8 1993 100.0
November 1984 0.5 1987 2.0 1990 100.0 1993 99.7
December 1984 0.6 1987 0.3 1990 99.4 1993 99.4
![Page 33: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/33.jpg)
33
Month of test
![Page 34: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/34.jpg)
34
Data issues – change of the test month
PISA 2000 / 2003PISA test conducted around April (2 months before GCSE’s)
PISA 2006 / 2009PISA test conducted in November (7 months before GCSE’s)
England has special dispensation to make this change (it did NOT occur in other countries)
![Page 35: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/35.jpg)
35
Data issues – change of the test month
Impact?
Imagine you gave a mock GCSE maths exam to one group of children in November and another in April.
You would expect former to perform worse than the latter.
In other words, PISA 2006/2009 test scores dragged down relative to PISA 2000/2003
By how much?
OECD estimates one year of schooling = 40 PISA test points
Change of five months ≈ 15 PISA test points.
![Page 36: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/36.jpg)
36
Non-response
![Page 37: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/37.jpg)
37
Data issues: PISA Non - response School Pupil
Year SourceBefore
replacementAfter
replacement
2000 Micklewright & Schnepf (2006) 59 82 81
2003 Micklewright & Schnepf (2006) 64 77 77
2006 Bradshaw et al (2007a) 77 89 89
2009 Bradshaw et al (2010a) 69 87 87
Not included in PISA 2003 international report
Investigations (e.g. Micklewright et al 2010):PISA 2000 maths scores upwardly bias by between 4 and 15 points PISA 2003 maths scores upwardly bias by around 7 points
![Page 38: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/38.jpg)
38
PISA Non - responseOf limited use to understand change over time.
Really want to know bias the impact of non-response bias in 2006 and 2009 aswell.
PISA 2009 – England missed target response rate (again) – but we know very little about the impact of this……….
NFER“the NFER was asked to provide some analysis of the characteristics of responding and non-responding schools in England, since it was here that school participation had failed to meet requirements. This showed no significant differences and it was accepted by the PISA sampling referee that there was no evidence of possible bias in the sample as a result of school non-participation”
![Page 39: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/39.jpg)
39
PISA Non - response
…. BUT what does this mean? No information on what the NFER actually provided
• “no significant differences” between responding and non-responding schools – Not surprising because of low power
• What school characteristics compared?
• What significance level used?
• Similar “evidence” was provided in PISA 2000 – but there was still a lot of bias in those figures
![Page 40: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/40.jpg)
40
School Pupil
SourceBefore
replacementAfter
replacement 1999 Martin et al (2000) 49 85 902003 Ruddock et al (2004) 40 54 862007 Sturman et al (2008) 78 86 88
Data issues: TIMSS Non - response
Less attention has been paid to non-response in TIMSS ………. but also England does rather poorly here too
NOTE the jump in school response rate in 2007 - and how this relates to the TIMSS trend.
![Page 41: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/41.jpg)
41
Cumulative impact on the PISA average test score trend in England
![Page 42: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/42.jpg)
42
How does this impact the PISA trend?
Four alternative PISA trends estimated making different assumptions about the comparability of the data.
(1) Raw data are unbiased(2) Correct for change in target population(3) As 2 but correct for change in test month(4) As 3 but correct for response bias
![Page 43: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/43.jpg)
432000 2001 2002 2003 2004 2005 2006 2007 2008 2009
0.2
0.25
0.3
0.35
0.4
0.45M
ean
test
scor
e (in
tern
atio
nal z
-sco
re)
Raw data
![Page 44: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/44.jpg)
442000 2001 2002 2003 2004 2005 2006 2007 2008 2009
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Estimate 1
Estimate 2
Mea
n te
st sc
ore
(inte
rnat
iona
l z-s
core
)Adjustment for change in target population
![Page 45: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/45.jpg)
452000 2001 2002 2003 2004 2005 2006 2007 2008 2009
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Estimate 1
Estimate 2
Estimate 3
Mea
n te
st sc
ore
(inte
rnat
iona
l z-s
core
)… and adjustment for change of test month
![Page 46: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/46.jpg)
462000 2001 2002 2003 2004 2005 2006 2007 2008 2009
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Estimate 1
Estimate 2
Estimate 3
Estimate 4
Mea
n te
st sc
ore
(inte
rnat
iona
l z-s
core
)… and adjustment for non response
![Page 47: International studies of pupil achievement](https://reader035.vdocument.in/reader035/viewer/2022062812/56816374550346895dd45008/html5/thumbnails/47.jpg)
47
Conclusions• Statements suggesting that England is “plummeting down”
international rankings may simply not be true.
• The decline seen by England in the PISA international rankings is not, in my opinion, statistically robust enough to base public policy upon.
• The decline in PISA test scores does not suggest that the Labour government’s investment in education was a waste of money, just as the ascendency in TIMSS rankings does not prove it was well spent.
• Indeed, even if the data were of high enough quality to accurately estimate changes over time, such statements seem to fall into the trap of confusing correlation with causation.