PISA Mathematics AssessmentAPEC Tokyo Feb 2010

Kaye Stacey

University of Melbourne, Australia

Data and images in this presentation are from OECD websites and official publications and from ACER publications on PISA in Australia.

The views expressed here are those of the author and do not represent the OECD or associates.

Outline

• What is PISA and what does it test?• What is mathematical literacy?• A small sample of results

– Country comparisons– Levels of proficiency– Performance of subgroups – Social gradient

• Possibilities for Computer-based Assessment of Mathematics

PISA: Programme for International Student Assessment

• Test years 2000, 2003, 2006, 2009, 2012, ..• 15 year olds• assesses “the knowledge and skills that students have

acquired at school and their ability to use them in everyday tasks and challenges”– reading literacy– scientific literacy– mathematical literacy

• statistically rigorous, to ensure that the results are as meaningful as possible, measuring– student performance – data on the student, family, school and system factors

Key features of PISA (from OECD)

• policy orientation– major aim is informing educational policy and practice– aim to significantly improve understanding of the outcomes of education

• concept of “literacy” (discussed later)• relevance to lifelong learning

– motivation to learn, – attitudes towards learning– learning strategies;

• surveys to explore features associated with educational success– characteristics of students and schools – trends monitored every 3 years;

• breadth– by 2006, around 90% of the world economy– nearly 400 000 students

• Recent studies tracking young people in the years after age 15 show PISA measures knowledge and skills relevant to a life success

Participating countries

• PISA 2000: 43 • PISA 2003: 41 • PISA 2006: 57

– nearly 400 000 students

• PISA 2009: 66 – PISA Plus : +9

• PISA 2012: about 90?

Asia-Pacific 2009

• China– Hong Kong– Macao– Shanghai

• Indonesia• New Zealand• Thailand• Japan• Korea• Australia• Singapore• Chinese Taipei• Chile• Peru• Panama• Argentina• Mexico• USA• Canada• +9 PISA Plus

– Malaysia, …

Survey methods

• Schools randomly selected by PISA (usually 150+)• Random sample of 35+ students per school

– between age 15 yrs 3 mths & 16 yrs 2 mths

• Strict sampling criteria to be included in reports – e.g. Netherlands in 2000 below required so not in trend

data

• Some countries oversample for their own purposes• Each student does 2 hour test and 30 min

questionnaire• Items are in rotating booklets (about 13)

– results of individual students not available/meaningful

TIMSS: Trends in International Mathematics and Science Study

• Independent body, not OECD• Tests every 4 years, since 1994/5• Grade based sample (years 4, 8, 12)• Tests randomly sampled intact classes

– hence teacher survey makes sense

• Aims to test achievement of curriculum goals– Careful and extensive curriculum comparisons

• More Asian countries have participated in TIMSS – Singapore’s high results very famous

Schedule of performance measures

• Additional cognitive assessments: – 2003: problem solving – 2006: computer based assessment of science– 2009: electronic reading – 2012: problem solving– 2012: computer based assessment of mathematics

Now preparing

Questionnaire components

• School context and attitudes– themselves and their homes– attitudes to learning

• School questionnaire, optional teacher questionnaires • TRENDS since 2000 (nearly)

2000 2003

2006 2009 2012

Country rankings

are always

of interest

– statistics

make comparison complicated

Statistically better than Australia - Maths

• PISA 2003– Hong-Kong-China, Finland, Korea, Netherlands,

Liechtenstein, Japan, Canada

• PISA 2006– Chinese Taipei, Finland, Hong-Kong-China, Korea,

Netherlands, Switzerland, Canada, Macao-China

• Movements: 5 stay above Australia, 2 drop to Australia’s group, 1 rises from Australia’s group, 2 new entrants

Percent of students at levels 5 & 6 in PISA mathematics

0

5

10

15

20

25

30

35

Australia2003

Finland2003

Korea2003

Australia2006

ChineseTaipei2006

Finland2006

Hong-Kong-China2006

Korea2006

What percent of students in top performance bands?

Percentage of students below Level 2 in PISA mathematics

0

5

10

15

20

25

Australia 2003 OECD average Australia 2006 OECD average2006

OECD: PISA Proficiency Level 2 “a baseline level of proficiency at which students begin to demonstrate skills that enable them to actively use mathematics”

What percent of students in lowest performance bands?

What is mathematical literacy?

What is mathematical literacy?

• PISA assesses “the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges”

• Reflect recognition that globalisation and computerisation are changing labour markets and societies, and that a different set of skills is needed

• US evidence:– greatest decline in jobs over the past decade has not

been in manual labour, but in routine cognitive tasks – those that can easily be done at less cost by computer (Levy & Murnane, 2006).

Mathematical literacy

• 2003/2006 "an individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgments, and to engage in mathematics in ways that meet the needs of that individual’s current and future life as a constructive, concerned and reflective citizen." 

• Strong links to other concepts– Mathematical modelling (in PISA framework)– Numeracy (but certainly not just “basic skills”)– Quantitative literacy

Sample domain items

• PISA: “Take the test”– Reading – page 13– Science – page 187– Mathematics – page 97

• Questionnaires– Also download from “MyPISA”

Reading

Reading

% teachers unaware

Reading

• Written text followed by questions

• This one: answer as graph

A test across countries needs…

• cultural breadth and balance in tests– bullying, Ministry of Education, …. – not a question of intersection of school curricula around the world

(TIMSS)– school curricula (e.g. reading graphs) influence success rate and

hence usefulness of items for constructing a measure

• high quality in translations– two “masters” for each item (English and French)– translations from both masters compared– back translation etc– some informal language will not translate

• corner vs vertex

English: corner or vertexFrench: vertex

Growing Up

• 6.1 A height of female in 1980 (given increase since then)• 6.2 Explain how graph shows growth rate of girls slows down after 12

yrs of age• 6.3 When are females taller than males of same age?

Growing Up (6.2 – growth rate girls)

• Classification– Scientific; Change and Relationships; Connections– Difficulty 574 PISA score points.

• The question requires students to:– Analyse different growth curves – Evaluate characteristics of data set, represented by graph.– Note and interpret different slopes along graphs.– Reason and communicate the results of this process, within explicit

models of growth.

• OECD average 45%• Most successful countries: Netherlands (77%), Finland

(68%), Belgium (64%), Canada (64%)• Large omission rates: Austria (44%) and Greece (43%).

Scoring for 6.2 (growth rate for girls)• Score (Code) 1 : Response refers to “change” of gradient of female graph, explicitly or

implicitly. • Code 11:Refers to reduced steepness, using daily-life language.

– It does no longer go up, it straightens out; The curve levels off; It is more flat after 12; The line of the girls’ starts to even out and the boys’ line just gets bigger; It straightens out and the boys’ graph keeps rising.

• Code 12: Refers to reduced steepness ,using mathematical language.– You can see the gradient is less; The rate of change of the graph decreases from 12 years on;

(uses words like “gradient”, “slope”, or “rate of change”)• Code 13: Compares actual growth (comparison can be implicit).

– From 10 to 12 the growth is about 15 cm, but from 12 to 20 the growth is only about 17 cm; The average growth rate from 10 to 12 is about 7.5 cm per year, but about 2 cm per year from 12 to 20 years.

• Score (Code) 0• Code 01: Student indicates that female height drops below male height, but does NOT

mention the steepness of the female graph or a comparison of the female growth rate before and after 12 years.– The female line drops below the male line.

• Code 02: Other incorrect responses. For example, the response does not refer to the characteristics of the graph, as the question clearly asks about how the GRAPH shows….– Girls mature early; Girls don’t grow much after 12.

Growing Up (6.2 – growth rate girls)

• Answer type:– Daily life language: over 70% of correct answers in 24

countries– Mathematical language : 56% of correct answers in

Korea– Comparing actual growth: common in Austria (34%);

Mexico (26%), Greece (23%), France and Turkey (19%).

• Common errors– Most common error: not referring to graph e.g. “girls

don’t grow much after 12”. – Around 40% of incorrect answers in France, Korea and

Poland refer to graph, only to show the female height drops below the male height. (concept of gradient??)

Heatbeat (M537) - graph

Heartrate = 200 – age

Heartrate = 208 – 0.7*age• Newspaper statement in

text alerts student to phenomenon

• Question 46.1: from which age does the recommendation increase?

• Question 46.2: write formula for most effective training heartrate (80% of max)

Possible solution to Q46.1

Meaning of “2 out of 3 in next 20 years”

How additional data affects average – complex multiple choice item

Interpreting an unusual

representation

Bookshelves

• Classification– Quantity; Occupational; Connections– Difficulty rating: 499 PISA score points (Mean is set to 500)

• The question requires students to:– Develop a strategy to connect two pieces of information for each

component: how many available, how many needed per set – Use logical reasoning to link that analysis across the components

to produce the required solution.– Communicate the mathematical answer as a real-world solution

(not 5.5 bookshelves)• Most successful:

– Finland and Hong Kong-China (74%), – Korea, the Czech Republic, Belgium and Denmark (72%).

• OECD average: 61% correct, 29% of students attempted & incorrect and 10% did not attempt.

CBAM 2012

2012 CBAM: computer based test of mathematics

• New opportunities for presentation of items to measure same material better– More attractive presentation– Better presentation (e.g. animation)– Better response formats (e.g. move an animation?)

• Able to test some aspects of doing mathematics by computer and so extend notion of literacy to better match world– What are these aspects?

• Recommendation 1 . Raising Visibility and Awareness of the Importance of Mathematical Literacy in the Workplace

• The focus should be:

– The nature of mathematical literacy: that it is anchored in real data, in the context of a particular workplace.

– That maths used in the workplace has economic benefits in the market-place.

– That mathematics may be present quite implicitly in jobs and tasks, which are not obviously mathematical.

– Many employees, regardless of their level of employment, are required to use mathematical literacy.

– That IT and mathematical skills are interdependent.

Mathematical Skills in the Workplace Final Report to the Science, Technology and Mathematics Council, UK, 2002 C. Hoyles, A. Wolf, S. Molyneux-Hodgson & P. Kent

What IT-maths skills in ML?Percent of students at levels 5 & 6 in PISA mathematics

0

5

10

15

20

25

30

35

Australia2003

Finland2003

Korea2003

Australia2006

ChineseTaipei2006

Finland2006

Hong-Kong-China2006

Korea2006

What IT-maths skills in ML?

Use 3-D views

See GoogleSketchUp demo

What IT-maths skills in ML?

Percent of students at levels 5 & 6 in PISA mathematics

0

5

10

15

20

25

30

35

Australia2003

Finland2003

Korea2003

Australia2006

ChineseTaipei2006

Finland2006

Hong-Kong-China2006

Korea2006

Plot graph

What IT-maths skills in ML?

Enter data?

Write formula?

What tools?

(graphics) calculator?

computer – what software?

mobile phone?

How well can you…..

use a spreadsheet

to plot a graph?

• “Do well by myself”– OECD total:

42.09%– Macao:

28.49%– Thailand

25.75%– Australia

58.42%

MyPISA query

What IT-maths skills in ML?

Exploremaths

Open Geogebra demo

Studying subgroups of students

• Gender• Ethnic and home background

– Migration– Language background– Indigenous students

• Social gradient

Social gradient

Mathematical literacy by socio-economic background (Australia)

Graphic shows:mean and confidence interval (white)

5th, 10th, 25th, 75th, 90th, 95th percentiles

Performance against social index (Science 2006)(Note wide spread)

Social gradient (Science 2006)(Sci-literacy score against social index)

From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up

Position of line; strength of relationship; gradient of line; curvature; length of line

From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up

Interpreting the social gradient• Strength of association (variance explained)

– Hong Kong 6.9% < Australia 11.3% < OECD 14.5%– Asian countries tend to be low

• Gradient– Australia 43 > OECD 40 > Hong Kong 26

• Length– Australia has less variation in social index than OECD– US has higher top level than OECD; same bottom

• Position– Australia does better than the OECD average

• Curvature – low and high groups do not differ on relationship (cf NZ, Canada)

• Australia – maths slightly less affected by social index than reading and science – relatively more affected by school

• NOTE: For almost all countries, the effect of school average ESCS outweighs effect of student’s own ESCS.

“Equally prepared for life?” OECD

• Reading PISA2000 – females significantly outscored males in all countries

• Mathematics PISA2003– males often outscored females

• Science PISA2006– no significant difference between males and females overall, but

some difference in patterns of strengths– no significant differences in attitudes to school science– marked differences in expectations of science career

Gender differences in science in schools with different social levels

Australia’s indigenous girls and boys

Graphic shows:mean and confidence interval (white)

5th, 10th, 25th, 75th, 90th, 95th percentiles

Information about PISA

http:// www.oecd.org

https://mypisa.acer.edu.au/

PISA: “Take-the-test”MyPISA: public database and analysis

MyPISA data request

Thoughts and discussion points

• Immense amount of information and excellent reports– Available to you through public databases (except secure ‘trend items’)– But no study answers all questions

• in-depth understanding of thinking e.g. of algebra• What has caused the results (e.g. Finland’s success; Asia’s success)

• An international study operates under severe constraints– Create items that work internationally to measure target construct validly– Anomalies can reveal differences

• e.g. Korea-Australia average, 103 vs 104

• Concepts of mathematical and scientific literacy– Major contribution to educational aspiration in many countries – Still developing

• How does computer-based mathematics affect definition of math literacy?• Has mathematical literacy changed from 2003 to 2012?

• Aim is to find the school systems, schools, teaching and societies that best prepare all future citizens for living productive and satisfying lives

Thank you

k.stacey@unimelb.edu.auhttp:// www.oecd.org

https://mypisa.acer.edu.au/

Stacey, K. & Stephens, M. (2008). Performance of Australian School Students in International Studies in Mathematics. Schooling Issues Digest 2008/1. Canberra

http://www.dest.gov.au/sectors/school_education/publications_resources/schooling_issues_digest/.