PISAPISA Reading FrameworkReading Framework

Comillas, SpainSeptember 2013

Juliette Mendelovits

September 2013

Leader, PISA reading framework and test development

Why do countries participate in y p pinternational studies of educational

achievement?achievement?

• To collect national-level data when there is noTo collect national level data when there is no other mechanism for doing so

• To compare themselves with other countries

• To be involved in – and help to shape - new p pdevelopments in education

PISA Survey Cycley y

2000 2003 2006 2009 2012 20152000 2003 2006 2009 2012 2015Reading LiteracyMathematical LiteracyScientific LiteracyQuestionnaire

Major domain cycle in yellow

In PISA, framework development takes place during the major domain cycle.

What is an assessment framework?

• An assessment framework is an explicit statement and discussion about what an assessment intends to measure.

• Its purposes are:– To guide test development– To give a common language to stakeholders for

di i f th bj tdiscussion of the subject– To ensure continuity from one year or one grade

level to anotherlevel to another– To communicate the purpose and features of the

assessment program to the publicassessment program to the public

Framework development pbegins with a definition of the

d idomain ...

PISA’s definition of readingReading literacy is understanding, using,

reflecting on and engaging with written textsreflecting on and engaging with written texts, in order to achieve one’s goals, to develop

’one’s knowledge and potential, and to participate in society. y

PISA uses four taskPISA uses four task characteristics in the construction

f ll di lit t kof all reading literacy tasks

• Context• Text format – text type• AspectAspect• Item format

Reported Subscales for PISA Reading 2000 and 2009

Text format– Continuous– Non-continuous Aspects– Mixed– Multiple

Aspects– Access and retrieve

I d– Multiple – Integrate and interpret

– Reflect and evaluate

Text formatText format• Continuous

I f ti• Non-continuous

I f ti– Information presented in sentences and

– Information presented in formats other than prose forsentences and

paragraphs• Argument

other than prose, for example

• Tables • Description• Exposition

I i

• Graphs • Diagrams

M• Instruction• Narration

• Maps• Forms

Mixed

Example of a mixed textmixed text

A text thatA text that comprises

continuous andcontinuous and non-continuous

partsparts

Mixed

Text formatText format• Continuous

I f ti• Non-continuous

I f ti– Information presented in sentences and

– Information presented in formats other than prose forsentences and

paragraphs• Argument

other than prose, for example

• Tables • Description• Exposition

I i

• Graphs • Diagrams

M

Stimulus comprising• Instruction

• Narration• Maps• Forms

comprising several texts of

any kind

Multiple

any kind

Multiple

Text formats

… further categorised by text types

• Continuous• Continuous

N ti

• Narration

• Exposition• Non-continuous p

• Description

• Mixed • Argumentation

• Instruction• Multiple

• Instruction

• Transaction

Aspects

• Access and retrieve

• Integrate and interpret• Integrate and interpret

• Reflect and evaluate

Sample Text from PISA 2000 and 2009MacondoMacondo

Dazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began They stayed upMacondo did not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessive toom-time and effort for them to grow accustomed to its obsessive toomtoom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion-head ticket windows, for a character who had died and was ,buried in one film, and for whose misfortune tears of affliction had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine of illusions that did not merit the emotional outburst of the audience. With that discouraging explanation many felt that they had been the victims of some new and showy gypsy business and they decided not t t t th i id i th t th l d h d tto return to the movies, considering that they already had too many troubles of their own to weep over the acted-out misfortunes of imaginary beings.

MacondoSample access and retrieve item

MacondoDazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began They stayed upMacondo did not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessivetime and effort for them to grow accustomed to its obsessivetoom-toom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion-head ticket windows, for a character who had died and was ,buried in one film, and for whose misfortune tears of affliction had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine

How much did it cost to go to the cinema in Macondo?

of illusions that did not merit the emotional outburst of the audience. With that discouraging explanation many felt that they had been the victims of some new and showy gypsy business and they decided not t t t th i id i th t th l d h d t

One centavoTwo centavosTh tto return to the movies, considering that they already had too many

troubles of their own to weep over the acted-out misfortunes of imaginary beings.

Three centavosFour centavos

MacondoSample integrate and interpret item

MacondoDazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began They stayed upAt the end of the passage why didMacondo did not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessive toom-

At the end of the passage, why did the people of Macondo decide not to return to the movies?time and effort for them to grow accustomed to its obsessive toom

toom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion-head ticket windows, for a character who had died and was ,buried in one film, and for whose misfortune tears of affliction had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine of illusions that did not merit the emotional outburst of the audience. With that discouraging explanation many felt that they had been the victims of some new and showy gypsy business and they decided not t t t th i id i th t th l d h d tto return to the movies, considering that they already had too many troubles of their own to weep over the acted-out misfortunes of imaginary beings.

MacondoSample reflect and evaluate item

MacondoDazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began They stayed upMacondo did not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessive toom-time and effort for them to grow accustomed to its obsessive toomtoom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion-head ticket windows, for a character who had died and was

Do you agree with the final judgement of the people of Macondo about the value of the movies?,

buried in one film, and for whose misfortune tears of affliction had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the

the movies? Explain your answer by comparing your attitude to the movies with theirs.

difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine

attitude to the movies with theirs.

of illusions that did not merit the emotional outburst of the audience. With that discouraging explanation many felt that they had been the victims of some new and showy gypsy business and they decided not t t t th i id i th t th l d h d tto return to the movies, considering that they already had too many troubles of their own to weep over the acted-out misfortunes of imaginary beings.

Example of students’ answers to Macondo reflect and evaluate itemreflect and evaluate item

Do you agree with the final judgement of the people of Macondo about the value of thepeople of Macondo about the value of the movies? Explain your answer by comparing your attitudeto the movies with theirs

Yes, I agree, there’s enough suffering in the world without making it up

to the movies with theirs.

world without making it up.No, people understand that when you go

to the movies what happens on the screen to the movies what happens on the screen isn’t real.

It d d If th fil i bbi h I f l lik It depends. If the film is rubbish, I feel like walking out, but if it’s good you are caught up in it and don’t mind if it’s not true.

Item formatItem formatAbout

• Selected responseMultiple choice

About55%

How much did it cost to go to the cinema– Multiple choice– Complex multiple choiceHow much did it cost to go to the cinema in Macondo?

One centavoTwo centavos

About

Two centavosThree centavosFour centavos

• Constructed responseCl d t t d

45%

D ith th fi l j d t f th l f– Closed constructed response– Short constructed response

Do you agree with the final judgement of the people of Macondo about the value of the movies? Explain your answer by comparing your attitude to the movies with theirs

– Extended constructed responsemovies with theirs.______________________________________________________________________________________

Why do countries participate in y p pinternational studies of educational

achievement?achievement?

• To collect national-level data whenTo collect national level data when there is no other mechanism for doing soso

• To compare themselves with other pcountries

• To be involved in and help to shape• To be involved in – and help to shape -new developments in education

PISA Survey Cycley y

2000 2003 2006 2009 2012 2015Reading LiteracyMathematical LiteracyScientific LiteracyQuestionnaire

PISA Survey Cycley y

2000 2003 2006 2009 2012 2015Reading LiteracyMathematical LiteracyScientific LiteracyQuestionnaireProblem Solving

PISA Survey Cycley y

2000 2003 2006 2009 2012 2015Reading LiteracyMathematical LiteracyScientific LiteracyQuestionnaireProblem SolvingFinancial LiteracyFinancial Literacy

PISA Survey Cycley y

2000 2003 2006 2009 2012 2015Reading LiteracyMathematical LiteracyScientific LiteracyQuestionnaireProblem SolvingFinancial LiteracyFinancial LiteracyComputer-based Science Digital

readingThree domains

ALLdomains

NEW DIRECTIONS IN PISANEW DIRECTIONS IN PISA READING:READING:

DIGITAL READING

What is digital reading?Reading literacy is understanding, using,

reflecting on and engaging with written reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potentialdevelop one s knowledge and potential, and to participate in society. (PISA definition of reading literac )(PISA definition of reading literacy)

• In the 21st century, proficiency in reading means proficiency in both print and digital media

• Digital reading is not just reading print text on a computer screen

Analogies …Analogies …Page of a goIn the print medium In the electronic mediumPage of a go

Page of a book Web page

A book A web site

Bibliography, references, “further reading”

Links on a web site to other sites

A very very very big library The Internet

In PISA digital reading isIn PISA, digital reading is conceptualised in terms of two

components

• Text processingText processing• Navigation

Relationship between navigation and text processing for digital reading taskstext processing for digital reading tasks

Example of a PISA digital reading task (1)g g ( )

Example of a PISA digital reading task (2)g g ( )

Example of a PISA digital reading task (3)g g ( )

DRA Sample ItemsDRA Sample Items

http://erasq.acer.edu.auhttp://erasq.acer.edu.au

Thank youyFor more information about PISA go tog

www.pisa.oecd.org