Reading Item Development

Juliette MendelovitsAustralian Council for Educational Research

Marian SainsburyN ti l F d ti f Ed ti l R h UKNational Foundation for Educational Research, UK

Test development process for a large scaleprocess for a large‐scale 

assessmentFramework development

Initial item writing Item developer  Cognitive laboratoriesInitial item writing preview

Cog t e abo ato es

Item revision

External reviewItem ready for trialItem ready for trial 

(or rejected)

Trial testingPsychometric and item developer 

i

Final instrument reviewinstrument

Reading itemsReading items

Features of Reading skills / /Features of 

the text / aspects / processes

QuestionsQuestions and 

answersanswers

The importance of text selection

A good textA good text …

i i h d i t ti• is rich and interesting• is optimally challenging – not too hard or too easy

• does not pose artificial challengesdoes not pose artificial challenges• is (more or less) equally accessible and equitable for different candidatesequitable for different candidates

• offers opportunity to pose searching iquestions

Sample Text from PISA 2000 and 2009

MacondoMacondoDazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began They stayed up all night lookingdid not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessive toom‐toom. They became indignant over 

That’s crazy ... But is there some to grow accustomed to its obsessive toom toom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion‐head ticket windows, for a character who had died and was buried in one film, and for whose misfortune tears of affliction 

An intriguing opening sentence ...What inventions???

truth in it?

had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine of illusions that did not merit the emotional outburst of the audience. With that discouraging explanation many felt that they had been the victims of some new and showy gypsy business and they decided not to return to the movies, considering that they already had too many troubles of their own to weep over the acted‐out i f f i i b i

What’s this about? ... A familiar thing from an unfamiliar 

timisfortunes of imaginary beings.perspective

A non‐continuous text (original version)Why are some of the stations 

marked with big circles?How would you get from x to y?

marked with big circles?

Why does this blue line look different to all the other lines?

A non‐continuous text (modified for PISA)

Bad texts: what to avoid

• Giving offence or creating disturbance:T ( id t i l )– Trauma (car accidents, violence)

– Sex, religion, politics or other emotionally contentious issues

– Nasty behaviour, violence, racism, immorality or irresponsibility Undesirable models drugs alcohol inducements to do– Undesirable models – drugs, alcohol, inducements to do anything potentially dangerous

– Bad language• Expecting too much or too little of students

– Texts that rely on esoteric knowledge– Texts that provide general knowledge that most students can be expected to have already

– Texts that are boring!!!Texts that are boring!!!

Text maps:analysing the features of the text

Stories (PIRLS literary)

• ThemeTheme

• Main events

• Main characters

• SettingSetting

• Language features

Example: Fly Eagle FlyExample: Fly, Eagle, Fly

h f d i f b i• Theme: freedom, importance of being true to one’s nature, fulfilling one’s potential

• Main events: farmer finds eagle chick; raises it as a chicken; friend argues for eagle’s true ; f g f gnature; eagle is finally free to fly

• Main characters: farmer friend eagleMain characters: farmer, friend, eagle• Setting: Africa• Language features: ‘you belong not to the earth but to the sky’; description of dawn

Text maps:analysing the features of the text

Non‐fiction (PIRLS informational)

• PurposePurpose

• Theme

• Content

• StructureStructure

• Presentational features

• Language features

Example: The Giant Tooth MysteryExample: The Giant Tooth Mystery

i i f i f i• Purpose: exposition of information• Theme: the development of ideas in the history of science

• Content: how people came to understand that p pfossils were the remains of extinct creatures

• StructureStructure• Presentational features: continuous prose; illustrations and diagrams with captionsillustrations and diagrams with captions

• Language features: definition of ‘fossil’

StructureStructure

Introduction: definition of fossil

Long ago People thought fossils were the bones of giantsg g p g g

Hundreds of years ago Palissey was imprisoned for suggesting extinct creatures

I th 1820 Wif f Gid M t ll f d i t t thIn the 1820s Wife of Gideon Mantell found a giant tooth

Observation 1 Observation 2 Observation 3

Tooth worn down

A plant eater

Age of rocks

A reptile

Reptiles do not chew food

But iguanas do!

A giant extinct reptile ‐ Iguanodon

Today We call them dinosaurs

Reading itemsReading items

Features of Reading skills / /Features of 

the text / aspects / processes

QuestionsQuestions and 

answersanswers

The importance of itemThe importance of item construction

What makes a good item?What makes a good item?

fl f h f k• Reflection of the framework• Centrality to the text• Centrality to the text• TransparencyTransparency• Equityq y• Appropriate item format

Multiple choice itemsMultiple choice items

Stem

Key

DistractersOptions

Multiple choice itemsMultiple choice items

St h ld b l t d t h ld b bl t• Stem should be clear – students should be able to answer question without reading options

• Do not use negatively worded stems• Do not use negatively‐worded stems• Only one of the response options should be the correct answercorrect answer

• Distracters should be plausible based on likely student errors or misconceptionsstudent errors or misconceptions

• Make sure each option is about the same length, and that the grammar of each option “fits” theand that the grammar of each option  fits  the stem

Constructed response itemsConstructed response items

• Used when:– The answer is straightforward and the text does gnot offer plausible distracters or

– Students are required to show a developedStudents are required to show a developed understanding drawing on different parts of the texttext

• Give clear expectations to students with i i lik “ l i ” “ i 3 ”instructions like “explain” or “give 3 reasons”

Developing coding guidesDeveloping coding guides

• Begin by writing a full‐credit answer to the questionq

• Define the levels of partial credit (if applicable) keeping in mind the completenessapplicable), keeping in mind the completeness and accuracy of the response

• Provide examples of partial credit responses

Refining the coding guide for PIRLSRefining the coding guide for PIRLS

• Full‐credit answers and partial credit responses defined at the time the items are written

• But 4th grade students express their answers in unpredictable ways which do not always fit theunpredictable ways which do not always fit the guide

Aft i iti l il t i f l d i• After an initial pilot in a few classes, and again after the field test, we use many examples of 

d ’ fi h id dstudents’ answers to refine the guides and develop training materials

You learn what the farmer’s friend was like from the things he did.Describe what the friend was like and give an example of h h did h h hi

2 – Complete Comprehension

what he did that shows this.

The response describes one plausible character trait (persistent, stubborn, nice, clever, friendly to animals, etc.). In addition, the response provides one example of the farmer’s friend’s actions that are evidence of the character trait.H d t i d H k t t i t t h th l t flHe was determined. He kept trying to teach the eagle to fly.He was kind to animals. He wanted the eagle to be free.

1 Partial Comprehension1 – Partial ComprehensionThe response provides one plausible character trait. Or, the response provides one example of the friend’s actions that are evidence of the friend’s characterthe friend s character. He is kind to animals.He takes the eagle to see the sun and fly away never to live among the chickenschickens. 

Some student responsesSome student responses …• He want to put things right. He want the eagle to fly as normal

eagles do and he don’t want it to be a chicken.• He makes the eagle be a chicken• I have learn that the eagle is cruel• I have learn that the eagle is cruel• He was a good friend he was always having a laugh with you• What things belong to the sky belong to the sky. The farmers g g y g y

friend told that the eagle belong to the sky not the earth• “Let nature choose its path” person. Wanted the eagle to be an

eagleeagle• I’ve learn that animals should have their own family like us• He showed confidence he showed strength because he searched g

after a storm for his calf and he never gave up• He liked to prove things

CodedCoded …• He was kind. He freed the eagle and let it become what it was• He want to put things right. He want the eagle to fly as normal

eagles do and he don’t want it to be a chicken.• “Let nature choose its path” person. Wanted the eagle to be an p p g

eagle• He liked to prove things• What things belong to the sky belong to the sky The farmers • What things belong to the sky belong to the sky. The farmers

friend told that the eagle belong to the sky not the earth• He makes the eagle be a chicken

I h l th t th l i l• I have learn that the eagle is cruel• He was a good friend he was always having a laugh with you• I’ve learn that animals should have their own family like usy• He showed confidence he showed strength because he searched

after a storm for his calf and he never gave up

The importance of unitThe importance of unit construction

A unit is a set of items built around aA unit is a set of items built around a piece of stimulus (or related pieces of 

stimulus)stimulus)

What makes a good unit?What makes a good unit?

• Items cover a range of difficulty

• Items cover a range of framework featuresItems cover a range of framework features

• Each item is independent of all the others

• Items as a group deal with the “heart” of the text

THANK YOU!THANK YOU!