Rod Johnson & Sandra Johnson Assessment Europe

Generalise not specialise: design implications for a

national assessment bank

Some history

Original AAP database

Re-conceptualisation

Assessment materials

Variety of subjects and formats – suitable for pupils aged 7 to 14

Pencil & paper and practical Several years’ of materials resource to be

recovered Existing AAP materials used to initially

populate the 5-14 National Assessment bank

Example 1

Example 2

Stimulus text In this example, Attila the Hen, a 420-word passage recalls events at Sunnycluck Farm just after all the hens have made their escape. Attila realises that the other hens are looking to her to lead them. Section A: 10 multiple-choice questions

Section B: a sequencing task

Section C: a summary completion exercise with 9 gaps

Example 2 continued

Example 2 continuedHere is a summary of part of the story after the farmyard battle. Fill each gap with one or more words. You may use words from the story or your own words. Attila watched as the men returned to the _______________________. 1 She decided to find out if ______________________ had survived. 2 Taking a ________________________ she __________________________. 3 4 Soon she had assembled __________________________. 5 Attila realised that her _________________________, the old hen, 6 was _______________________. 7 She decided she would have to _______________________ the other hens 8 by herself because they _______________________ her. 9

Example 3

A generalised ontology for storing assessment materialsTask: a set of questions, grouped into one or more sections or

subtasks, normally based on a shared stimulus (text, picture, video clip …)

Subtask: a collection of one or more items, based on the same stimulus material and usually, though not necessarily, sharing other common properties such as format, theme, level of difficulty; from the point of view of presentation subtasks are often labelled as sections; a subtask is characteristically the smallest unit of assessment whose external form can be independently stored

Item: normally the smallest element of assessment which can be scored, though computation of the item score may involve consideration of responses to several constituent, usually interdependent, subitems

Subitem: the smallest element of assessment for which a response can be recorded.

Example 4: single-item task

Example 2 Section C: multi-item subtaskHere is a summary of part of the story after the farmyard battle. Fill each gap with one or more words. You may use words from the story or your own words. Attila watched as the men returned to the _______________________. 1 She decided to find out if ______________________ had survived. 2 Taking a ________________________ she __________________________. 3 4 Soon she had assembled __________________________. 5 Attila realised that her _________________________, the old hen, 6 was _______________________. 7 She decided she would have to _______________________ the other hens 8 by herself because they _______________________ her. 9

Example 3 revisited: multi-subitem single-item task

Example 5: a 2-item task

Key question What should properly be stored in the

assessment bank?

Our answer Task images Resource materials Descriptive metadata

but not Marking details Test construction algorithms Statistical metadata

Distributed information system architecture

Materials database

(“item bank”)

Schools database

IMS (SSA)

Mark assignment

Test construction

Performance data

Usage data

Survey management

IMS (NA)

Mark assignment

Test construction

Performance data

Usage data

Test delivery