Applying a concept model of learning-oriented language assessment

to a large-scale speaking test

Liz Hamp-Lyons and Tony Green

Centre for Research in English Language Learning and

Assessment

University of Bedfordshire

2

Acknowledgement

•  This project was carried out with Round 4 funding from

Cambridge English Language Assessment

October 2014 University of Bedfordshire 2

3 University of Bedfordshire 3

In our study, we

• developed a concept model for learning-oriented language assessment (LOLA)

• piloted the model with current online materials for teachers engaged in Cambridge English: First preparation , and a small sample of Cambridge English: First Speaking test events

• considered how far a LOLA approach is reflected in these data

• made suggestions for ways to increase the language learning opportunities in the Cambridge English First speaking context

4

Starting from teaching and learning

The input to our language-specific model comes from the work of

1.  David Carless and colleagues at the Hong Kong Institute of Education (now

continuing at Hong Kong University)

2.  Paul Black and colleagues, including Dylan Wiliam, at Kings College London

3.  more recently and specifically in relation to language education, Jim

Purpura, Carolyn Turner and colleagues

Each of these projects/sources start from the inside and work out, i.e., from the

learner, the teacher, the classroom out to the wider context and to assessment

5 University of Bedfordshire 5

Ways to elicit language for assessment inside and outside the classroom Turner, C.E. & Purpura, J.E. (forthcoming)

6

The model of language-oriented assessment emerges from assessment for learning

Learning-oriented assessment Large-scale testing

Learning-focused tasks Judgement-focused tasks

Scaffolded task completion Support not permitted

Interactive/exploratory questioning Interlocutor/assessor questioning

Learner-involved assessment Learner-excluded assessment

Self-assessment Authority assessment

Peer evaluation Authority assessment

Learning-focused feedback Judgement-focused feedback

Interlocutor/assessor scaffolding Summary decision reporting

Immediate feedback Delayed or no feedback

Focus on feed-forward No feed-forward

Assessment is contingent Assessment is absolute

7

The critical addition of interactivityÜ learning-oriented language assessment

Learning-oriented language assessment is inherently interactive

Large-scale testing inevitably under- represents the constructs of language

8

L e a r n i n g - o r i e n t e d l a n g u a g e assessment

Large-scale testing

Learning-focused tasks Judgement-focused tasks

Scaffolded task completion Support not permitted

Interactive/exploratory questioning Interlocutor/assessor questioning Learner-involved assessment Learner-excluded assessment

Self-assessment Authority assessment Peer evaluation Authority assessment

Learning-focused feedback Judgement-focused feedback

Interlocutor/assessor scaffolding

Summary decision reporting

Immediate feedback Delayed or no feedback

Focus on feed-forward No feed-forward

Assessment is contingent Assessment is absolute

Learning-oriented language assessment is inherently interactive

Large-scale testing inevitably under-represents the constructs of language

Learning-oriented language assessment LOLA

9

Aims

•  We wanted to know if the LOLA model would work

•  Like Assessment for Learning (AfL) our model of Learning-Oriented

Language Assessment (LOLA) builds theory on practice and

observation in/of classrooms – of teachers, learners, materials and

activities

•  In looking at possibilities for a dynamic and productive relationship

between teaching/learning and assessment, we wanted in particular –

in keeping with our assessment for learning sources - to start from

teaching/ learning.

10

Investigating Cambridge English: First Speaking

•  Cambridge English Language Assessment has a history of aiming to

bring together assessment and learning (see, e.g., Jones 2013

Dubai). However, in looking at this relationship, Cambridge English

Language Assessment has usually started from the perspective of

assessment of learning, e.g., Jones, Hamilton & Saville (ILTA/AAAL

2013): “The renewed engagement with education creates pressure to

change the relationship between educational assessment and the

process of learning and teaching.“

11

Cambridge English and LOA Jones 2013

University of Bedfordshire 11

12

The Speaking Test Investigating interaction in Cambridge First Speaking Tests

13

Looking for the language + learning orientation in recorded speaking test events

• We began with the three key elements of assessment that is learning oriented identified by Black & Wiliam (1998) and Carless et al (2007):

•  learning tasks

•  student involvement in self and peer evaluation

•  feedback as feed-forward

•  The emphases on questioning and scaffolding also come from the work of Black and colleagues, and from recent work in Vygotskyan theory and dynamic assessment, as well as decades of classroom observation research reporting

14

Finding opportunities

• On a first pass through the video data, we did not find any evidence of these things actually happening.

•  At this point we began to theorise why we were not seeing what we hoped to see: as we looked more closely, it was increasingly borne in on us that the possibilities for a learning orientation are in the hands of the exam interlocutor.

• We thus needed to identify features of language learning-oriented behaviour. We therefore carried out detailed study of the video recordings and identified 7 behaviours that by their presence or absence could facilitate or inhibit the quality, quantity and kind of speaking the examinees produced.

15

Recognising what is not there

• The aspects of interlocutor behaviour we looked at are:

" body language

" topic maintenance and management

" intonation, pacing, pausing

" clarification strategies (or absence of)

" error correction or repair

" management of turn-taking in paired section

" elicitation of candidate-candidate interaction

16

Interactivity and speaking test TASKS

• Opening the first Task—Interview

Interlocutor (I): (eyes down) First of all (makes eye contact towards both candidates) we’d like to know something about you, so I’m (eyes down) going to ask you (glances up) some questions (eyes down) about yourselves (3 sec pause/silence, eyes still down)… (looking up) where are (looking at S1) you from .. (name)?

17

Interactivity and speaking test TASKS

Interlocutor (I): (eyes down) First of all (makes eye contact towards both candidates) we’d like to know something about you, so I’m (eyes down) going to ask you (glances up) some questions (eyes down) about yourselves (3 sec pause/silence, eyes still down)… (looking up) where are (looking at S1) you from .. (name)?

• We take our first impressions of people and contexts from body language. Here:

• Eye contact is problematic • Pausing may be misleading, or cause nervousness,

especially as the I does not nominate the speaker until the last moment

18

Interactivity and opportunities for learner engagement

• Topic maintenance and shift in the Interview:

S1: I like my Korean food and (0.50-1.03--very hard to understand pronunciation: looks like S2 is not understanding S1)

[During this 13 second turn there is no clarification request, request for repetition, or error correction, from I]

I: (smiles) ‘kay (draws breath creating a pause: renews eye contact) Do you come from a large or small family?

19

Interactivity and opportunities for learner engagement

•  Topic maintenance and shift in the first Task:

S1: I like my Korean food and (0.50-1.03--very hard to understand pronunciation: looks like S2 is not understanding)

[During this 13 second turn there is no clarification request, request for repetition, or error correction, from I]

I: (smiles) ‘kay (draws breath creating a pause: renews eye contact) Do you come from a large or small family?

Unannounced topic shifts are hard for NS to follow; more so and also demotivating for NNS in exam conditions

The opportunity for scaffolding, or for asking a clarification question, is missed.

20

The individual long turn

The examiner gives you a pair of photographs to talk about and you have to speak for 1 minute without interruption. As a reminder, the questions you have to answer about your photographs are written at the top of the page. When you have finished speaking, your partner has to answer a short question from the examiner about your photographs.

21

The individual long turn

•  The examiner gives you a pair of photographs to talk about and you have to speak for 1 minute without interruption. As a reminder, the questions you have to answer about your photographs are written at the top of the page. When you have finished speaking, your partner has to answer a short question from the examiner about your photographs.

• One candidate is asked to answer questions about a photo while the other listens; this may create an appearance of interaction but no interaction was observed.

• There were no requests for clarification and no error repair.

•  NB: This photo was downloaded from the Cambridge English Language Assessment webpage on Cambridge English: First

22

Language learning opportunities in the Individual long turn?

" In terms of the principles of LOLA we would suggest that the Long Turn is a structure with the potential for the Interlocutor to scaffold, reformulate or narrow the task to provide each candidate with enough grounding to get started on the intended task.

" Could this Task-type be moved from Task 2/Part 2 to Part 3? That should give the Interlocutor enough time to form a broad judgement of the general speaking proficiency of each candidate, in order to shape the task appropriately.

23

LOLA and Part 3- Collaborative Task

The task is to choose from a selection of pictures of holidays, and discuss: which ones would be best?

S2: So “C”, what do you think about we visit Europe by train? S1: By train? Um.. it’s a good... it‘s a good transport, train.. because

it’s quite safe.. you don’t have to take care where you are going… and you are very quickly from one city to the other one… and you can (mumbles)…

S1: [yeah… S2: [… different cities S1: (loudly) … and you can go with your friends S2: yes, yes S1: and I think for young people you have (hand movement) very..

cheap prices… to go by train around Europe S2: yes, yes S1: mm hmm S2: … and very safe S1: yeah

24

LOLA and Part 3- Collaborative Task

•  This was a quite natural interaction between 2 North European males (in their 20s, it seemed) both of whom were performing at or above the level of Cambridge First.

•  In the next extract, 2 Greek, younger test takers have clearly been fully prepared for this task type. Shown the pictures, they begin to speak instantly and use a lot of formulaic sequences.

25

LOLA and Part 3- Collaborative Task

S1: “Ye-es… I agree with you (flat)… but then um.. I think this job (points) would be quite interesting and er nice to hear about and er.. to have it (downtone).

S2: er.. I disagree with you (flat) because if you do that job for many… er … for long time, at last you ‘ll be bored I think (flat)

S1: (flat) Ye,,es.. -and you will (slightly lifts the tone and speeds delivery) be fat also. (M looks at her and laughs: she has not looked up, but smiles when she hears his chuckle)

The two students haven’t looked at each other until S1 (female) says “and you will be fat”. Has she suddenly ‘engaged her brain’ with what she is doing?

How can we tune candidates in to the importance of interaction before they are buried under the weight of exam-itis?

26

Self-directed or undirected interaction?

•  In terms of management of turn-taking

•  in Parts 3 and 4 this is all done by the candidates themselves, and in most cases it seems to result in stilted turn exchanges.

•  In terms of elicitation of candidate-candidate interaction

•  again, in Parts 3 and 4 this is all done by candidates themselves.

•  Could the Interlocutor take on a peer role in Part 4 Discussion?

•  To introduce a ‘third voice’ if the Interlocutor isn’t going to interact, the Discussion could be delivered on a computer screen in the exam room, creating a locus of engagement for the candidates.

27

The critical addition of interactivityÜ learning-oriented language assessment

Learning-oriented language assessment is inherently interactive

Large-scale testing inevitably under- represents the constructs of language

Learning-oriented assessment Test preparation practices Tasks designed to st imulate effective learning

Tasks limited to those mirroring the test

Learners active Learners passive Learners engage in self- and peer-evaluation

Learners take practice tests - receive scores

Focus on feedback Feedback beyond scores is rare Strategic use of questioning Teacher-delivered information Scaffolded performance Solo performance UPTAKE of feedback Learners may choose to take up

feedback: unsystematic

28

A key implication

•  In concluding our report on this study we argued that for effective LOLA to occur in large-scale speaking tests, more extensive observation and analysis of interlocutor language behaviours is needed to determine which of these is more critical, when, and why; and also, relatedly, fully scaffolded training of interlocutors for the interlocutor role is critical.

•  This suggests that there is a need for:

•  Interlocutor training materials development

•  trainer training for lead speaking examiners

•  ongoing observations of interlocutor skills implementation

•  refresher training for interlocutors on a periodic basis

29

Next Steps

In a further study we plan to

• explore how teachers prepare for Cambridge Speaking tests and how they integrate Cambridge test tasks into their teaching practices. We will make use of the framework for Learning Oriented Language Assessment (LOLA) developed in the this study. We will analyse classes and interviews with teachers using data from the Progetto Lingue Impact Study.