a multi-method approach to writing...a multi-method approach to writing scale development how can we...

A MULTI-METHOD APPROACH TO WRITING

SCALE DEVELOPMENT

ZEYNEP AKSIT

SUKRAN SAYGI

17 NOVEMBER 2017

EALTA SIG

BREMEN UNIVERSITY

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

How can we grade ~ 5,500 test-takers’

essays fairly, reliably, and fast?


Background

English Proficiency Exam

September

January

June

August

5,500

2,000

5,500

1200


What is in the EPE?

Reading Listening

Writing Performance

1. Listen to the

instructor in the

classroom, take-

notes of the lecture

2. Study at

home: read

books, lecture

notes, etc.

3. Demonstrate ability

to produce a text

appropriate to the

task & time

constraints

Real-life

Performance


Performance Task

Listening & Note-taking

Reading Writing

End-product= 250-300 word essay


How to design a scale? (Weigle, 2002)

Type?

Holistic

Users?

Assessor oriented

(Alderson, 1991)

What aspects of writing are the

most important?

How many scoring levels?

Six

How to report the score?

Combined into a total score

How to validate the scale?

(Knoch, ..)


Intuitive

Qualitative

Quantitative (CoE, 2001)

Measurement driven

Performance driven (Fulcher et al, 2011)

What aspects of writing are the

most important?


Scale development

Exp

ert

intu

itio

n

Lit review

Previewing writing samples E

mp

iric

al

Qualitative

- discourse

Quantitative

- tallying

Em

pir

ical

Validation


Preparation

Workshop

Analysis

1st Draft

Revision

2nd Draft

Validation – Dec 2017

Progression of the study


Pre-Workshop

Pool of descriptors

Pool of writing samples


Workshop

• 15 Judges

• 15 Sample writings

• ~100 Descriptors

• A facilitator


Day 1

1) Familiarization with the task

a) Conceptual background

b) Operationalization of the writing construct -long (!) discussion on the construct

c) Doing the task

d) Discussion based on their performance experience


Day 1

2) Three piles

3) Discussion on the results (recorded)

(more discussion on the construct!)

Top

Middle Bottom


Day 2

3) Essays re-read and ordered within the piles

4) Discussion on the results (recorded)

5) After discussions judges revised order & decided passing / failing papers

6) Received written feedback


After-Workshop

Recordings analyzed separately

- content

(descriptions, adjectives, categories)

- tally of descriptors used for each paper

Salient features noted for each band


Analysis of recordings

• Ordering of the papers in each pile.


Salient Features Task fulfillment Range of grammar Range of vocabulary 70% Signposting Paragraphing Coherence Linguistic accuracy Rhetorical organization Fluency Flow of ideas Communicativeness Connectedness


1st Draft of the Scale

30 more papers are

- graded individually according to the draft scale by

the researchers (2)

the testing committee (4)

the school administrators (3)


2nd Draft of the Scale

Remaining ~ 90 more papers were graded individually by two experienced instructors and one researcher

Fine-tuning


Issues

- Sample number

- Not real performances

- Papers representing different levels of proficiency – not systematic

- Hard for the judges to change their mindset (L, C, O)


Questions

- What do each judge understand from the distributed descriptor list?

- How do they decide which descriptor is more important for successful task completion?


Next Validation

- 2000 examinees in December 2017

- Validation

- Quantitative (reliability)

- Qualitative – judges’ cognitive processes


Thank you. [email protected]

a multi-method approach to writing...a multi-method approach to writing scale development how can we...

Documents