a multi-method approach to writing...a multi-method approach to writing scale development how can we...

A MULTI-METHOD APPROACH TO WRITING

SCALE DEVELOPMENT

ZEYNEP AKSIT

SUKRAN SAYGI

17 NOVEMBER 2017

EALTA SIG

BREMEN UNIVERSITY

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

How can we grade ~ 5,500 test-takers’

essays fairly, reliably, and fast?

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Background

English Proficiency Exam

September

January

June

August

5,500

2,000

5,500

1200

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

What is in the EPE?

Reading Listening

Writing Performance

1. Listen to the

instructor in the

classroom, take-

notes of the lecture

2. Study at

home: read

books, lecture

notes, etc.

3. Demonstrate ability

to produce a text

appropriate to the

task & time

constraints

Real-life

Performance

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Performance Task

Listening & Note-taking

Reading Writing

End-product= 250-300 word essay

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

How to design a scale? (Weigle, 2002)

Type?

Holistic

Users?

Assessor oriented

(Alderson, 1991)

What aspects of writing are the

most important?

How many scoring levels?

Six

How to report the score?

Combined into a total score

How to validate the scale?

(Knoch, ..)

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Intuitive

Qualitative

Quantitative (CoE, 2001)

Measurement driven

Performance driven (Fulcher et al, 2011)

What aspects of writing are the

most important?

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Scale development

Exp

ert

intu

itio

n

Lit review

Previewing writing samples E

mp

iric

al

Qualitative

- discourse

Quantitative

- tallying

Em

pir

ical

Validation

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Preparation

Workshop

Analysis

1st Draft

Revision

2nd Draft

Validation – Dec 2017

Progression of the study

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Pre-Workshop

Pool of descriptors

Pool of writing samples

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Workshop

• 15 Judges

• 15 Sample writings

• ~100 Descriptors

• A facilitator

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Day 1

1) Familiarization with the task

a) Conceptual background

b) Operationalization of the writing construct -long (!) discussion on the construct

c) Doing the task

d) Discussion based on their performance experience

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Day 1

2) Three piles

3) Discussion on the results (recorded)

(more discussion on the construct!)

Top

Middle Bottom

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Day 2

3) Essays re-read and ordered within the piles

4) Discussion on the results (recorded)

5) After discussions judges revised order & decided passing / failing papers

6) Received written feedback

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

After-Workshop

Recordings analyzed separately

- content

(descriptions, adjectives, categories)

- tally of descriptors used for each paper

Salient features noted for each band

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Analysis of recordings

• Ordering of the papers in each pile.

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Salient Features Task fulfillment Range of grammar Range of vocabulary 70% Signposting Paragraphing Coherence Linguistic accuracy Rhetorical organization Fluency Flow of ideas Communicativeness Connectedness

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

1st Draft of the Scale

30 more papers are

- graded individually according to the draft scale by

the researchers (2)

the testing committee (4)

the school administrators (3)

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

2nd Draft of the Scale

Remaining ~ 90 more papers were graded individually by two experienced instructors and one researcher

Fine-tuning

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Issues

- Sample number

- Not real performances

- Papers representing different levels of proficiency – not systematic

- Hard for the judges to change their mindset (L, C, O)

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Questions

- What do each judge understand from the distributed descriptor list?

- How do they decide which descriptor is more important for successful task completion?

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Next Validation

- 2000 examinees in December 2017

- Validation

- Quantitative (reliability)

- Qualitative – judges’ cognitive processes

A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT

Thank you. zaksit@metu.edu.tr