![Page 1: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/1.jpg)
![Page 2: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/2.jpg)
A MULTI-METHOD APPROACH TO WRITING
SCALE DEVELOPMENT
ZEYNEP AKSIT
SUKRAN SAYGI
17 NOVEMBER 2017
EALTA SIG
BREMEN UNIVERSITY
![Page 3: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/3.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
How can we grade ~ 5,500 test-takers’
essays fairly, reliably, and fast?
![Page 4: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/4.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Background
English Proficiency Exam
September
January
June
August
5,500
2,000
5,500
1200
![Page 5: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/5.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
What is in the EPE?
Reading Listening
Writing Performance
![Page 6: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/6.jpg)
1. Listen to the
instructor in the
classroom, take-
notes of the lecture
2. Study at
home: read
books, lecture
notes, etc.
3. Demonstrate ability
to produce a text
appropriate to the
task & time
constraints
Real-life
Performance
![Page 7: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/7.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Performance Task
Listening & Note-taking
Reading Writing
End-product= 250-300 word essay
![Page 8: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/8.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
How to design a scale? (Weigle, 2002)
Type?
Holistic
Users?
Assessor oriented
(Alderson, 1991)
What aspects of writing are the
most important?
How many scoring levels?
Six
How to report the score?
Combined into a total score
How to validate the scale?
(Knoch, ..)
![Page 9: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/9.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Intuitive
Qualitative
Quantitative (CoE, 2001)
Measurement driven
Performance driven (Fulcher et al, 2011)
What aspects of writing are the
most important?
![Page 10: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/10.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Scale development
Exp
ert
intu
itio
n
Lit review
Previewing writing samples E
mp
iric
al
Qualitative
- discourse
Quantitative
- tallying
Em
pir
ical
Validation
![Page 11: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/11.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Preparation
Workshop
Analysis
1st Draft
Revision
2nd Draft
Validation – Dec 2017
Progression of the study
![Page 12: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/12.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Pre-Workshop
Pool of descriptors
Pool of writing samples
![Page 13: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/13.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Workshop
• 15 Judges
• 15 Sample writings
• ~100 Descriptors
• A facilitator
![Page 14: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/14.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
![Page 15: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/15.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Day 1
1) Familiarization with the task
a) Conceptual background
b) Operationalization of the writing construct -long (!) discussion on the construct
c) Doing the task
d) Discussion based on their performance experience
![Page 16: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/16.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Day 1
2) Three piles
3) Discussion on the results (recorded)
(more discussion on the construct!)
Top
Middle Bottom
![Page 17: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/17.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Day 2
3) Essays re-read and ordered within the piles
4) Discussion on the results (recorded)
5) After discussions judges revised order & decided passing / failing papers
6) Received written feedback
![Page 18: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/18.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
After-Workshop
Recordings analyzed separately
- content
(descriptions, adjectives, categories)
- tally of descriptors used for each paper
Salient features noted for each band
![Page 19: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/19.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Analysis of recordings
• Ordering of the papers in each pile.
![Page 20: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/20.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Salient Features Task fulfillment Range of grammar Range of vocabulary 70% Signposting Paragraphing Coherence Linguistic accuracy Rhetorical organization Fluency Flow of ideas Communicativeness Connectedness
![Page 21: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/21.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
1st Draft of the Scale
30 more papers are
- graded individually according to the draft scale by
the researchers (2)
the testing committee (4)
the school administrators (3)
![Page 22: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/22.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
2nd Draft of the Scale
Remaining ~ 90 more papers were graded individually by two experienced instructors and one researcher
Fine-tuning
![Page 23: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/23.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
![Page 24: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/24.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Issues
- Sample number
- Not real performances
- Papers representing different levels of proficiency – not systematic
- Hard for the judges to change their mindset (L, C, O)
![Page 25: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/25.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Questions
- What do each judge understand from the distributed descriptor list?
- How do they decide which descriptor is more important for successful task completion?
![Page 26: A MULTI-METHOD APPROACH TO WRITING...A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT How can we grade ~ 5,500 test-takers’ essays fairly, reliably, and fast?](https://reader030.vdocument.in/reader030/viewer/2022040603/5e9b78bfd73640012827d82b/html5/thumbnails/26.jpg)
A MULTI-METHOD APPROACH TO WRITING SCALE DEVELOPMENT
Next Validation
- 2000 examinees in December 2017
- Validation
- Quantitative (reliability)
- Qualitative – judges’ cognitive processes