Pre-text revisited: Do we understand how writing prompts function?

Mark Chapman

1. The testing context2. Prompt effect literature3. Prompt categorization4. The dataset - essays and discourse variables5. Analyses of essays by prompt6. Significant differences in essays by prompt7. Implications for writing test design

Contents

• Writing test• Select 1 prompt from 2• 30 minutes to compose a response• Scored independently by 2 raters

• Effect of prompt wording on score awarded– Leu, Keech, Murphy & Kinzer, 1982– Brossell & Ash, 1984

• Effect of prompt wording on discourse variables– Greenberg, 1981 – Hirokawa & Swales, 1986

• Effect of different integrated prompts on discourse variables– Cumming et al., 2005 – O’Loughlin & Wigglesworth, 2007

Prompt effect in writing assessment

Problems with previous studies

• Used holistic score as measure of prompt effect

• Compared prompts that were very similar• Had small sample sizes• Analyzed stimuli differences not prompt • Little prompt categorization

1. What are the distinguishing characteristics of independent writing prompts?

2. How do these characteristics affect the test-takers’ final written products?

Research Questions

• Applied previous prompt taxonomies• Swales, 1982; Horowitz, 1991; Lim, 2009

Writing prompt characteristics

Prompt characteristic Examples

Domain Educational, occupational, personal, public

Response mode Argumentative or narrative

# of rhetorical cues 1 - 6

Open / focused Extent of contextualization or explicit guidance

The dataset

Prompts Responses per prompt

Responses by proficiency level

Discourse variables

6 60 20 15

Analyses

1. Factor structure of discourse variables (DVs)

2. Minimize number of MANOVA analyses to reduce risk of Type I error

3. MANOVA analyses of DV differences in responses elicited by each prompt

Factor Analysis

ComponentInitial Eigenvalues

Rotation Sums of Squared Loadingsa

Total % of Variance Cumulative % Total1 3.439 22.924 22.924 2.9122 2.027 13.510 36.435 2.3833 1.837 12.247 48.682 1.9644 1.486 9.907 58.588 2.0605 1.449 9.663 68.252 1.6856 1.070 7.134 75.386 1.2397 .841 5.606 80.992 1.3758 .744 4.962 85.953 9 .662 4.416 90.370 10 .491 3.276 93.645 11 .378 2.518 96.163 12 .302 2.011 98.175 13 .230 1.533 99.708 14 .037 .244 99.952 15 .007 .048 100.000

Extraction Method: Principal Component Analysis.

Factor structurePattern Matrixa

Component1 2 3 4 5 6 7

FREQ 1 .996 FREQ 3 -.805 FREQ 2 -.783 TTR -.358 CONCAUSi .847 CONLOGi .841 CONi .782 .414ERR/100 .978 TOTERR .932 AWL .919 FREQ AC .914 SYNLE .863 ASL .863 word # .991 CONTEMPi .978

MANOVA – lexical sophisticationMultivariate Testsa

Effect Value FHypothesis

df Error df Sig.Prompt Pillai's Trace .269 5.109 20.000 1416.00

0 .000

Wilks' Lambda .751 5.260 20.000 1165.085 .000

Hotelling's Trace .306 5.350 20.000 1398.000 .000

Roy's Largest Root .194 13.760c 5.000 354.000 .000

Tests of Between-Subjects Effects

SourceType III Sum of

Squares dfMean

Square F Sig.Prompt TTR .073 5 .015 3.262 .007

FREQ 1 1132.758 5 226.552 13.140 .000

FREQ 2 310.514 5 62.103 8.373 .000

FREQ 3 359.922 5 71.984 10.995 .000

MANOVA - cohesionMultivariate Testsa

Effect Value FHypothesis

df Error df Sig.

Prompt Pillai's Trace .232 5.946 15.000 1062.000 .000

Wilks' Lambda .776 6.222 15.000 972.118 .000

Hotelling's Trace .277 6.469 15.000 1052.000 .000

Roy's Largest Root .230 16.286c 5.000 354.000 .000

Tests of Between-Subjects Effects

SourceType III Sum of Squares df

Mean Square F Sig.

Prompt CONi 5956.622 5 1191.324 2.878 .015

CONCAUSi 3096.003 5 619.201 3.191 .008

CONLOGi17363.248 5 3472.650 12.865 .000

MANOVA - accuracyMultivariate Testsa

Effect Value FHypothesis

df Error df Sig.

Prompt Pillai's Trace .057 2.095 10.000 708.000 .023

Wilks' Lambda .943 2.103b 10.000 706.000 .022

Hotelling's Trace .060 2.111 10.000 704.000 .022

Roy's Largest Root .050 3.546c 5.000 354.000 .004

Tests of Between-Subjects Effects

SourceType III Sum of

Squares dfMean

Square F Sig.

Prompt TOTERR 1627.081 5 325.416 1.610 .157

ERR/100 67.973 5 13.595 .700 .624

MANOVA – academic vocabularyMultivariate Testsa

Effect Value FHypothesis

df Error df Sig.

Prompt Pillai's Trace .614 31.368 10.000 708.000 .000

Wilks' Lambda .472 32.111b 10.000 706.000 .000

Hotelling's Trace .933 32.856 10.000 704.000 .000

Roy's Largest Root .653 46.232c 5.000 354.000 .000

Tests of Between-Subjects Effects

SourceType III Sum of Squares df

Mean Square F Sig.

Prompt AWL 13.445 5 2.689 46.212 .000

FREQ AC 1129.167 5 225.833 31.165 .000

MANOVA – syntactic complexityMultivariate Testsa

Effect Value FHypothesis

df Error df Sig.

Prompt Pillai's Trace .194 7.611 10.000 708.000 .000

Wilks' Lambda .812 7.746b 10.000 706.000 .000

Hotelling's Trace .224 7.880 10.000 704.000 .000

Roy's Largest Root .182 12.896c 5.000 354.000 .000

Tests of Between-Subjects Effects

SourceType III Sum of Squares df

Mean Square F Sig.

Prompt SYNLE 115.864 5 23.173 10.222 .000

ASL 1066.851 5 213.370 10.200 .000

ANOVAs – fluency; writing score; temporal cohesive markers

ANOVA  Sum of Squares df Mean Square F Sig.

word # Between Groups 125644.025 5 25128.805 3.399 .005

Within Groups 2616995.350 354 7392.642    

Total 2742639.375 359      

Writing score Between Groups 235.356 5 47.071 1.149 .334

Within Groups 14507.300 354 40.981    

Total 14742.656 359      

CONTEMPi Between Groups 4157.306 5 831.461 9.115 .000

Within Groups 32292.152 354 91.221    

Total 36449.458 359      

• Holistic score awarded

• Accuracy

Little evidence of significant differences in . . .

• Lexical sophistication• Cohesion• Academic language use• Syntactic complexity• Fluency

Significant differences in . . .

Prompts responsible for most differences

Prompt Domain Response mode Focus # of cues

4 Occupational Argumentative Focused 6

5 Personal Narrative Focused 6

6 Personal Narrative Open 2

• Score interpretation• Construct definition

• Consider which domains are appropriate• Consider permissible response modes• Consider number of rhetorical cues

Implications for writing assessment

Thank you

For more information, please contactchapman.m@cambridgemichigan.org