lisa harris, university of south carolina comparison of student performance between teacher read and...

LISA HARRIS,UNIVERSITY OF SOUTH CAROLINA

Comparison of Student Performance Between Teacher Read and CD-ROM

Delivered Modes of Test Administration of English Language Arts Tests

Background

NCLB, IDEIA, Section 504 of Rehabilitation Act

Computer-based accommodations available on national tests AP exams SAT GRE GMAT, LSAT

Increased computer availability in schoolsIncreased use of computer accommodations on

state-wide accountability tests

Past Findings: Where do we go from here?

Read aloud administration Accommodation or

modification?

Other issues Cuing Pacing Computer skills Reading Level Item type

Past Findings

Read-aloud Accommodation Delivery Options “Live” reader with written script Cassette/CD with cassette/CD player CD played on computer with test booklet Video taped reader (head shot) Video taped reader, only text displayed Video tape with head shot and text Test on computer with computer generated voice Test on computer with human voice

Call for more research

Studies using math tests Possibly confounded by lack of content knowledge

(Crawford & Tindal, 2004; Bolt & Thurlow, 2006)

Few studies compare delivery methods Call for more research on read-aloud delivery via computer

(Miranda, 2004; Calhoon, 2000)

1999 Standards for Educational and Psychological Testing “Support should be provided for any assertion that scores

obtained using different items or testing materials, or different testing procedures, are interchangeable for some purposes” AERA, 1999, p.57

Research Questions

1. Does the English Language Arts (ELA) test have the same factorial structure between oral script and CD-ROM modes of test administration?

2. Controlling for prior ELA performance, are there differences in student performance between test administration modes?

a. Is there an interaction between test administration mode and student disability?

b. Does student performance vary between the oral script and CD-ROM modes of test administration?

3. How do the above results hold across grade levels?

Methods: Instruments

Palmetto Achievement Challenge Test (PACT) Part of the state accountability system Based on state academic content standards Grades 3-8 ELA, math, science and social studies CDs available in grades 5-8

Delivery Methods: CD-ROM

Follow along in test booklet

Record answers in test booklet

No passages or questions appear on screen

Human voice

CD Screen Shoots

OS Delivery

Teacher-reader follows a script.

Methods: Instruments

Depth of Knowledge Rubric for Selected-Response Items Development Description

DOK1: Verbatim recall and simple understanding

DOK2:Basic reasoning skills, simple extension beyond what is explicitly stated

DOK3:Complex reasoning DOK4:Extended reasoning, inference, planning.

Implementation

DOK Level Examples

DOK1: Questions focus on verbatim recall and simple understanding. Questions are related to parts of a text rather than the text as a whole.

According to the poem, how has the speaker already tried to solve the noise problem?

A. by telling the mother B. by asking Ray to whisper C. by giving Ray earplugsD. by shutting the bedroom door

DOK Level Examples

DOK2: Require basic reasoning skills, comprehension on the literal level, and simple extension beyond what is explicitly stated. Questions require some mental processing of the text or portions of the text.

What is the main idea of this passage?

A. Louis suffered from poor health. B. Louis could have been a great painter. C. Louis wrote books that many children enjoy. D. Louis used his talents throughout his life.

DOK Level Examples

DOK3: Questions require complex reasoning. Questions require an understanding of the text as a whole.

When the poet says “Like medals with their ribbons frayed and wavering” (lines 61–62), she is referring toA) victoryB) fishhooksC) trophiesD) fish scales

DOK Level Examples

DOK4: Questions require extended reasoning, inference or planning. Questions go beyond the literal text and require a deep, purposeful understanding of the text as a whole and/or and understanding of the text as a whole in relation to other texts.

Which word best describes the tone of “The Long Hill”? A. fearfulness B. encouragement C. wishfulness D. disappointment

Methods: Participants

Students in grades 6-8 who took ELA PACT in spring 2007 using the read-aloud administration

Oral Script CD-ROM

Total 4966 869

Grade 6 1681 259

Grade 7 1753 310

Grade 8 1532 300

Methods: Participants

Due to small sample size for non-LD students disability categories were collapsed into the following:

Category 1: students with learning disabilities

Category 2: all other disabilities including: speech/ language impaired, deaf or hard of hearing, blind and visually impaired, orthopedically impaired, autistic, emotionally disabled, educable mentally disabled, other health impaired, traumatic brain injury, and multiple-disabled due to small sample size

Demographic Characteristics of the Students Grade 6

Demographic

OSn = 1681

CD-ROMn = 259

Characteristic n % n %Gender

Female 511 30.0 91 35.0Male 1170 70.0 168 65.0

EthnicityAfrican American 890 53.0 115 44.0Hispanic 44 2.6 14 5.4White 719 43.0 123 47.5

Eligible for free or reduced lunch 1391 82.7 192 74.1

IEP LabelsSpeech/Language 270 13.4 32 10.8Learning Disabled 1180 58.8 194 65.8Educable Mentally Disabled 289 14.4 33 11.2Other Health Impaired 125 6.2 25 8.5

Disability Category1 (Learning Disability) 1034 61.5 169 65.32 (All others) 647 38.5 90 34.7


Demographic

OSn = 1753

CD-ROMn = 310


Female 543 30.0 86 28.0Male 1210 69.0 224 72.0


Eligible for free or reduced lunch 1444 82.5 242 78.1




Demographic

OSn = 1532

CD-ROMn = 300


Female 511 33.0 98 33.0Male 1021 67.0 202 67.0


Eligible for free or reduced lunch

1225 80.0 224 75.0



Methods: Data Analysis Measurement Invariance

CFA/ SEM Step 1: one-factor model was established for each group individually Step 2: one-factor model was established for each group

simultaneously Step 3: three levels of invariance were tested using 2 :

congeneric (no equality constraints), tau-equivalent (set factor loadings equal), parallel (set equal factor loadings and error variances). DOK1 was selected as the reference variable and the path (1) from

DOK1 to the ELA factor was set to 1. Jöreskog, 1971

Goodness-of-Fit Indices and Criterion 2 = chi-squared, insignificant indicates measurement invariance CFI=comparative fit index ( .95 or above) SRMR=standardized root mean square residual (.08 or below) RMSEA= root mean square error of approximation (.06 or below) Hu & Bentler, 1999

Methods: Measurement Invariance

One-Factor Model for Each Administration Mode

Note. = independent variable error variance; = factor loadings; =latent disturbance variance

1

3

4

2

5

6

1

2

3

4

5

6

x1=DOK1

x2=DOK2

x3=DOK3

ELA - CDx4=DOK4

x6=ER

x5=CR

x1=DOK1

x2=DOK2

x3=DOK3

x6=ER

1

3

4

2

5

6

1

2

3

4

5

6

ELA - OSx4=DOK4

x5=CR

Methods: Data Analysis

Differences in Student Performance MANCOVA with prior ELA ability as a covariate Prior ELA ability = total score on 2006 ELA PACT

Dependent variables DOK1, DOK2, DOK3, DOK4, CR, ER

Xijk = μ0 + (Effect of COVAR)+ (Main Effect of MODE) + (Main Effect of TYPE) + (Interaction Effect MODE*TYPE) + Residual

All analyses conducted separately for each grade because the tests are not on the same scale.

Fit of One-factor Model Grade 6

Mode df 2 p-value CFI SRMR RMSEAOS 9 34.41 < .01 .99 .019 .041CD-ROM 9 14.36 0.11 .99 .029 .047

Factor Loadings Error VariancesOS CD-ROM OS CD-ROM

DOK1* 1.00 (.77) 1.00 (.84) 2.75 (.41) 2.22 (.30)DOK2 .85 (.71) .74 (.71) 2.89 (.50) 2.85 (.50)DOK3 .72 (.69) .51 (.61) 2.24 (.52) 2.32 (.63)DOK4 .53 (.59) .52 (.67) 2.08 (.65) 1.71 (.55)CR .33 (.60) .27 (.58) .78 (.64) .74 (.66)ER 1.50 (.58) 1.48 (.59) 17.38 (.66) 22.07 (.66)

Goodness-of-fit Indices of the One Factor Model for Grade 6

Unstandardized (Standardized) Parameter Estimates of the One-Factor Model Grade 6


Mode df 2 p-value CFI SRMR RMSEAOS 9 32.60 < .01 .99 .018 .039CD-ROM 9 12.75 .17 1.00 .026 .037


DOK1* 1.00 (.68) 1.00 (.59) 2.48 (.53) 2.79(.66)DOK2 .79 (.62) .94 (.60) 2.23 (.62) 2.25 (.64)DOK3 .70 (.57) .92 (.60) 2.17 (.67) 2.18 (.64)DOK4 .79 (.59) 1.23 (.69) 2.52 (.65) 2.38 (.52)CR .77 (.75) .96 (.75) 1.00 (.44) 1.05 (.44)ER 2.17 (.59) 3.51 (.70) 18.78 (.65) 19.03 (.51)




Mode df 2 p-value CFI SRMR RMSEAOS 9 18.94 .026 1.0 .014 .027CD-ROM 9 15.53 .07 .99 .031 .051


DOK1* 1.00 (.60) 1.00 (.62) 2.51 (.638) 2.82 (.64)DOK2 1.15 (.62) .89 (.59) 2.97 (.705) 2.65 (.61)DOK3 1.25 (.64) .97 (.59) 3.28 (.791) 3.10 (.60)DOK4 1.20 (.68) 1.05 (.69) 2.31 (.808) 2.13 (.53)CR .96 (.72) .77 (.64) 1.19 (.799) 1.53 (.48)ER 2.81 (.60) 2.74 (.64) 20.26 (.812) 18.65 (.64)



Summary of Measurement Invariance Findings

Grade 6Model 2 df 2 2 df p-value CFI

SRMR RMSEACongeneric 52.18 19 - - - .99 .06 .04Tau-equivalent 62.2 24 10.02 5 .07 .99 .07 .04Parallel 72.26 30 10.06 6 .12 .99 .09 .04

Grade 7

Model 2 df 2 2 df p-value CFI SRMR RMSEA

Congeneric 49.62* 19 - - - .99 .06 .04Tau-equivalent 69.27* 24 19.65* 5 .001 .99 .09 .04Parallel 71.48* 30 2.21 6 .89 .99 .1 .04

Grade 8Model 2 df 2 2 df p-value CFI

SRMR RMSEA

Congeneric 35.52* 19 - - - 1.0 .04 .03Tau-equivalent 42.45* 24 6.93 5 .22 1.0 .06 .03Parallel 53.17* 30 10.72* 6 .09 1.0 .07 .03

Note: * p<05

Summary of Means by Delivery Mode Grade 6 (question 2a)

Model Mean SD2006 ELA PACT OS 25.41 7.41

CD 26.59 7.66

DOK1 OS 6.70 2.60CD 6.63 2.73

DOK2 OS 5.73 2.41CD 5.56 2.39

DOK3 OS 5.07 2.08CD 5.09 1.92

DOK4 OS 3.87 1.79CD 3.96 1.77

CR OS .87 1.11CD .85 1.06

ER OS 14.62 5.14CD 14.83 5.80



CD 35.11 12.66

DOK1 OS 5.04 2.15CD 5.09 2.06

DOK2 OS 3.91 1.89CD 3.85 1.88

DOK3 OS 4.25 1.80CD 4.30 1.85

DOK4 OS 4.57 1.97CD 4.74 2.14

CR OS 1.78 1.51CD 1.74 1.55

ER OS 15.39 5.39CD 15.48 6.08



CD 33.88 11.03

DOK1 OS 4.53 1.98CD 4.58 2.14

DOK2 OS 5.50 2.20CD 5.44 2.01

DOK3 OS 5.41 2.35CD 5.45 2.18

DOK4 OS 4.73 2.08CD 4.67 2.01

CR OS 1.50 1.58CD 1.54 1.60

ER OS 16.20 5.61

CD 16.45 5.64

Summary of Means by Delivery Mode and Disability Grade 6 (question 2b)

Ranged from 0.01 to 2.9. These small differences are consistent with

the non-significant interaction effect.

Results: Differences in Student Performance

Grade 6 Grade 7 Grade 8

Main Effect (MODE)

Wilks’ lamda

.99 .99 .99

F 1.3 0.94 .25

df 6, 1930 6, 2053 6, 1822

p-value

.26 .46 .96

Interaction (MODE*TYPE)

Wilks’ lamda

.99 .99 .99

F .67 1.25 .22

df 6, 1930 6, 2053 6, 1822

p-value

.67 .28 .97

Students Scoring At Each Proficiency Level

Grade 6 Grade 7 Grade 8Frequency Percent Frequency Percent Frequency Percent

Below Basic 1620 83 1833 88.9 1651 90.1Basic 301 15 219 10.6 175 9.6Proficient 19 1 11 0.5 6 0.3Advanced - - - - - -

Limitations

Implementation of the CD-ROM formatOpportunity to practiceBroad disability categoriesAge and content areaAccommodation decision-makingOther accommodations besides read-aloud?

Recommendations for Future Study

Further investigating read-aloud accommodation on specific disabilities

Differences in students who received OS vs. CD-ROM

Looking at DOK differently Item difficulty Analyzing complexity of reading passage

Investigating how the CD-ROM was used Did students repeat questions and passages?

lisa harris, university of south carolina comparison of student performance between teacher read and...

Documents

test administration

test bookletno passages

test bookletrecord answers

computer miranda

english language arts

computer generated voicetest

simple extension

prior ela performance