using computerized assessment to identify profiles of … · 2016. 6. 23. · ll=log likelihood,...

Barbara Foorman and Yaacov Petscher - Florida Center for Reading Research at

Florida State University Liz Brooke and Alison Mitchell - Lexia Learning

USING COMPUTERIZED ASSESSMENT TO IDENTIFY PROFILES OF READING & LANGUAGE SKILLS IN ELEMENTARY AND SECONDARY STUDENTS

¡ The Structure of Reading

¡ Identifying Profiles

¡ Connections to Instruction

AGENDA

How does reading relate to language? How do we reconcile the structure of reading with profiles of strengths and weaknesses?

THE STRUCTURE OF READING

FLORIDA CENTER FOR READING RESEARCH (FCRR) READING ASSESSMENT (FRA)

As part of federal grants, FCRR developed the FRA •  2010-2014: Developed computer-adaptive K-12

component skills battery (item tryouts, IRT analyses, & linking studies. Called the FCRR Reading Assessment (FRA)

•  2010-2015: Conducted cross-sectional and longitudinal

study of reading & language development.

STRUCTURE OF READING K - 3

Foorman et al. (2015) Reading & Writing

Decoding fluency

Syntax

Phonological awareness

Vocabulary

Listening Comp

Oral language

Reading Comprehension

STRUCTURE OF READING 4 - 10

Foorman, et al. (2015) Journal of Educational Psychology

Decoding fluency

Syntax

Vocabulary

Oral language

Reading Comprehension

72% - 99% variance

BACKGROUND ON PROFILES

•  Targeting instruction to profiles of students’ strengths & weaknesses is central to teaching

•  Profiles often based on descriptive data of reading errors, text reading levels, or learning profiles.

•  Regression-based techniques used to quantify profiles of good & poor readers and profiles within poor readers.

•  Regression-based approaches use arbitrary achievement cut points (e.g., below 40th percentile on standardized test).

•  A latent class approach (LCA) utilizes multiple measures to reduce measurement error and improve reliability and stability of classification.

•  When the latent variable is continuous, the approach is often called latent profile analysis (LPA). LPA used here.

¡ What are the latent profiles of reading and language skills in a large, representative sample of Florida students in grades K-10?

¡ What are the relations among the latent profiles and a norm-referenced reading test in K-2 and a latent variable of reading comprehension in grades 3-10?

RESEARCH QUESTIONS

METHOD

•  Participants: 7,752 students in K-10; 2295 in K-2 and 5,457 in 3-10. Representative of Florida demographics.

•  Procedure: - K-2 FRA individually administered in two 45-min sessions in mid-year; 3-10 FRA administered in computer lab in two 45-min sessions in mid-year. - SESAT Word Reading administered in small groups in K; teachers administered SAT-10 and FCAT as usual.

•  Design and analysis: FRA raw scores converted to Z scores. Latent profile and general linear modeling were conducted at each grade (with linear step-up correction to correct against false discovery rate).

Construct Task/Abbreviation Grade Phonological Awareness Phonological awareness (PA) K

Alphabetics

LS Letter Sounds (AP1/2; LS) K

Decoding Word Reading (WR) G1 + G2

Encoding Spelling (Spell) G2

Oral Language

Vocabulary Vocabulary Pairs (VOC) K-2

Syntax Sentence Comprehension (SC) K

Listening Comprehension Following Directions (FD) K-2

CONSTRUCTS/FRA SCREENING TASKS IN GRADES K-2

Construct Tasks/Abbreviation Word recognition Word Recognition (WRT)

Academic Language

Vocabulary (morphological awareness) Vocabulary Knowledge (VKT)

Discourse (verb tense, anaphora, connectives) Syntactic Knowledge (SKT)

Reading Comprehension Reading Comprehension (RCT)

CONSTRUCTS/FRA SCREENING TASKS IN GRADES 3-10

Grade(s) Test/Subtest Kindergarten Stanford Early Scholastic Achievement Test

(SESAT) Word Reading 1-10 Stanford Achievement Test (10th ed; SAT-10)

Reading comprehension 3-10 Florida Comprehensive Assessment Test

(FCAT 2.0) Reading

STANDARDIZED READING OUTCOMES IN GRADES K-10

Note. A latent factor score for Reading Comprehension was created from the developmental scale scores from the FRA’s RCT, SAT-10, and FCAT 2.0

¡  Print-related measures were moderately correlated in K (PA with LS, .48, and with SESAT WR, .58; and SESAT WR with LS, .51). OL measures were moderately correlated with each other.

¡  In G1-G2, print-related measures were more strongly correlated: SAT-10 with WR (.75 in first and .58 in G2); WR & Spell in G2 (.77). Oral language measures were moderately correlated with each other in all three grades and VOC was moderately correlated with SAT-10 in G1 (.58) & G2 (.62).

¡  The three RC measures were strongly correlated, with the RCT bivariate correlations ranging from .67 in G8 to .81 in G5. FCAT and SAT-10 correlations ranged from a low of .71 in G8 to a high of .81 in G3 & G5. VKT and SKT were moderately correlated in these grades (.31 to .46) as were the bivariate correlations of WRT (.29 to .51).

CORRELATIONS

What are the challenges associated with identifying profiles of student strengths and weaknesses? What are ways to do so that are meaningful, but remain reliable and valid and can be assessed efficiently?

IDENTIFYING PROFILES

LATENT PROFILE ANALYSIS – WHAT?

Hybrid Models

Continuous Measures

Categorical Measures

Continuous Latent Factor Analysis Item Response Theory

Categorical Latent Latent Profile Analysis

Latent Class Analysis

LATENT PROFILE ANALYSIS – WHAT?

Vocabulary

PPVT EVT SYN

Class

PPVT EVT SYN

LATENT PROFILE ANALYSIS – WHY?

WHY ADAPTIVE MEASURES?

3-6 hours!!!

Note. LL=log likelihood, AIC=Akaike Information Criteria, aBIC=sample adjusted Bayes Information Criteria, -2LL=log likelihood ratio test. Values in bold represent Final selected class. *p<.001. 19

Latent profile model fit for kindergarten through grade 5 and grade 8

Grade Profiles Parameters LL AIC aBIC -2LL K 2 19 -3255.01 6548.01 6624.87

3 22 -2681.62 5407.25 5496.24 1146.77*

4 28 -2653.41 5362.82 5476.08 56.42*

5 34 -2629.75 5357.51 5465.04 47.32*

6 40 -2618.44 5316.88 5458.68 22.63*

1 2 10 -2818.26 5656.52 5705.49

3 14 -2785.14 5598.28 5666.84 66.24*

4 18 -2768.46 5572.92 5661.01 33.36*

5 22 -2752.99 5549.98 5657.71 30.94*

6 26 -2743.43 5546.86 5674.17 19.12*

2 2 13 -3768.29 7562.59 7624.79

3 18 -3697.13 7430.26 7516.38 142.33*

4 23 -3669.02 7384.03 7494.08 56.22*

5 28 -3655.54 7367.07 7501.04 26.96*

6 33 -3642.95 7355.89 7513.78 25.17*

3 2 10 -2202.90 4425.81 4438.14

3 14 -2173.78 4375.56 4392.83 58.24*

4 18 -2154.42 4344.83 4367.04 38.73*

5 22 -2129.87 4303.74 4330.88 49.10*

6 26 -2104.72 4261.43 4293.51 50.30*

4 2 10 -2166.75 4353.49 4365.49

3 14 -2140.35 4308.69 4325.50 52.80*

4 18 -2112.79 4261.58 4283.18 55.12*

5 22 -2097.77 4239.54 4265.95 30.04*

6 26 -2087.22 4226.44 4257.65 21.10*

5 2 10 -2451.39 4922.79 4935.95

3 14 -2405.92 4839.83 4858.25 90.96*

4 18 -2383.61 4803.22 4826.91 44.61* 5 22 -2363.13 4770.25 4799.20 40.97*

Note. VOC=Vocabulary Pairs; FD=Following Directions; PA=Phonological Awareness; LS=Letter Sounds; SC=Sentence Comprehension

20

Latent Profile Analysis of FRA Measures in Kindergarten (N=422)

-3

-2

-1

0

1

2 VOC FD PA LS SC

Kind

erga

rten

Z-S

core

c1 c2 c3 c4 c5 c6

c6; 19% c3; 42% c4; 23%

c5; 7%

c1; 7%

c2; 2%

350

400

450

500

550

SESA

T

1 2 3 4 5 6

group

<.0001Prob > F37.11F

Distribution of SESAT

350

400

450

500

550

SESA

T

1 2 3 4 5 6

group

<.0001Prob > F37.11F

Distribution of SESAT

Note. The average absolute value of the standardized difference in SESAT WR performance across all classes was Hedge’s g = 1.10, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

K SESAT WR BY LATENT CLASSES

22

Latent Profile Analysis of FRA Measures in Grade 1 (N=989)

Note. VOC=Vocabulary Pairs; FD=Following Directions; WR=Word Reading

-4

-3

-2

-1

0

1

2 VOC FD WR

Gra

de 1

Z-S

core

c1 c2 c3 c4 c5

c2; 35% c5; 43%

c1; 1% c2; 35%

c3; 3%

Note. The average absolute value of the standardized difference in SAT-10 RC performance across all classes was Hedge’s g = 1.43, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G1 SAT-10 RC BY LATENT CLASSES

24

Latent Profile Analysis of FRA Measures in Grade 2 (N=884)

Note. VOC=Vocabulary Pairs; FD=Following Directions; Spell=Spelling; WR=Word Reading

-2

-1

0

1

2 VOC FD Spell WR

Gra

de 2

Z-S

core

s

c1 c2 c3 c4 c5 c6

c5; 32%

c2; 10%

c4; 32%

c3; 15% c6; 5% c1; 52%

Note. The average absolute value of the standardized difference in SAT-10 RC performance across all classes was Hedge’s g = 1.48, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G2 SAT-10 RC BY LATENT CLASSES

Latent Profile Analysis of FRA Measures in Grade 5 (N=659)

Note. VKT=Vocabulary Knowledge Task; WRT=Word Recognition Task; SKT=Syntactic Knowledge Task

-4

-3

-2

-1

0

1

2 VKT WRT SKT

Gra

de 5

Z-S

core

c1 c2 c3 c4 c5

c5; 7 %

c4; 34 % c3; 53 %

c2; 4 %

c1; 1 %

-3

-2

-1

0

1

2

3

RC fa

ctor it

self

1 2 3 4 5

c

<.0001Prob > F132.73F

Distribution of frc

-3

-2

-1

0

1

2

3

RC fa

ctor it

self

1 2 3 4 5

c

<.0001Prob > F132.73F

Distribution of frc

Note. The average absolute value of the standardized difference in latent RC performance across all classes was Hedge’s g =2.53, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G5 RC FACTOR BY LATENT CLASSES

Latent Profile Analysis of FRA Measures in Grade 8 (N=629)

Note. VKT=Vocabulary Knowledge Task; WRT=Word Recognition Task; SKT=Syntactic Knowledge Task

-3

-2

-1

0

1

2 VKT WRT SKT

Gra

de 8

Z-S

core

c1 c2 c3

c2; 25%

c1; 72%

c3; 3%

-2

0

2

RC fa

ctor it

self

1 2 3

c

<.0001Prob > F196.02F

Distribution of frc

-2

0

2

RC fa

ctor it

self

1 2 3

c

<.0001Prob > F196.02F

Distribution of frc

Note. The average absolute value of the standardized difference in latent RC performance across all classes was Hedge’s g =2.19, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G8 RC FACTOR BY LATENT CLASSES

¡  LPA identified 5-6 classes in K-5 and only 3 in secondary grades.

¡  Latent profiles significantly related to standardized reading outcomes, accounting for 24% (in G3) to 61% (G9) of the variance, with the mode being 42%.

¡  Range of Hedges g (for average absolute values of the standardized difference in reading outcome across all latent classes) was 1.10 (in K) to 2.53 (in G5).

¡  Profiles above G5 fell into a pattern of low, medium, and high. ¡  The 5-6 reading and language profiles found in the elementary

grades reflect heterogeneity of skills.

SUMMARY OF RESULTS

¡  Fact that latent profiles accounted for substantial dif ferences in RC in a large diverse sample of students spanning 11 grades contributes to a field dominated by (a) unreliable or unstable classifications, and (b) LCA of clinical samples (e.g., Catts et al., 2012; Justice et al., 2015) or low-performing students (Logan & Petscher, 2010; Brasseur-Hock et al., 2011).

¡  Heterogeneity of skill profiles in the elementary grades in contrast to the low, medium, & high profiles in the secondary grades suggests focusing on differentiating instruction in K-5.

¡  Importance of FRA academic language (vocab and syntax) to RC, more than word recognition, apparent in G3 and above.

CONCLUSIONS

¡  Cross-sectional rather than longitudinal. [Replications of findings across grades helps.]

¡  Profiles and their relations to reading outcomes are limited to the measures used. [At least a latent variable of reading comprehension was used, which related strongly to the single measures in the FRA reading and language measures.]

¡  Next step is to test results of the heterogeneous profiles from this exploratory LPA with confirmatory latent class analysis.

LIMITATIONS/NEXT STEPS

Immediate and Actionable Data

CONNECTION TO INSTRUCTION

How do we make these types of results meaningful for educators?

HOW DO EDUCATORS MAKE SENSE OF ALL OF THIS DATA??

ONLY 16% OF EDUCATORS FEEL PREPARED

National Center for Literacy Education (2014)

16%

QUESTIONS TO DRIVE INSTRUCTION &

DECISIONS

¡ As a whole school or district, what percent of my students are at-risk for reading failure? What percent are on track for reading success?

¡ Have we seen growth in my students across the year? If so, how much growth?

¡ What subgroups of students are on track for reading success and which subgroups are at-risk for reading failure?

¡ Do I see any patterns of strengths and weaknesses across grades in my district/school? § This will help address PD and Curriculum needs

QUESTIONS ADMINISTRATORS NEED ANSWERED

¡ How many of my students are § Above grade level, § On grade level or § Below grade level?

¡ If they are above grade level, how far above? If they are below, how far below?

¡ Most importantly, how do I help those who are below, get to where they need to be?

QUESTIONS TEACHERS NEED ANSWERED

¡ What percent of my students are on track for reading success?

¡ What patterns of strengths and weaknesses do I see in my class? How can I adjust my instruction to meet the needs?

¡ What do these patterns mean for my small group instruction? Are there certain students who need the same type of instruction? If so, who are they and what is the instruction they need?

QUESTIONS TEACHERS NEED ANSWERED

“The goal (of assessment) is to gain enough information about student progress to make

effective decisions while minimizing the time spent administering assessments.”

- Torgesen, 2006, p.3

REMEMBER…

¡ Include oral language measures in screening, diagnostic, and progress monitoring of reading to accurately predict reading comprehension.

¡ Describe profiles of strengths and weaknesses in a valid, reliable, and efficient manner.

¡ Use profiles to differentiate instruction.

CONCLUDING THOUGHTS

bfoorman@fcrr.org ypetscher@fcrr.org

lbrooke@lexialearning.com amitchell@lexialearning.com

THANK YOU!