http://miblsi.cenmi.org using cbm to predict meap reading test scores
TRANSCRIPT
http://miblsi.cenmi.org
Using CBM to Predict MEAP Reading Test Scores
2
Our Research Team
Anna Harms Patrick SorrelleCristy Coughlin
Ed HuthCheyne LeVesseurChristine Russell
3
1. Curriculum-based measures for reading provide us with tools to predict performance on high-stakes assessments
2. We must know how our screening tools and cut scores are designed to function
3. We must use a data-driven process for selecting screening tools and cut scores
3 Big Ideas to Take Away
4
Background
4
Purpose of MiBLSi
A statewide structure to create capacity for an integrated behavior and academic Multi-Tiered System of Support (MTSS) that can be implemented with fidelity, is sustainable over time and utilizes data-based decision making at all levels of implementation support
Provides guidance, visibility, funding, political support for MiBLSi
Students
Building Staff
Building Leadership Team
LEA District Cabinet and
Implementation Team
Across State
Multiple ISD/LEA Teams
All staff
All students
Multiple schools w/in local district
Provides guidance, visibility, funding, political support, and implementation supports
Provides coaching and TA for LEA and/or ISD Teams
Provides guidance and manages implementation
Provides effective practices to support students
Improved behavior and reading
ISD Cabinet and Implementation
Team
Regional Technical
Assistance
MichiganDepartment of
Education/MiBLSi Leadership
Multiple LEAs w/in intermediate district
Provides guidance, visibility, funding, political support, and implementation supports
MiBLSi Statewide Structure of Support
Who is supported?
How is support provided?
7
2014
New ISDs
Previous ISDs
Check-In Partnership ISDs
8
2014
Partnering Districts
9
• A need to accurately respond to requests for recommendations on how to select an appropriate set of benchmark goals and cut points for risk
• DIBELS Next, Introduction of the Composite Score
• AIMSweb Default Criteria• University of Oregon, Release of an Alternate
Set of Benchmark Goals and Cut Points for Risk
• A desire to “see for ourselves” how well different benchmark goals and cut points for risk function with the MEAP reading test
What Led to This Study
10
• A desire to become better consumers of research by learning how to conduct research
• We did NOT set out to create a new set of benchmark goals and cut points for risk
What Led to This Study
11
Provided initial information during an MTSS Coordinator webinar late Fall 2013
Further invited ISDs partnering with MiBLSi and held follow-up meetings with ISDs that expressed interest
Took those who volunteered and were able to successfully export, de-identify, and submit data to MiBLSi by early Winter 2014
Recruitment
12
Information on Study SampleNumber of ISDs, Districts and Students
Districts Included in the Study
Students Included in the Study
ISD 1 11 18,180
ISD 2 2 2,456
ISD 3 3 2,312
ISD 4 1 224
Total: 4 17 23,172
13
Information on Study SampleDemographics
RaceState Totals
Study District Totals
White 69.0% 79.0%
African American 18.3% 1.9%
Hispanic 6.4% 13.2%
Asian 2.9% 2.8%
Multi-Racial 2.5% 2.5%
American Indian 0.8% 0.5%
Native Hawaiian 0.1% 0.1%
Socioeconomic Status
State Totals
Study District Totals
Eligible Free and Reduced Lunch
48.1% 33.8%
14
Information on Study Sample2012 MEAP Reading Results
Grade Level
State-wide Average Meeting
Proficient or Above
Study DistrictsAverage Meeting
Proficient or Above
3rd Grade 66% 77%
4th Grade 68% 80%
5th Grade 70% 81%
6th Grade 68% 77%
7th Grade 62% 73%
8th Grade 66% 77%
15
Information on Study SampleSample Size Ranges by Grade
DIBELS NextFall, Winter and Spring
2011 - 2012Fall 2012
2nd Grade 1006 - 1624
3rd Grade 1342 - 1570 1006 - 2604
4th Grade 1483 - 1609 1342 - 2446
5th Grade 1451 - 1662 1483 - 2297
6th Grade 519 - 528 365 - 592
AIMSwebFall, Winter and Spring
2011 - 2012Fall 2012
6th Grade 413 - 417 441 - 456
7th Grade 687 - 708 806 - 808
8th Grade 732
16
This presentation will not provide detailed background information about:
• MTSS• Universal Screening• DIBELS Next• AIMSweb• MEAP
We are assuming that you have familiarity with these topics or have a way to deepen your knowledge outside of this presentation.
Assumptions
17
Reading CBM Correlation with High-Stakes Assessments
18
Assessments Used Within the Study
DIBELS NextFall 2011
Winter 2012
Spring 2012
Fall 2012
Composite 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th
DORF Words Correct 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th
DORF Accuracy 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th
Retell 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th
Daze 3rd-6th 3rd-6th 3rd-6th 3rd-6th
NWF WWR 2nd
NWF CLS 2nd
2012 MEAP Reading 3rd-6th
AIMSwebFall 2011
Winter 2012
Spring 2012
Fall 2012
6th - 7th 6th - 7th 6th - 7th 6th - 8th
R-CBM X X X X
MAZE X X X X
2012 MEAP Reading X
19
• MEAP is a standards-based test that measures how well Michigan students in 3rd through 9th grade are achieving
• Includes tests in reading, writing, math, science and social studies
• Reward, Focus and Priority schools are chosen based on MEAP results
• MEAP is the measuring stick for “Adequate Yearly Progress”
• Parents utilize MEAP results as a standard for school success
In Michigan MEAP is Important
20
2012 MEAP Reading Results
Grade Level
State-wide Average Meeting
Proficient or Above
3rd Grade 66%
4th Grade 68%
5th Grade 70%
6th Grade 68%
7th Grade 62%
8th Grade 66%
21
What if we didn’t have to wait and see how students do on the MEAP?
What if we could quickly
and accurately predict a student’s
future reading performance?
22
Good news, we have universal screening tools to help us!
Rea
ding
gra
de le
vel
Grade level corresponding to age 1 2 3 4
4
3
2
1
5
2.5
5.2
Early Intervention Changes Reading Outcomes
At Risk on Early Screening
Low Risk on Early Screening
3.2
Control
With research-based core but without extra instructional intervention
4.9
Interventio
n
With substantial instructional intervention
Torgesen, 2008
24
• Systematic assessment of all students within a school or district, on academic and/or social-emotional indicators for the purpose of identifying system-wide needs as well as intervention supports
• Provides information for problem solving
at multiple educational levels
• Identify those who are at-risk for poor
performance on other critical measures• Later reading assessments• High-stakes assessment / MEAP
Why Universal Screening?
Students
Building Staff
Building Leadership Team
LEA District Cabinet and
Implementation Team
25
CBM Correlations with
Fall 2012 MEAP Results
Universal Screening Concurrent
Fall 2012
MEAPFall 2012
Universal Screening Previous Spring 2012
Universal Screening Previous Winter 2012
Universal Screening Previous
Fall 2011
26
• A correlation tells us how two things relate
• Does not tell us what causes the outcomes
• When two variables correlate well, we can predict results on one based on the other
Why Would a Correlation Be Helpful?
27
When is a correlation meaningful?
Expected Correlations Coefficient
Very LargeMeasuring the same thing possibly in a different format
.70 - .99
LargeMeasuring the same construct but through different assessments
.5 - .69
ModerateMeasuring similar things .30 - .49
Small/TrivialMeasuring unrelated things .00 - .29
Hopkins, 2002
28
CBM Correlations with
Fall 2012 MEAP Results
Universal Screening Concurrent
Fall 2012
MEAPFall 2012
Universal Screening Previous Spring 2012
Universal Screening Previous Winter 2012
Universal Screening Previous
Fall 2011
ConcurrentAssessments
29
CBM to MEAP Correlations Concurrent Fall 2012
Third Grade Fourth Grade Fifth Grade Sixth Grade
Composite .65 .64 .65 .53
DORF WC .60 .59 .60 .49
DORF Accuracy
.51 .45 .45 .33
DAZE .53 .59 .61 .47
Large =
Moderate =
DIBELS Next
AIMSwebSixth Grade Seventh Grade Eighth Grade
R-CBM .54 .55 .55
MAZE .49 .48 .56
30
Higher Correlation = More Variance Explained
31
Value of the DIBELS Next Composite
DORF Score Predicting
MEAP
DIBELS Composite Score Predicting
MEAP
3rd Grade .60 .65
4th Grade .59 .64
5th Grade .60 .65
6th Grade .49 .53
Additional Variance Explained by
DIBELS Composite Score
7.36%
6.41%
6.50%
3.56%
32
DIBELS Next Correlations with
4th Grade Fall 2012 MEAP Results
Universal Screening 4th Grade
Fall 2012
MEAP4th Grade Fall 2012
Universal Screening 3rd Grade
Spring 2012
Universal Screening 3rd Grade
Winter 2012
Universal Screening 3rd Grade
Fall 2011
33
DIBELS Next Correlations with
4th Grade Fall 2012 MEAP Results
Previous Fall 2011
Previous Winter 2011
Previous Spring 2011
Concurrent Fall 2012
Composite .63 .60 .63 .64
DORF WC .59 .60 .60 .59
DORF Accuracy
.47 .39 .46 .45
DAZE .50 .52 .46 .59
Large =
Moderate =
34
When a classroom of students are screened in
reading
Some students will be on track
Some students will be at risk
We’ve CBMed…Now What?
Some students will be at significant risk
35
• At Risk for later low reading performance
• How do we know if this prediction is accurate?
• AUC• Sensitivity• Odds / Conditional Percentages
At Risk For…
36
• The probability a screening test will correctly classify a pair of students from two different categories (at-risk, not at-risk)
• A measure of a test’s overall diagnostic accuracy (e.g., rapid strep throat test)
Area Under the Curve (AUC)
37
AUCs Range From .5 to 1.0
38
Range across all assessment times within the study:
AUC RangeDiagnostic accuracy of DIBELS Next and
AIMSweb to MEAP
DIBELS Next AIMSweb
2nd Grade .81 - .85 6th Grade .78 - .79
3rd Grade .82 - .85 7th Grade .76 - .84
4th Grade .79 - .83 8th Grade .83
5th Grade .80 - .81
6th Grade .76 - .77
39
• Predict later performance on high-stakes reading assessments
• Nearly all correlations between CBM Reading and MEAP fall in the moderate to large categories
• Majority of AUC values are at or above .80
• DIBELS Next and aimsweb are efficient assessments that allow us to predict with some accuracy up to a year in advance
CBM Reading Assessments
40
We must know how our
screening tools and cut scores
are designed to function
41
Benchmark Goals and Cut Points for Risk Influence Our Decisions
Example A:Fall ORF
benchmark=70 wcpm
Example B:Fall ORF
benchmark = 97 wcpm
42
1. We cannot let any student who might be at risk slip through the cracks
2. We want an accurate picture without overwhelming our resources to provide intervention
Methods for Establishing Benchmark Goals and Cut Points for Risk
43
Classification Accuracy
Sensi
tivity
Specifi
city
44
If we don’t want to miss any students who might be at risk…
…then we may favor the sensitivity statistic when setting goals and cut points for risk
Sensitivity: Proportion of students correctly classified at-risk / nonproficient on both the screening and another critical assessment
Sensitivity
45
Decision rule for establishing the benchmark goal:
• Sensitivity of .90 or higher
Example: University of Oregon
Center on Teaching and Learning Cutpoints
46
GR2 Fall
GR3 Fall
GR4 Fall
GR5 Fall
GR6 Fall
GR2 W
inter
GR3 W
inter
GR4 W
inter
GR5 W
inter
GR6 W
inter
GR2 Spr
ing
GR3 Spr
ing
GR4 Spr
ing
GR5 Spr
ing
GR6 Spr
ing
GR3 Con
curre
nt
GR4 Con
curre
nt
GR5 Con
curre
nt
GR6 Con
curre
nt
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail
CTL Cut-point Sensitivity
Sen
siti
vity
Val
ue
47
GR2 Fall
GR3 Fall
GR4 Fall
GR5 Fall
GR6 Fall
GR2 W
inter
GR3 W
inter
GR4 W
inter
GR5 W
inter
GR6 W
inter
GR2 Spr
ing
GR3 Spr
ing
GR4 Spr
ing
GR5 Spr
ing
GR6 Spr
ing
GR3 Con
curre
nt
GR4 Con
curre
nt
GR5 Con
curre
nt
GR6 Con
curre
nt
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail
CTL Cut-point Sensitivity
Sen
siti
vity
Val
ue
SENSITIVITY = 0.90
48
CTL True Positive(Sensitivity)
False Positive
Identified At Risk(Positive)
False NegativeTrue Negative(Specificity)
Identified On Track(Negative)
49
• Quickly identify students that are likely to need additional support to prevent later academic difficulty.
DMG Development of Benchmark Goals and Cut Points for Risk:
50
Did not focus on one single rule or standard.
Primary consideration: • Odds a student has of meeting a future
reading goal based on their DIBELS Next score.
Secondary consideration: • Marginal percents and logistic regression.
DMG Development of Benchmark Goals and Cut Points for Risk:
51
DMG Development of Benchmark Goals and Cut Points for Risk:
Cut Point
Odds of Meeting Future Reading Goals and Passing High-Stakes Assessments
At or Above Benchmark: 80-90%
Below Benchmark: 40-60%
Well Below Benchmark: 10-20%
52
Grade 2 Grade 3 Grade 4 Grade 5 Grade 60
20
40
60
80
100
89 8991 91
82
65
59
73
61
47
38
31
47
29
21
Odd of Passing MEAP: DMG Composite Fall 2011
53
Grade 7 Grade 80
20
40
60
80
100
89 90
64
55
41
20
Benchmark
Strategic
Intensive
Odds of Passing MEAP: AIMSweb R-CBM
Fall 2011 to MEAP Fall 2012
54
• CTL Benchmark Goals emphasize sensitivity
• DMG Benchmark Goals emphasize odds / conditional percentages
• What happens when we look at the sensitivity of the DMG goals or the odds of the CTL goals?
Consider…
55
GR2 Fall
GR3 Fall
GR4 Fall
GR5 Fall
GR6 Fall
GR2 W
inter
GR3 W
inter
GR4 W
inter
GR5 W
inter
GR6 W
inter
GR2 Spr
ing
GR3 Spr
ing
GR4 Spr
ing
GR5 Spr
ing
GR6 Spr
ing
GR3 Con
curre
nt
GR4 Con
curre
nt
GR5 Con
curre
nt
GR6 Con
curre
nt
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail
CTL Cut-point Sensitivity
Sen
siti
vity
Val
ue
SENSITIVITY = 0.90
56
GR2 Fall
GR3 Fall
GR4 Fall
GR5 Fall
GR6 Fall
GR2 W
inter
GR3 W
inter
GR4 W
inter
GR5 W
inter
GR6 W
inter
GR2 Spr
ing
GR3 Spr
ing
GR4 Spr
ing
GR5 Spr
ing
GR6 Spr
ing
GR3 Con
curre
nt
GR4 Con
curre
nt
GR5 Con
curre
nt
GR6 Con
curre
nt
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
50
100
150
200
250
80
97
111 13
2 150
100 11
5 130 15
0
151
111 12
3 144 15
5
162
97
111 13
2 150
Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail
CTL Cut-point Value CTL Cut-point Sensitivity
Sen
siti
vity
Val
ue C
ut-p
oin
t Valu
e
SENSITIVITY = 0.90
57
GR2 Fall
GR3 Fall
GR4 Fall
GR5 Fall
GR6 Fall
GR2 W
inter
GR3 W
inter
GR4 W
inter
GR5 W
inter
GR6 W
inter
GR2 Spr
ing
GR3 Spr
ing
GR4 Spr
ing
GR5 Spr
ing
GR6 Spr
ing
GR3 Con
curre
nt
GR4 Con
curre
nt
GR5 Con
curre
nt
GR6 Con
curre
nt
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
50
100
150
200
250
52
70
90
111
107
72
86
103 12
0
109
87
100 11
5 130
120
70
90
111
107
80
97
111 13
2 150
100 11
5 130 15
0
151
111 12
3 144 15
5
162
97
111 13
2 150
Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail
DMG Cutpoint Value CTL Cut-point Value
CTL Cut-point Sensitivity
Sen
siti
vity
Val
ue C
ut-p
oin
t Valu
e
SENSITIVITY = 0.90
58
GR2 Fall
GR3 Fall
GR4 Fall
GR5 Fall
GR6 Fall
GR2 W
inter
GR3 W
inter
GR4 W
inter
GR5 W
inter
GR6 W
inter
GR2 Spr
ing
GR3 Spr
ing
GR4 Spr
ing
GR5 Spr
ing
GR6 Spr
ing
GR3 Con
curre
nt
GR4 Con
curre
nt
GR5 Con
curre
nt
GR6 Con
curre
nt
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
50
100
150
200
250
52
70
90
111
107
72
86
103 12
0
109
87
100 11
5 130
120
70
90
111
107
80
97
111 13
2 150
100 11
5 130 15
0
151
111 12
3 144 15
5
162
97
111 13
2 150
Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail
DMG Cutpoint Value CTL Cut-point Value
DMG Cutpoint Sensitivity CTL Cut-point Sensitivity
Sen
siti
vity
Val
ue C
ut-p
oin
t Valu
e
SENSITIVITY = 0.90
59
Odds of Passing MEAP: DMG-CTL Benchmark Fall 2011 DORF WC to Fall 2012 MEAP
Grade 2 Grade 3 Grade 4 Grade 5 Grade 60
20
40
60
80
100
9289
9289 84
97 95 96 95 95
DMG
CTL
60
Odd of Passing MEAP: Strategic Words Correct Fall 2011
Grade 2 Grade 3 Grade 4 Grade 5 Grade 60
20
40
60
80
100
72
67
75 75
57
88
83 85
77 79
DMG
CTL
61
Odd of Passing MEAP: Intensive Words Correct Fall 2011
Grade 2 Grade 3 Grade 4 Grade 5 Grade 60
20
40
60
80
100
50
37
51
47
24
60
52
63
54
61
DMGCTL
62
We should look at it from the perspective of districts, schools, and grade level
teachers!
Perhaps…
63
What are the Implications for Decision Making?Example: 3rd Grade Fall 2011
CTL WC DMG WC DMG Composite
64
DIBELS CTL
DIBELS DMG
MEAP
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Same District Fall 2011 3rd Grade
65
Same District Fall 2012 3rd Grade Marginal Percents
DIBELS CTL
DIBELS DMG
MEAP
DE Assessment
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
66
Resources to Make Informed Decisions about Measures
67
http://miblsi.cenmi.org/MiBLSiModel/Evaluation/UniversalScreening.aspx
MiBLSi Universal Screening Webpage
68
• Universal Screening Resources Overview
• FAQ on the Topic of Universal Screening
• Universal Screening Annotated Bibliography
• Considerations for Selecting a Universal Screening Measure
• CCSS: Reading and Math
• Sample of Completed Considerations Document (DIBELS Next, SRSS, easyCBM Math, Discovery Education Assessment, NWEA MAP)
MiBLSi Universal Screening Webpage
69
Considerations for
Selecting/Reviewing a
Universal Screening Measure
70
Document Overview
Section Questions
Need in District 1
General Features 2-9
Fit with Current Initiatives/Priorities 10-12
Evidence/Technical Adequacy 12-16
Readiness for Replication 17-19
Resources & Supports 20-32
Capacity to Implement 33-34
71
In God we trust, all others must bring data.
- W. Edwards Deming
See for yourself. . .