ENHANCE UpdateResearch Underway on the Validity of the Child Outcomes Summary (COS) Process

ECO Center Advisory Board Meeting

March 8, 2012Arlington, VA

ENHANCE is funded by grant R324A090171 from the the U.S. Dept. of Education, Institute for Educational Sciences

Topics• Project design - review• Data collection progress• What we are learning

– Implementation– Child assessment study – State data study

• Next steps

Project Design

1. Conduct a program of research to examine the validity of ratings generated by COS and identify conditions that lessen validity.

2. Revise the COS and supporting materials based on study findings.

3. Identify a series of validity analyses that can feasibly be conducted in states to allow each state to examine the validity of its own COS data on an ongoing basis.

ENHANCE Project Objectives

Validity – What Are We Trying to Demonstrate?

• Validity is NOT a characteristic of an assessment or measurement device.

• Validity is a characteristic of the data produced by the tool and how these data are used.

• Are data valid for the purpose of…..• Implications:

– State A’s COS data could be valid; – State B’s COS data could not be.

Standards for Educational and Psychological Testing(1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education

Validation Process

• Validation process: – Develop propositions (validity argument) -

If data were valid for this use, then we would see….

– Collect evidence to examine each of those propositions

Examples of propositions in the COS Validity Argument

3. Children differ from one another with regard to level of functioning in the 3 outcome areas as reflected in COS ratings.

7. Functioning (COS ratings) in an outcome area at time 1 is related to functioning in that area at a later point in time.

9. COS ratings will be related to the nature and severity of the child’s disability.

Design

Real World

Not controlled Conditions

Design:37 Project Data Collection Sites

19 Programs Part C • Illinois• Maine• Minnesota• New Mexico• North Carolina• Texas • Virginia

18 Districts Part B Preschool • Illinois• Maine• Minnesota• New Mexico • South Carolina• Texas

9

Studies and Data Collection Progress

ENHANCE Studies

• Provider Survey• Team Decision-Making• Comparison with

Child Assessments • State Data Study

Provider SurveyGoals• Learn about COS implementation – processes in use• Identify providers’ knowledge and training experiences • Describe perceptions about if COS produces an accurate

rating and influences on that• Understand impact of COS on practice• Process & Sample

• Online survey responses• All providers in program who participate in COSStatus• Survey underway, continues through April

Team Decision-Making Study

Goals• Examine understanding and

application of outcomes and rating criteria

• Describe team process • Identify if ratings are consistent with

evidence discussedProcess & Sample• Video teams discussing COS ratings

• 210 children’s teamsStatus• Data collection underway• Code videos this summer & fall

What Are We Learning?

Considerable variability across states and even across programs, within a state Training Ongoing staff support and quality assurance Teaming (not just for COS) Parent involvement Timing and Process

Implication: Results will tell us about COS validity under

real-world conditions

Implementation

Number of Providers in COS Ratings - Preliminary

1 2 3 4 5 or more0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

EI % (n = 99)ECSE % (n = 61)

Perc

enta

ge o

f COS

form

s

Number of providers

Goals• Compare entry and exit COS ratings

to BDI-2 and Vineland-II scores• Compare conclusions from COS

and assessments

Process & Sample• Longitudinal, external assessor at

program entry & program exit • 216 children

Study Status• Local, trained assessors in place• Recruiting families since Aug. 2010• Sample shows expected variability,

including initial COS ratings, tool scores

Comparison with Child Assessments Study

Comparison with Child Assessments Study – Preliminary…

3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.

Validity argument claims

Distributions of Preliminary COS Ratings (1-7)

EI(n=71)

ECSE(n=49)

Ratings

3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.

10. COS ratings in the corresponding outcome areas are moderately correlated with:

o the social-emotional (Outcome 1), o cognitive (Outcome 2), o communication (Outcome 2), and o adaptive (Outcome 3) domain scores of assessment tools.

Validity argument claims

• Methods table

Methods

• Preliminary correlations between COS Ratings and assessment tools

• What expect to see?

Methods

.70

.42

.00

1.0

Preliminary Correlations: COS Ratings & Assessment Scores

ECSE larger COS-Assessment Correlations than EI

COS with relevant assessment tool domains EI Mean (range)

ECSE Mean (range)

Positive Social Relationships 0.33 (.20-.52) 0.61 (.51-.68)

Acquiring and Using Knowledge and Skills 0.25 (.22-.28) 0.70 (.67-.75)Taking Appropriate Action to Meet Needs 0.49 (.36-.64) 0.47 (.40-.57)

Correlations: BDI-2 and Vineland-II Domains

Vineland-II and BDI-2 EI Mean (range)ECSE Mean

(range)

Personal-Social 0.54 0.78

Cognitive/Communication 0.5 (.37-.63) 0.74 (.69-.83)

Adaptive /Motor 0.49 (.43-.67) 0.73 (.66-.86)

ECSE larger BDI-Vineland Correlations than EI

Methods

• COS Group 1 – Ratings of 1, 2, 3

• COS Group 2 – Ratings of 4, 5

• COS Group 3 – Ratings of 6, 7

Outcome 1: Positive Social Relationships

EI(n=71)

ECSE(n=49)

EI(n=71)

ECSE(n=49)

Outcome 2: Acquiring and Using Knowledge and Skills

EI(n=71)

ECSE(n=49)

Outcome 3: Taking Appropriate Action to Meet Needs

Summary of preliminary findings

Patterns• Means for groups generally follow expected directions

on assessment tools• Group comparisons showed expected differences• Effect sizes were nearly all larger for ECSE than EI

For COS – assessment tool comparisonsFor comparisons between assessment tools by the

same external assessor

More data are needed for final conclusions

State Data Study

Goals• Examine characteristics of COS data

and relationships to other variables• Look for consistency in patterns

across states to test claimsSample• All valid COS data within the state for a reporting year • 15-18 states conducting all analyses• Additional states sharing select analysesStatus• Piloted procedures with 3 Part C, 3 Part B Preschool states• Now working with 4 Part C, 6 Part B Preschool states• Recruiting more states, requesting data

7. Functioning, as reflected in the COS rating, in an outcome area at time 1 is related to functioning in that area at a later point in time.

Validity argument claims

Year Taking Appropriate Action to Meet Needs

State A 08-09 0.47State A 09-10 0.52State A 10-11 0.50State B 08-09 0.61State B 09-10 0.61State C 10-11 0.59

Correlations:Entry and Exit Ratings Part B 619

Preliminary state data

3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.

14. COS rating distributions at entry will be related to the disability-related characteristics of the population served by states.

Validity argument claims

Part C 08-09 entry ratings across states Taking appropriate action to meet needs

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

State AState BState D

1 2 3 4 5 6 7

COS Ratings

Next Steps and Reactions

Next steps

• Gather more state data• Complete data collection

involving local programs/districts• Analyze provider survey results• Code videos

• Questions? Reactions?• Implications for the national

data? • Implications for ECO?

Questions? Reactions?

Find out more

• ENHANCE Website– http://ENHANCE.sri.com

• ECO Center Website– http://www.the-ECO-center.org

• Contact ENHANCE staff– E-mail: ENHANCE@sri.com