patterns in child outcomes summary data:
DESCRIPTION
Patterns in Child Outcomes Summary Data:. Analytic Approaches and Early Findings from the ENHANCE Project. Cornelia Taylor, Lauren Barton, Donna Spiker September 19-21, 2011. Measuring and Improving Child and Family Outcomes Conference New Orleans, LA. Today’s session. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/1.jpg)
Patterns in Child Outcomes Summary
Data:
Cornelia Taylor, Lauren Barton, Donna Spiker
September 19-21, 2011 Measuring and Improving Child and Family Outcomes Conference
New Orleans, LA
Analytic Approaches and Early Findings from the ENHANCE Project
![Page 2: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/2.jpg)
• Provide a brief update about ENHANCE
• Identify the purpose and approach of the state data study
• Describe some preliminary findings from initial states involved in the state data study
• Explain how other states could examine their own data in the same way as that presented
• Discuss any emerging implications for validity of the COS and for interpreting individual state data
Today’s session
![Page 3: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/3.jpg)
Origins of ENHANCE
Need for Outcomes
Data – Challenging to Collect
![Page 4: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/4.jpg)
Origins of ENHANCE
Need for Outcomes
Data – Challenging to Collect
COS Process
Implemented > 40 States,
Little Systematic
Validation for Use in
Accountability
![Page 5: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/5.jpg)
Origins of ENHANCE
Need for Outcomes
Data – Challenging to Collect
COS Process
Implemented > 40 States,
Little Systematic
Validation for Use in
Accountability
?Investigate… Learn
![Page 6: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/6.jpg)
Early Evidence
Belief in potential for COS process to be valid based on:• Existing literature: team-based decision-making can
be reliable and valid• Existing literature: teams are effective in identifying
individual children’s functioning so that they can plan and deliver appropriate services
• Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups
• Anecdotal data from trainers: participants reach decisions fairly easily and consistently
![Page 7: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/7.jpg)
ENHANCE
• Project launched by the Early Childhood Outcomes Center (ECO) and SRI International
• Funded by the U.S. Dept. of Education, Institute for Educational Sciences – July 1, 2009
• Series of studies designed to find out:– the conditions under which the Child Outcomes Summary
(COS) Process produces meaningful and useful data for accountability and program improvement
– the positive and/or negative impact of the COS process on programs and staff
– what revisions to the form and/or the process are needed
![Page 8: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/8.jpg)
Four ENHANCE Studies
1) Comparison with Child Assessments
2) Team Decision-Making3) Provider Survey4) State Data Study
![Page 9: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/9.jpg)
Studies 1-3:34 Project Data Collection Sites
17 Part C (Birth to 3)• Illinois• Maine• Minnesota• New Mexico• Texas • North Carolina
17 Part B Preschool (3-5)• Illinois• Maine• Minnesota• New Mexico• Texas • South Carolina
9
![Page 10: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/10.jpg)
Goals• Compare COS ratings to BDI-2, Vineland-II scores
Program Entry Program Exit
• Compare conclusions from COS and assessmentsSample• 108 children - birth to 3• 108 children - 3 – 5 yearsStudy Status• Recruiting families• About ½ of the sample enrolled• See expected variability in sample (ages, disability types)
and initial COS ratings/assessment scores
Comparison with Child Assessments Study
![Page 11: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/11.jpg)
Team Decision-Making Study
Goals
• Learn more about the implementation of the COS process, including how the team reaches a decision about a rating and what is discussed.
• Do COS ratings assigned match the developmental level of the behaviors presented in the meeting?
• What is team understanding of outcomes and rating criteria?Sample• 180 children each from Part C & Part B 619
½ entry & ½ exit meetingsStudy Status• Starting data collection now in about ½ the sites• 19 videos received• Expect to start coding videos Summer 2012
![Page 12: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/12.jpg)
Goals• What processes are being used to determine COS ratings?• What is the impact of the COS process on practice? • What have providers learned about the COS? • What else would be helpful? Sample• All providers in the program who participate in the COS process are invited to participateStudy Status• Developing survey content• Survey expected Spring 2012
Provider Survey
![Page 13: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/13.jpg)
Goals• Analyze characteristics of COS data and relationships to
other variables• Look for consistency in patterns across statesExamples of Questions• Are patterns in COS data across states consistent with those
predicted for high quality data?• How are COS ratings related to hypothesized variables (e.g.,
disability type) and not to other variables (e.g., gender)? • How are team variables related to COS ratings? Sample• All valid COS data within the state for a reporting year • 15-18 states conducting all analyses• Additional states sharing select analyses
State Data Study
![Page 14: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/14.jpg)
• Refined procedures for gathering data tables by gathering data from a preliminary group of 6 states Mostly states used procedures and
generated data tables A few provided formatted data files
for SRI to analyze
• Beginning to analyze data from that preliminary group
• Soon will request data from other states in state data study and permission to use relevant data additional states have already analyzed and shared
State Data Study: Status
![Page 15: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/15.jpg)
State Data Study:Preliminary Data from 5 States
3 Part C (Birth to 3) 3 Part B Preschool (3-5)
![Page 16: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/16.jpg)
How would these data analyses be conducted?• States would send data to SRI annually
– de-identified data files OR– aggregate output or reports from a set of
requested analyses• Examples of analyses include
– the distributions of entry and exit COSF scores– relationships between outcomes– relationships between outcomes across time– relationships of outcome scores to other factors
such as disability and gender
![Page 17: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/17.jpg)
What data would I need to submit?
• Data collected at entry and exit from Part C and Part B 619 programs– COSF ratings– Additional child descriptors (e.g. race, gender,
primary disability)– Variables that describe the setting or composition
of the services
![Page 18: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/18.jpg)
How will I submit data?
• De-identified data files – Templates developed in MS Excel– Submitted through a secure server
• Analyzed data– Table shells developed in MS Word and MS Excel– Submitted through secure server or emailed
![Page 20: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/20.jpg)
Questions? Comments? Reactions?
![Page 21: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/21.jpg)
Entry rating distributions
![Page 22: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/22.jpg)
• What should entry ratings look like? Should they differ across outcomes? Where do most of the ratings fall? How much should the extremes of the scale be
used ( 1 or 7)?
Entry Rating Expectations
![Page 23: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/23.jpg)
Entry Data Analysis
• The following data are from 3 Part C programs and 2 Part B programs
• All data are from 08 – 09• The data are entry cohorts
– i.e. all children who entered during the FFY
![Page 24: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/24.jpg)
Part C 08-09 entry ratings across states; Outcome A
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
State AState BState D
1 2 3 4 5
6 7
![Page 25: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/25.jpg)
Part C 08-09 entry ratings across states; Outcome B
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
State AState BState D
1 2 3 4 5
6 7
![Page 26: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/26.jpg)
Part C 08-09 entry ratings across states; Outcome C
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
State AState BState D
1 2 3 4 5
6 7
![Page 27: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/27.jpg)
Outcome A – Average Entry Ratings
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Part C (n =3)Part B (n=2)
![Page 28: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/28.jpg)
Outcome B – Average Entry Ratings
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Part C (n =3)Part B (n=2)
![Page 29: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/29.jpg)
Outcome C – Average Entry Ratings
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Part C (n=3)Part B (n=2)
![Page 30: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/30.jpg)
• The difference in distributions between Part C and Part B are largest for Outcome C Children in Part B enter with higher ratings
Things to notice
![Page 31: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/31.jpg)
Part C 08-09 average ratings across outcomes
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
OC-AOC-BOC-C
![Page 32: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/32.jpg)
• Variations in patterns across outcomes
Things to Notice
![Page 33: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/33.jpg)
• More that ½ of all children enter with a COS rating of 3,4 or 5 across outcomes.• An average of 12% of children enter at with the very
lowest (1) or the very highest (7) across outcomes.• The typical entry distribution has most children
towards the middle of the distribution.
Conclusions Across Part C and Part B
![Page 34: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/34.jpg)
No Action Interpretation: You may be serving a population that is higher or lower functioning that other states.
Pattern Check: if the distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution.
Action Interpretation: Your providers may be systematically misunderstanding the definition of COS rating points.
![Page 35: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/35.jpg)
• Correlations between entry ratings• Cross tabs of entry ratings by:
Program Primary disability Race/ethnicity
Additional Entry Analysis
![Page 36: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/36.jpg)
Exit distributions
![Page 37: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/37.jpg)
Part C 08-09 exit ratings across states; Outcome A
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State AState DState E
1 2 3 4 5
6 7
![Page 38: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/38.jpg)
Part C 08-09 exit ratings across states; Outcome B
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State AState DState E
1 2 3 4 5
6 7
![Page 39: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/39.jpg)
Part C 08-09 exit ratings across states; Outcome C
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State AState DState E
1 2 3 4 5
6 7
![Page 40: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/40.jpg)
Part B 08-09 exit ratings across states; Outcome A
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State AState DState E
1 2 3 4 5
6 7
![Page 41: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/41.jpg)
Part B 08-09 exit ratings across states; Outcome B
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State AState DState E
1 2 3 4 5
6 7
![Page 42: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/42.jpg)
Part B 08-09 exit ratings across states; Outcome C
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
State AState DState E
1 2 3 4 5
6 7
![Page 43: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/43.jpg)
Outcome A – Average Exit Ratings
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
Part C (n=3)Part B (n=3)
![Page 44: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/44.jpg)
Outcome B – Average Exit Ratings
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
Part C (n=3)Part B (n=3)
![Page 45: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/45.jpg)
Outcome C – Average Exit Ratings
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
Part C (n=3)Part B (n=3)
![Page 46: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/46.jpg)
Part C 08-09 average exit scores across outcomes (state n = 3)
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
OC-AOC-BOC-C
![Page 47: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/47.jpg)
Part B 08-09 average exit scores across outcomes (state n=3)
1 2 3 4 5 60.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
OC-AOC-BOC-C
![Page 48: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/48.jpg)
• Variation in ratings across outcomes• The exit distribution is shifted toward a higher rating
than is the entry distribution• For Part B, the average percent of children with a
rating of 7 is much higher for Outcome C than for the other two outcomes
Things to Notice
![Page 49: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/49.jpg)
No Action Interpretation: You may be serving a lower functioning group than other states• If this interpretation is
true, it should also be apparent in your entry distribution
Pattern Check: the distribution of exit scores in your state is not skewed towards the higher end of the rating scale.
Action Interpretation: The children in your programs may not be making expected gains.
![Page 50: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/50.jpg)
• Choosing a metric for looking at paired distributions Progress categories Side-by-side entry exit comparisons
• Both of the above can be completed using the COS calculator 2.0
Entry-Exit Paired Distribution
![Page 51: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/51.jpg)
• How many points the child’s rating changed between entry and exit?
• What would you expect to see?
Exit rating minus Entry rating
Exit Rating Entry Rating Exit rating – Entry rating
7 3 4
4 4 0
5 2 3
3 4 -1
![Page 52: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/52.jpg)
Part C exit score – entry score; 08-09
-4 -2 0 2 4
![Page 53: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/53.jpg)
Part B exit score – entry score; 08-09
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
State BState CState D
-2 0 2 4
![Page 54: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/54.jpg)
Things to Notice
• Most children’s ratings increase 1, 2, or 3 points, or they stay the same
• Very few children have ratings that decrease• However, more children have ratings that decrease in
Part C than in Part B
![Page 55: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/55.jpg)
No Action Interpretation: Your programs are very effective and children make large gains (verify!).
Pattern Check: if a large percentage of children in your state make large increases in their ratings
Action Interpretation: Providers are not using the scale correctly and may be inflating exit ratings and/or deflating entry ratings.
![Page 56: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/56.jpg)
Additional entry-exit analysis
• Correlations between entry and exit• Progress categories by other variables (e.g., disability
type, primary language)
![Page 57: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/57.jpg)
• The distribution of entry scores in your state seems to be heavily weighted towards one end or the other of the distribution
• The distribution of exit scores in your state is not skewed towards the higher end of the rating scale.
• A large percentage of children in your state make large increases in their ratings
Summary of pattern checks
![Page 58: Patterns in Child Outcomes Summary Data:](https://reader036.vdocument.in/reader036/viewer/2022062400/56815259550346895dc08de9/html5/thumbnails/58.jpg)
Find out more
• ENHANCE Website– http://ENHANCE.sri.com
• ECO Center Website– http://www.the-ECO-center.org
• Contact ENHANCE staff– E-mail: [email protected]