Provider Perceptions of the Child Outcomes
Summary Process
Lauren Barton and Cornelia TaylorOctober 27, 2012
Measuring and Improving Child and Family Outcomes ConferenceMinneapolis, MN
• Brief ENHANCE project update• Describe provider survey study• Share findings• Discuss implications
Today’s Session
ENHANCE Project Update
• Focus Under what conditions are COS data meaningful and
useful for accountability and program improvement? Impact of COS on programs/staff Identify needed revisions to COS form and guidance
• SRI International was funded by
U.S. Dept. of Education, IES -- July 1, 2009
ENHANCE
Four ENHANCE Studies
1) Comparison with Child Assessments
2) Team Decision-Making3) State Data Study4) Provider Survey
Studies 1,2, & 4:34 Project Data Collection Sites
17 Part C (Birth to 3)• Illinois• Maine• Minnesota• New Mexico• Texas • North Carolina• Virginia
17 Part B Preschool (3-5)• Illinois• Maine• Minnesota• New Mexico• Texas • South Carolina
Comparison with Child Assessments Study
Goals• Compare COS ratings to BDI-2, Vineland-II scores
Program Entry Program Exit
• Compare conclusions from COS and assessments
Study Status• Final stages of recruiting families
Find Out More• See poster at reception for promising findings from preliminary analyses
Team Decision-Making Study
Goals• Learn more about implementation of the COS process.• What is team understanding of outcomes and rating criteria?
Study Status• Videos coming in• Early stages of coding videos
Find Out MoreTabletop discussion tomorrow morning
What is quality COS process? Share draft video coding protocol
State Data Study
Goals• Analyze characteristics of COS data• Are patterns consistent with those predicted
for high quality data?
Status• Analyzing and compiling data
Find out more• See reception poster with early findings
We are looking for more states to participate.
Participate at different levels:• Share results from analyses you already do for APR OR• Share de-identified child level data set used in your APR
analyses• Work with SRI to either:
Share expanded data set* that we analyze and discuss with you OR
Conduct extra analyses and share results with project
* Expanded data set examples: include extra child descriptors (race, gender, primary disability) or variables describing the setting or composition of services
Participating in the State Data Study
Benefits of Participation
• See how your data fits relative to other state patterns• Have more analyses to learn from—extra tables for deeper
understanding• Technical assistance around unique issues in your data• Make a difference, generating new knowledge about a topic
that matters
For more information, contact [email protected] (650) 859-5314
ENHANCE Provider Survey
Question:How is COS implementation going?
Method• Online survey sent to providers in program participating in
the assessment and video studies
Study Status• Completed data collection• Early stages of analysis
Provider Survey
Provider Survey Sample
856 providers in 8 states Primary population
oEI: 472 (55%) oECSE: 302 (35%) oMix: 82 (10%)
Roleso 50% early interventionists/teacherso 38% therapists and asst. (SLP, OT, PT)o 9% coordinators/psychologistso 3% other
75% previously worked with children without disabilities in some capacity.
Provider experience with COS ratings
• 51% 31 or more COS ratings• 21% 10 or fewer COS ratings
Provider training
• Asked about training across a variety of formats (e.g., in person, technology, one-on-one, material review)
• How many total hours of training do you think most providers reported? None?
1-4 hours?
5-8 hours?
9-12 hours?
13 hours or more?
ECO recommends 8-12 hours of training
to get familiar with the COS process
Provider training on COS Process is limited
• 90% of providers received some training• 68% reported 4 hours or less of training
COS Process: Time involved
• On average, how long does it take to identify a child’s outcome ratings and document it on the form? Identify rating Discuss child’s functioning if directly related to rating decision
or exceeds discussions that would have occurred anyway Time to complete information on the form Do NOT include data entry of form, if done after form is
completed.1-15 min., 16-30 min., 31-45 min., 46-60 min., more than 60 min.
COS Rating Process is Fairly Brief
• 36% 1-15 minutes• 35% 16-30 minutes
• EI reported less time than ECSE
What do you think?A team process?
Teaming practices vary a lot from program to program.
• How often are COS ratings actually decided in teams?
• Do the teams include more than one professional?
• How often are families involved?
COS Process: More than one professional usually was involved.
75% had most ratings decided by a
team (at least 1 professional in addition to respondent)
79% had at least one other professional provide input about the child’s functioning
COS Process: Family involvement was limited.
34% had family members present for most of their COS rating decisions.
68% considered input from family members in determining the rating
Functioning across settings and assessment data was considered in most COS ratings.
75% used information about the child’s functioning from multiple settings and situations 76% used information from one or more assessment
tools during rating decisions
Most felt their COS process matched what it was supposed to be.
78% thought the process used for deciding COS ratings matched how it was supposed to be done
Most felt the process was thorough enough to be effective.
• 85% felt there was enough information about each outcome area to determine a COS rating
• 80% felt there was enough time to review child functioning in each outcome area.
• 85% in most of the decisions, all involved considered information carefully in an effort to decide an accurate rating.
Most providers felt that most COS ratings given were accurate.
76% felt confident that most of the ratings given were accurate
88% reported that NONE of the COS ratings were
selected to make the program look good
Skills Needed for COS Ratings
• Training should focus on skills essential for COS ratings. For example: Understanding age-expected functioning Comparing specific child’s skills to age-expected Understanding the 3 child outcomes Discussing functioning in outcomes with others
• To what extent do you think providers report having these kinds of skills?
Providers said they understood the content behind COS ratings.
Age-expected functioning• 89% understood age expected
functioning • 92% knew how to compare children’s
functioning to what is age expected
The three child outcomes • 85% understood the three child
outcomes • 83% knew how to discuss functioning
in the outcomes with others * Endorsed statements of understanding as mostly true or very true.
• How much do providers Understand why COS data are collected, Realize what happens with the data, and Feel they can explain the need for gathering COS
data?
Same or different?
Understanding the Purpose and Uses for COS Data
Few providers understood why and how COS data are used.
• 65% understood why COS data are
being collected
• 37% understood what happens with
the data
• 52% knew how to explain the need
for child outcomes data to others
What kinds of feedback and support are available to providers?• What do providers think about the system of
feedback and support available related to the COS? Are there people available to help them? Is ongoing support adequate? Are there people reviewing the COS forms?
Excellent, good, adequate, fair, poor, or a combination??
“Tell me and I forget, teach me and I may remember,
involve me and I learn.”- Benjamin Franklin
Limited ongoing support for providers with the COS process
82% someone is available to provide support if I ask for it
50% someone in my program provides support
47% ongoing support related to the COS process is adequate
37% someone in my program checks completed COS forms for accuracy
• What do providers say about how the COS process impacts them? Change awareness? Influence assessment or IFSP/IEP outcomes? Influence conversations and relationships with others? Impact on time for other activities?
Does it seem? Positive? Neutral? Negative?
How do providers describe the impact of COS on their practice?
Neutral impact of COS process on practice
Overall impact of COS on your work with children and families
Specifics reported about COS Process:• 2% had negative impacts on relationships with families• 17% improved the assessment process• 31% takes time away from other important actives• 30% helps focus discussion on the whole child
0 10 20 30 40 50 60 70 80
negative/ very nega-
tive, 7
neutral; 68
positive/ very posi-
tive, 25
Summary and Implications
Summary: COS Process
• Most received limited training and support
• Providers felt comfortable with background content
• Had limited understanding about what happens with the data or how to explain it
• Didn’t feel like the COS process impacts their work much
Got quality COS data?
Implications and Discussion
• Implications of findings For professional development? For supervision and administrators of programs? For data quality?
• Learning from providers Have others done surveys or focus groups to learn
from providers? Why or why not? How can surveys like this inform
improvement activities in your state?
Find out more
• ENHANCE Website– http://ENHANCE.sri.com
• ECO Center Website– http://www.the-ECO-center.org
• Contact ENHANCE staff– Email: [email protected]