understanding and using the results from the ncseam family survey batya elbaum, ph.d. ncseam...
TRANSCRIPT
Understanding and Using the Results from the
NCSEAM Family Survey
Batya Elbaum, Ph.D.NCSEAM
Measuring Child and Family OutcomesNECTAC National TA MeetingBaltimore, MD, August 2007
Part C Indicator #4
“Percent of families participating in Part C who report that early intervention services have helped the family a) know their rights, b) effectively communicate their children’s needs, and c) help their children develop and learn.”
Sample results Mean = 594, S.D. = 147, S.E. = 10 Percent of families who report that
early intervention services have helped the family a) know their rights = 59% b) effectively communicate their
children’s needs = 54% c) help their children develop and
learn = 65%
What do the numbers mean?
Interpreting the measures and percentages
Measuring the 3 subindicators
To measure something means to locate it on a continuum.
Do we need three different measurement rulers for the three specific OSEP outcomes, or can we use a single measurement ruler?
Measuring the 3 subindicators
NCSEAM analyses of responses from thousands of families confirmed that the three outcomes selected by OSEP as family outcome indicators are all part of a single hierarchy of positive family outcomes.
Measuring the 3 subindicators Families appear to achieve these
outcomes in a very consistent order. Families who report that EI helped them
know their rights almost invariably report that EI also helped them help their child develop and learn.
Families who report that EI helped them effectively communicate their children’s needs almost invariably report that EI also helped them know their rights and help their child develop and learn.
Locating families on the pathway
The data tell us that there is a continuum of “Impact of EI Services on Families”, such that the impact can range from very low to very high.
----------------------------------------------------- Very Low Very High
Different families experience different levels of positive impact as a result of their participation in early intervention, and so different families will be located at different points along this continuum.
Locating families on the pathway
When we put numbers on the continuum, we create a “measurement ruler” measuring the impact of EI for each family
Fam-1 Fam-2 Fam-3
↓ ↓ ↓
-----|----------|----------|----------|-----
400 500 600 700
Item locations on the rulerItem
Calibration Item
Stem: Over the past year, Early Intervention services have helped me and/or my family:
678 Participate in typical activities for children and families in my community.
656 Know about services in the community.
640 Know where to go for support to meet my family's needs.
625 Keep up friendships for my child and family.
609 Know where to go for support to meet my child's needs.
584 Be more effective in managing my child's behavior.
576 Make changes in family routines that will benefit my child with special needs.
576 Do activities that are good for my child even in times of stress
570 Improve my family's quality of life.
565 Feel that I can get the services and supports that my child and family need.
563 Get the services that my child and family need.
562 Feel that my family will be accepted and welcomed in the community
559 Feel more confident in my skills as a parent.
559 Feel that my child will be accepted and welcomed in the community.
556 Communicate more effectively with the people who work with my child and family.
553 Understand how the Early Intervention system works.
546 Understand the roles of the people who work with my child and family.
539 Know about my child's and family's rights concerning Early Intervention services.
534 Be able to evaluate how much progress my child is making.
516 Understand my child's special needs.
498 Feel that my efforts are helping my child.
498 Do things with and for my child that are good for my child's development.
Location of key items related to Indicator C4
Indicator 4c: Help their children develop and learn. IFS items: “Understand my child's special needs.” [516] “Do things with and for my child that are good for my child's development.” [498]
Indicator 4b: Effectively communicate their children’s needs. IFS Item: “Communicate more effectively with the people who work with my child and family.” [556]
Indicator 4a: Know their rights.IFS Item: “Know about my child's and family's rights concerning Early Intervention services.” [539]
556
539
516
Measurement of family outcomes at different levels
Just as we can locate individual families on the ruler, we can take the average measure for all families in the state and locate the state on the ruler.
Similarly, we can take the average of all families in a particular program and locate that program on the ruler.
What is the meaning of the percentage?
The percentage reported to OSEP is the percent of families with measures that are at, or above, a certain point on the measurement ruler.
Percent of measures above the adopted standards
standards
Sample descriptive interpretation Approximately 90-95% of families agreed,
with approximately two-thirds of families expressing strong or very strong agreement, that early intervention helped them:
Understand their child’s special needs. Do things with and for their child that
are good for their child’s development. Feel that they can handle the
challenges of parenting a child with special needs.
Sample descriptive interpretation Approximately 80-90% of families agreed,
with approximately 50-60% expressing strong or very strong agreement, that early intervention helped them:
Know about their child’s and family’s rights concerning early intervention services.
Know where to go for support to meet their child’s needs.
Be more effective in managing their child’s behavior.
Sample descriptive interpretation Approximately 70-75% of families
agreed, with about 45% expressing strong or very strong agreement, that early intervention helped them:
Communicate effectively with the people who work with their child and family.
Know about services in the community.
Sample descriptive interpretation
Approximately 65-70% of families agreed, with about 40% expressing strong or very strong agreement, that early intervention helped them:
Participate in typical activities for children and families in the community.
The meaning of “agreement” The raw percentages of families that
agree (in any category of agreement) with the key items related to the indicators is higher than the percentages reported on the indicator. Why?
Because the standard that most states adopted for the indicators is a higher standard than a simple “agree.”
Evaluating the percentages on the indicator
Against what standard is the percentage being evaluated?
What percent of families do we want to be at or above the measure that represents our idea of where families should be on the continuum of positive family outcomes?
The APR Report on C4
What to cover in relation to your data
Sample Method Results Interpretation
Sample Describe the sampling plan Describe the extent to which the
obtained sample is representative of the population
State your survey response rate
Method: Instrument
Describe the instrument Provide a copy of the instrument Report reliability estimates for the
instrument (based on NCSEAM pilot, .90 or above using 22 items)
Method: Procedure Describe how the survey was
distributed/administered (face-to-face, mail survey, online)
State whether the survey was available in languages other than English
Describe efforts to ensure the participation of low-literacy families
Results: Data analysis Describe how the data were
analyzed Describe how the percent on the
subindicators was calculated
Results: Reliability Report the reliability of the data Report a confidence interval for
the percent on each subindicator (this confidence interval will depend on the size of your sample)
Results: Validity Face validity: Does the instrument
appear to address what we are supposed to be measuring?
Convergent validity: Do the results accord with results obtained from other sources (e.g., monitoring data) that address the same thing?
Interpretation Report progress (or slippage) in
relation to target Consider the results in relation to:
Effectiveness of improvement activities
Extent of implementation of improvement activities
Improving Performance on the Indicator
Short-term goals Increase response rates Ensure equal access to
participation Build a broad base of
understanding of the indicator and how it is measured
Longer-term goals Improve performance on the indicator Document implementation of
improvement activities Examine the relation between
implementation and improvement so as to better understand “what works”
Improve performance on the indicator for all subgroups
