center for social services research school of social welfare, uc berkeley the child and family...
TRANSCRIPT
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
The Child and Family Services Review Composite Scores:A “Great Start”
Barbara Needell, MSW, PhD Center for Social Services Research
University of California at Berkeley
*Schuerman, J., & Needell, B. (2009). The Child and Family Services Review Composite Scores: Accountability off the Track. Chicago: Chapin Hall at the University of Chicago
http://www.chapinhall.org/research/report/child-and-family-services-review-composite-scores-accountability-track
The Performance Indicators Project is a collaboration of the California Department of Social Services and the University of California at Berkeley,
and is supported by the California Department of Social Services and the Stuart Foundation
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
CFSR Overview
• Review and evaluation of state foster care and adoption services federally funded by Titles IV-B and IV-E of the Social Security Act
• Carried out by the Children’s Bureau of the Administration on Children and Families (ACF)
• GOAL: Evidence-based accountability and performance improvement of state child welfare systems
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
The CFSR Process
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Flaws in Performance orFlawed Process?
• States not in “substantial conformity” must develop a Program Improvement Plan (PIP)
• To date, no state has passed the CFSR at the initial stage – every state has had to develop a PIP
• Financial penalties may be incurred if the PIP is not successfully completedBut there are weaknesses in the CFSR process
(we will focus on the National Standards and composite scores)
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
State Demographic Variation
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Systemic State Differences in Caseload Inclusion Criteria
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Data Quality Weaknesses
• Database (AFCARS) not designed to measure longitudinal performance, but ACF attempts to stitch together files to construct a pseudo-longitudinal database
• Prone to record linkage errors and still does not include all data needed to track performance over time
• Data is not of adequate quality to warrant the severity of consequences
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Practice and Policy Conflicts between Measures
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Equal Count/Weighting of States Despite Differences in Child Population Size
CA and VT are weighted equally, although there
are 60x as many children served in CA foster care as in VT
foster careCA
VT
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Statistical Method Used is Complicated and Not Superior to a Simpler Approach
• ACF uses principal components analysis (PCA), which is often used to simplify a large number of similar variables
• This statistical manipulation does not necessarily produce a better result than a more straightforward approach, such as a simple average
• Because the composite measure is an abstraction, it is not clear what actions should be taken to improve scores and may even encourage gaming of the system
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Arbitrary and statistically inappropriate decisions in using PCA
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Arbitrary National Standard and Adjustment
• National standard set at 75th percentile with no clear justification
• 75th percentile is actually that for the normal curve, a theoretical statistical distribution
• State outcome data are not necessarily normally distributed
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Flawed Method to Develop Minimum Improvement
• CFSR2 requires a percentage improvement by states, based on their baseline performance in a given year
• Amount of required improvement determined using a complex calculation involving sampling error, which is not justifiable statistically
• The national standards largely use “p”-- the percent of time an event happens, which should not be used in this context
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Example
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Summary/Conclusion• Flaws and issues with the national standards and composite scores
are substantial enough to take a fresh look at “substantial conformity”• Note: We have not discussed other areas of the CFSR at all today—
the State Data Profile, the Case Record Reviews, the Stakeholder Interviews, etc—and all of these may also be in need of review/reform
• Not clear that reasonable outcome indicators can be developed from the currently available AFCARS
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Summary/Conclusion• A fully longitudinal national foster care database is
necessary to adequately measure performance (e.g., State Data Center Foster Care Database,California Child Welfare Performance Indicators ProjectAnalytic Assistance for Child Welfare…in North Carolina.)
• We have the ability NOW to reform the CFSR process, including the performance measures that come from administrative data
• The development of a functional CFSR process can encourage and foster true improvement in state child welfare systems
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
CSSR.BERKELEY.EDU/UCB_CHILDWELFARE
Barbara [email protected]
510 290 6334
Presentation Developed by Christine Wei-Mien Lou