measuring child welfare agency performance: advantages and challenges of state, county, university...
DESCRIPTION
u Government Performance Results Act of 1993 (GPRA) u Annual Outcomes Report to Congress mandated by Adoption and Safe Families Act (ASFA) of 1997 u Statewide Data Indicators in Child and Family Services Reviews -- a subset of the Annual Outcomes—from National Child Abuse and Neglect Data System (NCANDS) and Adoption and Foster Care Analysis and Reporting System (AFCARS) u California Child Welfare System Improvement and Accountability Act (AB636) became law in 2001 and went into effect in January 2004 Outcomes, outcomes, everywhereTRANSCRIPT
![Page 1: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/1.jpg)
Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County,
& University Collaboration
National Association of Welfare Research and Statistics45th Annual WorkshopMadison, Wisconsin
August 31, 2005
Kelly CrossSan Bernardino County HS
Terry V. ShawCenter for Social Services ResearchUniversity of California at Berkeley
The Performance Indicators Project is funded by the California Department of Social Services and the Stuart Foundation
![Page 2: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/2.jpg)
Measuring Child Welfare Outcomes
Child In Child OutA bunch of stuff happens
*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference
![Page 3: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/3.jpg)
Government Performance Results Act of 1993 (GPRA)
Annual Outcomes Report to Congress mandated by Adoption and Safe Families Act (ASFA) of 1997
Statewide Data Indicators in Child and Family Services Reviews -- a subset of the Annual Outcomes—from National Child Abuse and Neglect Data System (NCANDS) and Adoption and Foster Care Analysis and Reporting System (AFCARS)
California Child Welfare System Improvement and Accountability Act (AB636) became law in 2001 and went into effect in January 2004
Outcomes, outcomes, everywhere
![Page 4: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/4.jpg)
Quarterly distribution of county specific outcome
indicators data• Includes national standards (from AFCARS),
but also draws heavily on previous work done by CWDA and UCB using entry cohort measures
• Mirrors Family to Family Outcomes
• Retains key process measures (e.g., child visits, time to investigation)
![Page 5: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/5.jpg)
Statewide Data Indicators from
AFCARS Stability Of Foster Care
Placement Length Of Time To
Reunification Foster Care Re-entries Length Of Time To Adoption
![Page 6: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/6.jpg)
Why do we use entry cohort measures in addition
to measures from AFCARS?
![Page 7: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/7.jpg)
Who is in AFCARS? AFCARS contains data on children in
foster care during a federal fiscal year
Each reporting period’s submission is a separate dataset. Reporting periods are linked together by the Children’s Bureau to form the annual databases. ANNUAL DATABASES ARE NOT LINKED TO EACH OTHER.
![Page 8: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/8.jpg)
11/02 11/03 11/04
Data snapshots can be biased
Source: Aron Shlonsky, University of Toronto (formerly at CSSR)
![Page 9: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/9.jpg)
California EXAMPLE: Age of Foster Children
(2003 first entries, 2003 exits, July 1 2003 caseload)(2003 first entries, 2003 exits, July 1 2003 caseload)
22
31
22 20
54
30
24 2219
5
24 24
32
16
0
10
20
30
40
50
<1 1-5 6-10 11-15 16+
Age in Years
Perc
ent
EntriesExitsPoint in Time
![Page 10: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/10.jpg)
California EXAMPLE:Median Length of Stay in Months
(1998-2000 first entries, 2001 first spell exits, July 1 2002 first spell caseload)
1821
13
22
14
2126
17 1821
2528
2419
46
0
10
20
30
40
50
All Kin Foster FFA Group
Placement Type
Mon
ths
Entries
Exits
Point inTime
![Page 11: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/11.jpg)
• Federal Measure: Of all children who were adopted during the year, what % had been in care for less than 24 months? (national standard = 32%)
• State enriched: Of all children entering care for the first time, what % are adopted in less than 24 months? (we do not have state standards)
![Page 12: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/12.jpg)
Percent of children exiting care to finalized adoption in less than 24 months
(32% National Standard)
• Baseline: 100 kids exiting to adoption, 33 of them within 24 months=33%. Substantial conformity achieved!
Two pronged approach (1) Faster adoption for 100 children, 50 of them within 24 months=50%,
(2) adoptions for 100 kids in long term care
• 2 years later: 200 kids exiting to adoption, 50 within 24 months=25%. Substantial conformity NOT achieved!?!
![Page 13: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/13.jpg)
Are you getting better or worse? Data from the Multi State Data Archive
Adoption within 24 Months
State A
0%
5%
10%
15%
20%
25%
30%
1990
1991
1992
1993
1994
1995
1996
1997
Ado
pted
FederalState
year
Source: Chapin Hall Center for Children
![Page 14: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/14.jpg)
Why don’t we have state standards ?
![Page 15: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/15.jpg)
The Cycle of Experiences in the Child Welfare The Cycle of Experiences in the Child Welfare SystemSystem
CounterbalancedCounterbalancedIndicators ofIndicators of
SystemSystemPerformancePerformance
PermanencyPermanencyThroughThrough
Reunification,Reunification,Adoption, orAdoption, orGuardianshipGuardianship
LengthLengthOf StayOf Stay
StabilityStabilityOf CareOf Care
Rate of Referrals/Rate of Referrals/Substantiated ReferralsSubstantiated Referrals Home-BasedHome-Based
Services vs.Services vs.Out-of-HomeOut-of-Home
CareCare
Positive Positive AttachmentsAttachments
To Family,To Family,Friends, andFriends, andNeighborsNeighbors
Use of LeastUse of LeastRestrictiveRestrictive
Form of CareForm of Care
Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families,
Reentry to CareReentry to Care
![Page 16: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/16.jpg)
Lack of understanding about the limitations of the national standards, and pressure to
achieve “substantial conformity” (pass), could
drive changes in policy and practice that may not be best
for children and families.
![Page 17: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/17.jpg)
Kelly CrossSan Bernardino County HS
![Page 18: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/18.jpg)
AB636 Components
• Quarterly distribution of county specific outcome indicators data• County Self Assessment• Peer Quality Case Review• County Self Improvement Plan• Continuous monitoring of outcomes
![Page 19: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/19.jpg)
Limitations of Administrative Data
• Adapting a case management system into a mechanism for tracking longitudinal outcomes.
• A three-year cycle of outcome evaluation•Still are not ready to set a baseline•Must refine measurement methods, data clean up and training.
![Page 20: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/20.jpg)
Examples of County Work to Examine Data and Improve Practice
• Changing policy•Social worker monthly contacts•Out of home abuse.
• Confounding policy and data issues•ICWA status•Recurrence of maltreatment
• Associated referrals• “Substantial risk,” & “Sibling abused, child at
risk”
![Page 21: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/21.jpg)
Examples of County Work to Examine Data and Improve Practice
continued
• Complex outcomes—Required health visits•One of several complex measures in development
• Identify CHDP standards• Allow data entry lag• Revise code to account for different time
periods when children are in compliance
• An example of improved practice—Reentry•Examining data helped target those children
most at risk of returning to care after reunification
•Expanded use of Public Health Nurses
![Page 22: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/22.jpg)
AB636=State / County Partnership
• Shifts focus from process measured compliance to outcome based review system, but requires linking outcomes to related processes.
• Data are our friends, not our dictators. • Requires county collaboration with community partners (SIPs signed by Boards of Supervisors).
• Promotes sharing of promising practices among counties.
![Page 23: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/23.jpg)
More Advantages of University Involvement
• Participate regularly on state and county workgroups and committees.
• Share programming code and seek input from county partners on its continual improvement.
• Ensure availability to answer ad hoc
questions
![Page 24: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/24.jpg)
Data and Policy Committees
• Continual refining of measurement process requires both Data and Policy Committees.
• Policy committee—interprets regulatory implications and decides general structure that measurement will conform to (e.g., inclusion of guardianship in measures).
• Data committee—determines specific data
collection & analysis steps necessary to implement measurement guidelines decided by policy group.
![Page 25: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/25.jpg)
Examples of County Work to Examine Data and Improve Practice
continued
• Complex outcomes—Required health visits•One of several complex measures in development
• Identify CHDP standards• Allow data entry lag• Revise code to account for different time
periods when children are in compliance
• An example of improved practice—Reentry•Examining data helped target those children
most at risk of returning to care after reunification
•Expanded use of Public Health Nurses
![Page 26: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/26.jpg)
UCB Website
cssr.berkeley.edu(Child Welfare Services Reports)
includes
age, ethnicity, gender breakoutskin vs non-kin
for all AB636 measures and more
use “Datadude” to examine performance over time
![Page 27: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/27.jpg)
Lessons Learned
•It takes time.
•Keep accurate records of development & policy decisions. •Participant turn over (state, county, & university)
•Discovering new populations
![Page 28: Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County, University Collaboration National Association of Welfare Research](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b957f8b9ab0599c359f/html5/thumbnails/28.jpg)
State Websites http://www.dss.cahwnet.gov/cdssweb/
(Child Welfare Systems Improvements)
http://www.childsworld.ca.gov/CDSSCounty_1954.htm
(AB636 Quarterly Reports)
Kelly Cross [email protected]
(909) 388-0174