supervisory core - calswec · web view(trainer's slides powerpoint [ppt] slide show) slide 1...

SUPERVISOR CORE:MANAGING FOR RESULTS

- Trainer’s Guide -

TRAINING TIPS AND CONTENT

PART 1

1. IntroductionTake several minutes for welcome, introduction of trainer, and trainees (depending on size of group), and brief overview of session agenda.

Child welfare services agencies are required by the C-CFSR to monitor the impact of their programming efforts for children, youth, and families through performance on a series of outcome measures. This training module provides a basic understanding of: outcome evaluation concepts, specific measures tracked by the C-CFSR process, and useful approaches for applying outcome information to monitor, assess, and improve practice.

2. Learning Objectives

KnowledgeK1. The trainee will be able to recognize key concepts and terminology for outcome

measurements (e.g., median, point in time, entry cohort, measures, rolling year).

K2. The trainee will be familiar with the outcome measures on the quarterly, county-specific reports created as part of the C-CFSR process (i.e., the difference between the federal and state-enhanced measures).

K3. The trainee will be able to explain how to navigate the Child Welfare Dynamic Report System website of the UC Berkeley Center for Social Services Research, and understand that all data on the C-CFSR quarterly, county-specific reports (as well as much more detailed information) can be located there.

K4. The trainee will be able to explain the interrelated nature of the outcome measures for safety, permanency, and well-being that are tracked on the quarterly county-specific outcome reports.

Supervisor Core | Managing for Results: Trainer’s Guide |Version 1.1, July 2009

1

Skills

S1. The trainee will be able to demonstrate the ability to ask policy and practice questions related to trends identified in the C-CFSR outcome measures.

S2. The trainee will be able to develop a plan to improve the delivery of child welfare services in his or her agency through monitoring outcomes with the C-CFSR process.

ValuesV1. The trainee will value the need to supervise and monitor in ways to achieve

ASFA outcomes.

V2. The trainee will value ongoing recognition of trends identified in the outcome measures that enhance culturally sensitive and responsive practice by helping target services appropriately to ethnic, age, and gender groups.

V3. The trainee will value incorporating various practice issues into supervision:A. Outcomes vs. processB. Fairness and equityC. Incorporating data

3. Background and Concepts for Outcomes in Child Welfare

(Trainer's Slides PowerPoint [PPT] Slide Show)

Slide 1

Supervisor Core Training: Managing for Results

Original presentation was created for Version 1.0 by Daniel Webster, Barbara Needell, Wendy Piccus, Aron Shlonsky, & Lynn Usher.

Revised for Version 1.1 by Shared Vision Consultants and CalSWEC to include outcome updates and slides available from presentations posted on the CSSR website.

Data-Informed Practice—What do I Care?

The child welfare system has traditionally not done a very good job of monitoring and evaluating its performance.


2

Background and Concepts for Outcomes in Child Welfare:Data Informed Practice -- What do I Care?

Part 1:

Slide 3The Current Placement System*

(Highly Simplified)

Child In Child OutA bunch of stuff happens

*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference

Current Placement System—Highly Simplified

Child welfare policy and practices need to be guided by clear and specific goals, and progress toward those goals requires good performance data. Yet—in spite of the large quantity of information often collected about children in out-of-home care—child welfare agencies are often unable to provide quick and reliable responses to questions posed by policymakers, administrators, and the public.

Slide 4 Data Skepticism

ACTIVITY: Ask trainees to call out reactions they have to the idea of using data in their work, reasons that they may not like data, etc. Use flip chart to record answers.

Sentiment among child welfare agency staff is that, while they may be inundated by data, this information is not useful for planning, evaluation, or other decisions. Their experience has often been that data systems used by the agency were designed for other purposes and cannot provide answers to questions that are pertinent to their practice. Further, they likely feel that entering data into CWS/CMS is a waste of time and takes away valuable time that they could be doing actual casework. This attitude is largely due to a legacy of state data systems like CWS/CMS as being “roach motels” of data—workers enter information in, but virtually nothing ever comes out. Or, when data does come out, it is not


3

(continued) presented in a way that is easily accessible or useful.

A first step, then, in overcoming this aversion to data is to make the connection for workers between the information they input into CWS/CMS about their cases, and how successful their agency is helping its clients. To do so requires analyzing data that has been put into CWS/CMS and sharing it—on an ongoing basis—in a format that workers can understand. When workers hear the word “data,” they immediately think of “MATH” or “STATISTICS” and can be intimidated or turned off. A common reaction is “I hate math! I became a social worker so that I could help people—not work with numbers!” Another likely reaction is, “I've been a social worker for 25 years, and I know very well what area most needs work in our agency—it’s placement disruptions (or reunifying kids more quickly, or cutting down on the number of kids reentering care, etc.).” These typical responses are not unreasonable—given that workers have probably never seen examples of how data can be useful.

If supervisors can be shown that information drawn regularly from CWS/CMS is a form of feedback on the impact of their efforts (as evidenced by performance on important goals such as preventing recurrence of maltreatment, or achieving reunification for children, etc.), their distrust of the usefulness of data will decrease. It is important to recognize that each worker’s practice experience provides a very real and valid perspective of their agency’s areas of success and failure. This validation can be used to point out to supervisors that each of them is actually a data person whether they realize it or not. That is, their experience is their data source from which they draw conclusions and make decisions about what areas have greater or lesser importance. Extrapolating from each worker's experience with their specific caseload to the information on all workers’ caseloads that is captured in CWS/CMS, provides a more comprehensive picture of where


4

(continued) a child welfare agency needs to focus its efforts.

Clearly, improvements could be made in all parts of the child welfare system—but limited time and resources demand that efforts be made in the areas of greatest need. As a supervisor, it is critical to know where your attention would best be focused—and to keep on top of trends because the area of focus changes over time. In the ER unit—are referrals and/or substantiations going up or down? For which age or ethnic groups? Are these trends reflected in entries to care? In the permanency unit—are reunifications increasing or not? If they are, is our re-entry rate the same or are many of the kids we send home later coming back into care? Drawing on information in CWS/CMS will help supervisors to identify these trends and keep track of progress toward addressing them. Regularly updating staff in unit meetings—and celebrating any progress or success—can positively reinforce and motivate staff.

Fancy statistics are not necessary. Simple line or pie charts can easily point out those areas where the agency is succeeding best and where it most needs to focus its efforts to help families. Supervisors will begin to understand that access to timely data and the ability to analyze this information can help agency staff to identify and respond to changing population needs—and enable workers to monitor the results of their work. Overcoming skepticism regarding the usefulness of data is therefore a crucial first step toward better outcomes—and thus for improved lives of children and families.

Ask for ways that might make using data and outcomes measures more accessible for supervisors and workers. Record suggestions on flip chart. If training staff from only one or two counties, ask them to tell you general trends in child welfare in their county (e.g., how many children enter care each year, how long do children generally stay in care, what proportion of children reunify after 12 months in care, etc.).


5

Record these answers and check them against actual trends for the counties during the live demo of the UC Berkeley website. Then underscore the necessity to make these efforts to be aware of key trends, keep staff informed about areas of need, and to track progress, because…monitoring child welfare agency performance is no longer simply a good idea—it's the law.

Slide 5

The California Child Welfare Outcomes & Accountability System

• Quarterly distribution of county specific outcome indicators data

• Peer Quality Case Review• County Self Assessment• County Self Improvement Plan• Continuous monitoring of outcomes

California’s Child Welfare Outcomes & Accountability System

The Children and Family Services Review (CFSR) is an ongoing nationwide process conducted by the U.S. Department of Health and Human Services. The first stage of the CFSR examined service delivery by child protection, foster care, adoption, family preservation, family support, and independent living programs in all states.

The CFSR examines seven outcomes for children and families as well as seven systemic factors. States were measured against national standards. During the first round of the CFSR, no state achieved substantial conformity on all measures. Thus, all states are implementing a Program Improvement Plan (PIP). The CFSR reviews all states periodically to follow progress and may impose fiscal penalties on those states that fail to improve.

In 2003, California did not achieve conformity on any of the seven outcomes, and passed only one of seven systemic factors. The state faced $18 million in penalties if it had failed to meet federal requirements by 2005. In January 2008, the federal government levied a nearly $9 million penalty against the state for failing to meet the outcome goals regarding re-entry and placement stability. Loss of this funding during times of fiscal hardship will likely result in loss of services to children and families.

Assembly Bill 636, the Child Welfare Outcomes and Accountability Act of 2001, was developed


6

(continued)

The California Child Welfare Outcomes & Accountability System

• Quarterly distribution of county specific outcome indicators data

• Peer Quality Case Review• County Self Assessment• County Self Improvement Plan• Continuous monitoring of outcomes

as a means to carry out California's PIP. The bill provides a legislative framework for monitoring and assessing county performance to ensure the safety, health and well-being of the state’s children. Since the passing of this legislation, California was instrumental in influencing the federal government to review and subsequently revise their federal outcome measures in 2007. Although many of the revised federal measures mirror the previously enhanced California measures, there are still a few vital outcome areas that California chooses to track in the absence of federal mandates. Those measures include the “Timely Response” of social workers regarding Immediate and 10-day referrals and “Timely Social Worker Visits.” Once again, California is leading the way in the creation of well-being measures for children who are in foster care. As of June 2009, the measures for health, dental, and educational outcomes for children in care were under development.

Each county has created a self-assessment that identifies strengths and areas for improvement, as well as a system improvement plan (SIP) to outline actions needed to improve county performance with respect to the outcomes.

Key Concepts to Understand Child Welfare Outcomes

Before examining the specific outcomes tracked under The California Child Welfare Outcomes and Accountability System, it is important to know the background about the reasons (i.e., sound methodological basis) why many of the measures were chosen.


7

Trainee Handout # 1

Caseload Snapshots Versus Entry Cohorts

J an. 1, 2008

J an. 1, 2009J an. 1, 2007

Longitudinal Data Analysis—The Advantage of Using Entry Cohorts

Caseload Snapshots Versus Entry Cohorts1

(Handout #1)

A limitation of traditional child welfare research is that the data used are often comprised of periodic snapshots of the caseload of children in care at a given point in time. This information is important for providing basic management accountability; however, it does not accurately capture the experience of all children who come in contact with the child welfare system, and can be misleading when examining issues such as length of stay or placement moves in care.

Point in time estimates are biased. They over-represent children who have the worst experiences in care—that is, those who tend to stay in care for long periods. Slide #6 illustrates this phenomenon: the x-axis represents a timeline from January 1, 2007, up to January 1, 2009, and the 10 numbered horizontal lines represent the duration in care of 10 children in foster care. A point in time snapshot taken of the caseload on January 1, 2008, will only observe about half the children (line numbers 1, 2, 3, 5, 8 and 10 from the top). The other children will not be detected by this snapshot. Therefore, any mean or median measure of length of time spent in care will be biased by the children observed in the snapshot who have longer lengths of stay. Placement stability estimates would be similarly biased. Using snapshot data, staff may not recognize the opportunity to effectively focus resources by assuming that most children in the system are poorly served.

1 This discussion adapted from: Webster, D., Needell, B., & Wildfire, J. (2002). Data are your friends: Child welfare agency self-evaluation in Los Angeles County with the Family to Family initiative. Children and Youth Services Review, 24(6/7), 471-484.


8

(continued)Caseload Snapshots Versus Entry Cohorts

J an. 1, 2008

J an. 1, 2009J an. 1, 2007

ACTIVITY: Referring to PPT slide #6, ask trainees to call out how many children would comprise the calendar year 2007 entry cohort (Answer: 7). Then ask trainees to call out how many children would comprise the calendar year 2008 entry cohort (Answer: 3).

Fun Exercise with Medians and Means

“How long do children stay in care in your county?” is an important question that people often ask. The following brief exercise illustrates the need to reflect on the approach taken to answer the question about length of stay.

ACTIVITY: Prior to class, trainer cuts out rows (make more as needed) from the following table and distributes one row each to student at the beginning of this exercise. (The trainee given the “Bill Gates” row is asked to keep his/her identity secret).

WORK TITLE INCOMEUCB Ph.D., Consultant Roughly minimum wage

(let’s say, $10 K)Administrative Assistant $20 K

Administrative Assistant $20 K

Social Worker I $40 K

Social Worker I $40 K

Social Worker II $50 K

Social Worker II $50 K

Supervisor $60 K

Supervisor $60 K

Branch Manager $75 K

Branch Manager $75 K

Bill Gates, CEO Microsoft

$3 Billion

Ask students (except for Bill Gates) to state their


9

(continued)Caseload Snapshots Versus Entry Cohorts

J an. 1, 2008

J an. 1, 2009J an. 1, 2007

work title and income, and then to line themselves up from highest to lowest income. The instructor points out that the line up is another way to talk about a “DISTRIBUTION” and asks the class to identify the MEDIAN income from the lineup. The class will see that the median income is $50 thousand.

The instructor then asks Bill Gates to identify himself and to join the lineup, and asks the class to identify the median income now that a new person has been added. The class will see that the median income will still be $50 thousand. Thus the students will see that that the median income is a reasonable description of the middle point of the incomes—even with an extreme observation.

Finally, the instructor will ask Bill Gates to step away, and for the class to identify the MEAN income from the lineup minus Bill Gates. The students will find that that the average or mean income is about $46 thousand. When Bill Gates rejoins the group and the class is asked to re-compute the mean income, the average income will be over $250 million—a poor description of the individuals’ incomes (i.e., much too high for everyone except Bill Gates, for whom the description is much too low).

The instructor can use the exercise to point out how medians are a much more reasonable concept to use when talking about measures (particularly length of stay) that may contain extreme, “outlier” data points.

The Cycle of Experiences in the Child Welfare System

As a last word before the break, it is important to underscore the interrelated nature of interpreting outcome data. This is a main reason why the federal “standards” are inherently misleading—i.e., if one “achieves conformity” on a given measure, that is considered success. However this “success” may adversely affect other outcomes and therefore cannot be viewed in isolation.

Slide 7 The Cycle of Experiences


10

Trainee Handout # 2The Cycle of Experiences in the Child Welfare SysThe Cycle of Experiences in the Child Welfare Systemtem

CounterbalancedCounterbalancedIndicators ofIndicators of

SystemSystemPerformancePerformance

PermanencyPermanencyThroughThrough

Reunification,Reunification,Adoption, orAdoption, or

GuardianshipGuardianship

ShorterShorterLengthsLengthsOf StayOf Stay

StabilityStabilityOf CareOf Care

Rate of Referrals/Rate of Referrals/Substantiated ReferralsSubstantiated Referrals HomeHome--BasedBased

Services vs.Services vs.OutOut--ofof--HomeHome

CareCare

Maintain Positive Maintain Positive AttachmentsAttachmentsTo Family,To Family,

Friends, andFriends, andNeighborsNeighbors

Use of LeastUse of LeastRestrictiveRestrictive

Form of CareForm of Care

Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families,

Reentry to CareReentry to Care

Slide 7 (continued)

(Handout #2)

There is no “gold standard” for determining success on any outcome. Any particular measure, viewed in isolation, tells you nothing useful about county performance.

Data tells a story, and the relationship between outcomes must be taken into account when evaluating performance. For example, while we may seek to reduce the number of children entering care—if/when we succeed in doing so, we must be aware of how the smaller population entering care will likely be more difficult to work with (i.e., we have excluded more families from oversight by the agency—but those families that do require intervention have more challenging circumstances). Thus a reduction in entries may affect other measures such as time to permanency or placement stability.

The supervisor needs to understand, therefore, that monitoring and drawing conclusions about any given outcome must be done with respect to the relationships between outcomes. (Other potential relationships—reduction in entries could lead to higher recurrence of maltreatment; increase in reunification could lead to increase in reentry; increase in guardianship or adoption could lead to decrease in reunification.) Also, other factors outside of child welfare, such as an economic downturn, etc., may lead to an increase in entries even in spite of best practice.

By being aware of the interrelated nature of outcomes, the supervisor can make a more realistic assessment of the effect their work is having on their clients. They will not be overly pessimistic, for example, when the number of children reunified during a period of time drops—if they are also monitoring entries to care and note that a smaller number of children with greater needs is coming into care.

By the same token, they will not prematurely think


11

The Cycle of Experiences in the Child Welfare SysThe Cycle of Experiences in the Child Welfare Systemtem










CareCare







Slide 7 (continued)

that they don’t need to continue efforts to improve reunification services just because an increasing number of children are going home from the agency. They will also know to also track the proportion of those reunifying who later return to care—to ensure that the apparent, successful reunification events are not being followed by an unintended negative outcome (re-entry).

Finally, the supervisor will be able to use this understanding of relationships between the outcomes to more clearly, forcefully, and accurately “tell the story” of how well children and families are being helped—whether the supervisor is motivating staff in a unit meeting, communicating needs for practice change to a child welfare director, or being “accountable” to relevant members of the community (e.g., judges, child advocates, legislators, board of supervisors). A couple of initiatives in California that focus on using outcome data:

Family to Family, an initiative developed by the Annie E. Casey Foundation and also supported by the Stuart Foundation, is being implemented in 22 counties that represent 84 % of the children in foster care. The four core components are: community partnerships; recruitment, training and support of foster parents; team decision-making; and self-evaluation. Self-evaluation, the use of data, allows community partners to focus on neighborhoods with the highest concentration of CPS referrals and entries into foster care. Using Family to Family strategies, children who must be removed from the home can be placed with foster families in their neighborhoods. http://www.aecf.org/initiatives/familytofamily.

California Permanency for Youth Project, a three-year program supported by the Stuart Foundation and the Walter S. Johnson Foundation, is dedicated to ensuring that all youth leave the child welfare system with a permanent lifelong connection with a caring adult. The project collaborates with the courts, group homes, foster families, adoption agencies and the child welfare system to help them achieve permanency for foster youth in Alameda, Monterey, San Mateo and Stanislaus counties. The project also


12

http://www.aecf.org/initiatives/familytofamily

The Cycle of Experiences in the Child Welfare SysThe Cycle of Experiences in the Child Welfare Systemtem










CareCare







works to improve public policy, increase training and raise public awareness on the issue. It tracks its success by utilizing data on older youth in foster care and on exits from care. http://www.cpyp.org.

Summarize points supporting why data is important for supervisors:

It is critical to know where your limited time would best be focused—information on all workers’ caseloads captured in CWS/CMS provides a more comprehensive picture of where a child welfare agency is performing well and where it needs improvement.

Regularly updating staff in unit meetings—and celebrating any progress or success—can positively reinforce and motivate workers.

Regular use of data and understanding relationships between the outcomes “tells the story” of how well children and families are being helped—whether supervisors are motivating staff in a unit meeting, communicating needs for policy and practice change to a child welfare director, or being “accountable” to relevant members of the community (e.g., judges, child advocates, legislators, board of supervisors).

Break


13

http://www.cpyp.org/

Using Outcome Data in Child Welfare Practice

Part 2:

PART 2

4. Using Outcome Data in Child Welfare Practice

This part of the seminar will introduce the specific outcome measures tracked under the California Outcomes and Accountability System, the quarterly data report, and the UC Berkeley website. The UC Berkeley website demonstration is a “live” online session that requires a high-speed Internet connection.

The California Quarterly Outcomes and Accountability County Data Report

What is it, and what’s on it?

The California Outcomes and Accountability quarterly report provides summary-level federal and state outcome measures that are intended to be the basis for ongoing county self-assessment of its performance over time. Each county is expected to use data in the report for the self-assessment process—that is, to determine reasons for current performance as well as to help plan for measurable improvement.

The California Outcomes and Accountability quarterly report contains information on the following areas: CWS participation rates along with the three federal child welfare “domains” of safety, permanency, and well-being. A condensed version of the report layout is included in trainees’ handout packets. The most recent version of the complete quarterly report for all California counties and the state as a whole is available at http://cssr.berkeley.edu/ucb_childwelfare/ccfsr.aspx

California Quarterly Outcomes & Accountability County Data Report (Trainee Handout #3)

Child Welfare Services Participation Rates: provides data on the number, and number per 1,000 children in the county/state, for key child welfare indicators. It is intended as background information to assist your county in analyzing your county’s performance by the outcome indicators.


14

http://cssr.berkeley.edu/ucb_childwelfare/ccfsr.aspx

Safety Outcomes: designed to reflect the effectiveness of efforts to protect children from abuse/neglect by reporting instances of abuse and neglect at various stages of child welfare services and process, measures which reflect the frequency of social worker contact with children and the speed of face-to-face investigation of abuse/neglect allegations.

Permanency Outcomes: designed to reflect the number of foster care placements for each child, the length of time a child is in foster care, and the rate that children re-enter foster care after they have returned home or other permanent care arrangements have been made.

Child and Family Well-Being Outcomes: designed to reflect the degree to which children in foster care retain relationships with the family and extended communities with whom they were associated at the time of their removal from their parents.

The county data report is produced on a quarterly basis—that is, every three months. Each quarter, a data extract from CWS/CMS is queried to create the reports. Counties are allowed a two-month window after the close of each quarter to comply with CWS/CMS data entry requirements. Thus, taking the 2007 Quarter 2 extract from CWS/CMS as an example, information—about any activity within a child’s case history that occurred prior to June 30, 2007—could be entered into CWS/CMS up until August 31, 2007, and still be included in the 2007 Quarter 2 data extract.

Slide 9Federal Versus State-Enhanced Measures

Is Performance getting better or worse?

Adoption within 24 Months

Data Source: Multi-State Data Archive, Chapin Hall Center for Children

State A

0%

5%

10%

15%

20%

25%

30%

Ado

pted

ExitsEntries

State A

0%

5%

10%

15%

20%

25%

30%

Ado

pted

ExitsEntries

Federal Versus State-Enhanced Measures

Unlike point-in-time counts that contain information on children who are still in care at a given point in time, or exit cohorts that examine children leaving care and are potentially confounding by mixing long- and short-stayers in care, longitudinal data systems capture information on all children who enter foster care and follow them for the entire time they are in care. Information on a cohort of children as they move through out-of-home care does not systematically exclude children from the study population if they leave care.

As Fred Wulczyn of the Chapin Hall Center for Children has observed, “if you were trying to evaluate a cancer therapy or HIV treatment—would you look only at patients who survived (or only at those who died) to determine the effectiveness of the intervention—or would you follow all patients from the time the treatment


15

started?” The answer is obvious.

Reliable information on children’s child welfare experiences requires following children from first entry to care through placements within and exits from the system, and, for some children, subsequent re-entries to care. Supervisors, workers, and administrators need to evaluate performance and base decisions on the most comprehensive and accurate information possible. The AB 636 “state enhanced” measures, therefore, use entry cohort analysis to supplement the federal standards and provide this information. Many of the new measures adopted in 2006 by the Administration for Children and Families (ACF) were part of California’s enhanced measures adopted through the AB 636 legislation. As mentioned above, California continues to monitor and track data regarding social worker timely response and monthly visits to children even though there is not a mandate to do so. Other measures enhanced by California include the well-being measures regarding sibling placements, least restrictive placements, ICWA/cultural considerations and the post foster care outcomes.

What the ¡ ! is a "Rolling Year" ? 2

Use graphics on PowerPoint slides to explain this topic. Use flip chart if further explanation is needed.

2 Part of this discussion based on: Hurley, D. (2004). Completing the county self-assessment, Module 3: Analyzing child welfare outcome indicators. California Social Work Education Center: Berkeley, California.


16

2008 2009

Jan. 1

Mar. 1 Mar. 1

Dec. 31

Jul. 7 Jul. 7

Jan. 1 Dec. 31

Dec. 31 Dec. 31

Tracking an Entry Cohort for 1 Year

Slide 11

Rolling Years:Quarter 1 Extract

Jan. 1

2008 2009

Dec. 31Mar. 31 Mar. 31

Jan. 1 Dec. 31Jan. 1

2007

Extract Cut-off No data after

April 1

Last day of data prior to cut-offMarch 31, 2009

ENTRY COHORT

Apr. 1

April 1, 2007 – March 31, 2008

Last date able to track entry for full 12 monthsprior to cut-off.

(Trainee Handout #4)Slide 12


Jan. 1

2008 2009

Dec. 31Jun. 30 Jun. 30


2007


July 1

Last day of data prior to cut-offJune 30, 2009

ENTRY COHORT

Jul. 1

July 1, 2007 -- June 30, 2008


Rolling Years

Each quarter, most measures are available using the most recent data available for that measure using “ROLLING YEARS.” A rolling year examines an outcome for children in a year-long cohort, but the year being tracked will not always be equal to the January–December calendar year. A rolling year can be April 1 (of one year) to March 31 (of the following year); other rolling year periods can be July 1(of one year) to June 30 (of the following year), or October 1 to September 30. The January 1 to December 31 calendar year is also a rolling year.

There are 4 Rolling Year Time Periods (corresponding with the 4 quarterly data extracts from CWS/CMS):

Q1: April 1–March 31

(extract cut-off: April 1st)

Q2: July 1–June 30 (extract cut-off: July 1st)

Q3: October 1-September 30

(extract cut-off: October 1st)

Q4: January 1–December 31

(extract cut-off: January 1st)

The rolling year used depends on the outcome being examined. For example, using 2009 Quarter 2 data, UC Berkeley can look at first entries to care for children who enter July 1, 2008–June 30, 2009, reunifications within 12 months for children who entered July 1, 2007–June 30, 2008, and adoptions within 24 months for children who entered July 1,


17


Jan. 1

2008 2009

Dec. 31Mar. 31 Mar. 31


2007


April 1

Last day of data prior to cut-offMarch 31, 2009

ENTRY COHORT

Apr. 1

April 1, 2007 – March 31, 2008


Slide 12


Jan. 1

2008 2009

Dec. 31Jun. 30 Jun. 30


2007


July 1

Last day of data prior to cut-offJune 30, 2009

ENTRY COHORT

Jul. 1

July 1, 2007 -- June 30, 2008


2006–June 30, 2007.

As new data become available each quarter with the most recent extract from CMS, these measures can be “rolled” forward to track a 365-day cohort up to the most recent study cut-off date. For example, for 2009 Quarter 3, information on October 1, 2008, to September 30, 2009, will be available. This gives users of the website the ability to monitor trends over time without the problems associated with quarterly measures (e.g., seasonal effects, very little data in small counties).

Pop Quiz on Rolling Years(Trainee Handout #4)

ACTIVITY: Pop Quiz. Ask trainees to take 3 minutes to answer the questions (listed below) on Handout #4. After 3 minutes, ask trainees to call out correct answers to questions. Review rolling years concept if necessary.

Pop Quiz on Rolling Years:When the 2009 Quarter 3 data are available (i.e., last day of data: September 30, 2009), the next progressive quarter’s worth of information can be added in tracking outcome measures.

Question 1: In the 2009 Q3 data extract, what is the most recent time period used to measure first entries to care?

Answer: The time period is October 1, 2008, to September 30, 2009.

Question 2: In the 2009 Q3 data extract, what is the most recent time period used to measure the percent of children reunified within 12 months?

Answer: The time period is October 1, 2007, to September 30, 2008, since this is the most recent cohort of children for whom each can be followed for


18

an entire 12 months to end up in the September 30, 2009, data extract.

Question 3: In the 2009 Q3 data extract, what is the most recent time period used to measure the percent of children adopted within 24 months?

Answer: The time period is October 1, 2006, to September 30, 2007, since this is the most recent cohort of children for whom each can be followed for an entire 24 months to end up in the September 30, 2009, data extract.

5. Taking Your First Steps Toward Data Analysis

UC Berkeley Dynamic Website Introduction

The introduction of the UC Berkeley Child Welfare Dynamic Report System Website requires a high-speed Internet connection. The trainer must be very familiar with the layout and information available on the UC Berkeley Child Welfare Dynamic Report System Website. Screen shots are provided in the Trainee’s PowerPoint slides (slide numbers are indicated below) strictly as points of reference for organizing the training.

Slide 13Child Welfare Dynamic Web Site

UC Berkeley Child Welfare Dynamic Report System Website

All the data on the California Outcomes quarterly report are posted on the websites of the Center for Social Services Research at University of California, Berkeley (http://cssr.berkeley.edu/ucb_childwelfare/), or on the California Department of Social Services site (http://www.dss.cahwnet.gov/research/). In addition to the summary-level numbers provided on the county data reports, data on the UC Berkeley Child Welfare Dynamic Report System website include much more detailed information for each of the California Outcomes and Accountability measures (i.e., measures can be examined by age group, ethnicity, gender, and, in most cases, placement type) as well as for other outcomes. Be sure to click on and read the methodology link for any outcome analysis that you are examining. The methodology description provides important detail


19

http://www.dss.cahwnet.gov/research/

http://cssr.berkeley.edu/ucb_childwelfare/

CWS Quarterly

Outcomes Report

Slide 15

National Standards• In California, the Center for Social Services

Research attempts to replicate each of the measures and composite scores, break them out by child welfare and probation agencies, and report/update quarterly.

• The goal is to improve State performance on all measures (every improvement reflects a better outcome for children)

Source: Center for Social Services Research, UC Berkeleyhttp://cssr.berkeley.edu/ucb_childwelfare

about how the analysis was conducted.

The quarterly data report includes a number of tabbed pages which allow the user to compare the latest outcome data with an established baseline or another period of time. The report also provides a unique view of the composite measures. The composite viewer tab shows the outcome performance relative to the national standard. The goal is to improve state performance on all measures.

There is also a tab which details the methodology for each measure and provides a simple explanation for each outcome indicator. There are also tabs that show the past dates for the retrieval of past outcome data. Finally, there are graphs that depict the outcome performance over time at a one-shot glance.

Other important features to explore on the UC Berkeley Child Welfare Dynamic Report System website:

All Available Outcomes by Specified County —provides all reports available on the UC Berkeley Report System website pertaining to a county that a user specifies. Select your county from a drop-down menu for county-specific information. With the dynamic site you can also choose different variables to review over time. http://cssr.berkeley.edu/ucb_childwelfare/

Data Highlights —are one-page documents about statewide trends and developments in child welfare as interpreted from CWS/CMS data. A useful exercise would be to substitute your own county data for the statewide data. An index of available highlights is available at http://cssr.berkeley.edu/CWSCMSreports/Highlights/

Composite Viewer —shows the outcome


20

http://cssr.berkeley.edu/CWSCMSreports/Highlights/

http://cssr.berkeley.edu/CWSCMSreports/Highlights/

http://cssr.berkeley.edu/ucb_childwelfare/

Measure Contributions to

Composites

C1.1 (22%)

C1.2 (21%)

C1.3 (12%)

C1.4 (46%)

0%

20%

40%

60%

80%

100%

Composite 1

Reunification Within 12 Months (Exit Cohort)

Median Time To Reunification (Exit Cohort)

Reentry Following Reunification (Exit Cohort)

Reunification Within 12 Months (Entry Cohort)

Note: Measures may not sum to exactly 100% due to rounding.Source: Center for Social Services Research, UC Berkeley

http:/ / cssr.berkeley.edu/ucb_childwelfare

Slide 17Measure Contributions to

Composites

C2.1 (15%)

C2.2 (19%)

C2.3 (22%)

C2.4 (18%)

C2.5 (26%)

0%

20%

40%

60%

80%

100%

Composite 2

Adoption Within 24 Months (Exit Cohort)

Median Time To Adoption (Exit Cohort)

Adoption Within 12 Months (17 Months In Care)

Legally Free Within 6 Months (17 Months In Care)

Adoption Within 12 Months (Legally Free)




Composites

C3.1 (33%)

C3.2 (25%)

C3.3 (42%)

0%

20%

40%

60%

80%

100%

Composite 3

Exits to Permanency (24 Months In Care)

Exits to Permanency (Legally Free At Exit)

In Care 3 Years Or Longer (Emancipated/Age 18)




Composites

C4.1 (33%)

C4.2 (34%)

C4.3 (33%)

0%

20%

40%

60%

80%

100%

Composite 4

Placement Stability (8 Days To 12 Months I n Care)

Placement Stability (12 To 24 Months I n Care)

Placement Stability (At Least 24 Months I n Care)

Note: Measures may not sum to exactly 100% due to rounding.Source: Center f or Social Services Research, UC Berkeley

http:/ / cssr.berkeley.edu/ ucb_childwelfare

performance relative to the national standard for:

Safety Measures:

S1_1: No Recurrence of MaltreatmentS2_1: No Maltreatment in Foster Care

and Permanency Composites:

Composite 1: Reunification CompositeComposite 2: Adoption CompositeComposite 3: Long-term Care CompositeComposite 4: Placement Stability

Composite

The series of slides #16 through #20 depict how the individual measures of each permanency composite are weighted. These slides correspond to the Trainee Supplement New Federal Measures Listed with National Standards. For trainees who have great interest in this subject, the Trainee Supplement Federal Outcome Descriptors describes the computations for each outcome measure, as well as the rates for referrals, substantiations, entries, and ‘in care’, which are also found on the CSSR website.

Slide #21 shows the overall structure for the relationships among measures, components, composites, and federal goals.

Mailing List Link —for email updates from the UC Berkeley site. You will be notified of changes, updates, and additions to the site. To sign up for email updates go to: http://cssr.berkeley.edu/ucb_childwelfare/mailinfo.aspx

Presentations —Presentations by UC Berkeley child welfare researchers are available on the website and can provide supplemental information for understanding and reading data found on the website. http://cssr.berkeley.edu/cwscmsreports/presentat


21

http://cssr.berkeley.edu/cwscmsreports/presentations/

http://cssr.berkeley.edu/ucb_childwelfare/mailinfo.aspx

http://cssr.berkeley.edu/ucb_childwelfare/mailinfo.aspx

Measure Contributions to

Composites

Note: Measures may not sum to exactly 100% due to rounding.Source: Center f or Social Services Research, UC Berkeley

http:/ / cssr.berkeley.edu/ ucb_childwelfare

C4.1 (33%)C3.1 (33%)C2.1 (15%)C1.1 (22%)

C4.2 (34%)C3.2 (25%)

C2.2 (19%)C1.2 (21%)

C4.3 (33%)C3.3 (42%)

C2.3 (22%)C1.3 (12%)

C2.4 (18%)

C1.4 (46%)C2.5 (26%)

0%

20%

40%

60%

80%

100%

Composite 1 Composite 2 Composite 3 Composite 4

Slide 21

SafetyI ndicator 1

S1.1

I ndicator 2 S2.1

Permanency

Composite 1:Reunification

Composite 2: Adoption

Composite 3: Long- Term

Composite 4: Stability

Component A Component B

Component A Component B Component C

Component A Component B

Measure 1 Measure 2 Measure 3 Measure 4

Measure 1 Measure 2 Measure 3 Measure 4 Measure 5

Measure 1 Measure 2 Measure 3

Measure 1 Measure 2 Measure 3

ions/

Static Site—Information is still available on the static website, although the new dynamic website is more user-friendly. http://cssr.berkeley.edu/cwscmsreports/defaultStatic.asp

In conjunction with explaining the CWS/CMS dynamic website, county-specific data tools should also be discussed. For example, Safe Measures reports or Business Objects reports are also useful tools for monitoring and assessing performance outcomes. When all tools are used in conjunction, supervisors can make the link between the macro picture of the federal outcome measures and standards and state, county, unit and worker outcomes and performance. It may be helpful to select one outcome area to demonstrate. For example, a supervisor could compare “visitation with parents” with “reunification within 12 months” in their unit and in their county. The data could assist the supervisor to address visitation and reunification issues with staff.

Break

PART 3

6. Outcome-Based Management—A Hands-On ExercisePeople learn best by doing. This final portion of the seminar will give trainees the opportunity to apply some of information covered in Parts 1 & 2 and to begin to grapple with looking at outcome data, interpreting results, and formulating plans to positively affect targeted outcomes. About 45 minutes should be spent working with the data; and the remaining 15 minutes can be used for reporting back.

Outcomes Based Management—A Hands-on


22

http://cssr.berkeley.edu/cwscmsreports/defaultStatic.asp

http://cssr.berkeley.edu/cwscmsreports/defaultStatic.asp

http://cssr.berkeley.edu/cwscmsreports/presentations/

Outcomes Based Management—a Hands-on Exercise

Part 3:

The following slides are provided courtesy of:Center for Social Services ResearchUniversity of California, Berkeleyhttp://cssr.berkeley.edu/ucb_childwelfare

Slide 23

Exercise(Handouts #5-16)

The trainer should break the trainees into groups of 5–7 people. Each group will work together on interpreting the outcome data in their handout packet. The trainer should circulate between the groups to answer questions and help keep them on task.

After forming the different groups, the trainer should underscore suggestions on what to think about as they review the data (these are listed on Handout #5). Not all the following points are applicable to this in-class exercise, but should be borne in mind whenever they work with outcome information in their county.

Things to keep in mind as you examine outcome data 3 :

Begin your analysis by determining which outcomes (and possibly for which subgroups) are “high priority” and “high performance” areas within your county.

Examine trends in the outcomes over time (e.g., has the number and rate of entries changed over the past several years? Has the proportion of children reunified within 12 months changed?).

Also remember that sometimes data may not answer “why” in relation to practice questions. For example, in the first round of the CSFR California failed to meet the required standards in the PIP regarding placement stability, among other outcomes. The data was able to show there weren’t improved outcomes in those areas, however it did not tell the story of “why” that was so. Data is needed to help focus the questions and discussion to drill-down to the “why”. In the case of the first round CSFR, the

3 Some of these points adapted from: Hurley, D. (2004). Completing the county self-assessment, Module 3: Analyzing child welfare outcome indicators. California Social Work Education Center: Berkeley, California.


23

Trainee Handout #6California:

Ethnicity and the Path through the Child Welfare System 2007(Missing Values & Other Race Excluded from % Calculations,18 years of Age)

6.214.9 14.6 18.6

26.3 19.9

32.728.5 27.2 26.2

25.227.7

50.452.9 48.0

10.3 3.0

44.751.8 50.2

2.43.9 4.1 3.50.5 0.8 1.2 1.6 1.4 1.3

0%

20%

40%

60%

80%

100%

Population(10,007,501)

Allegations(492,764)

Substantiations(107,524)

Entries (36,077)

I n Care (72,221)

Exits* (36,190)

NativeAmerican

Asian/PI

Hispanic

White

Black

Source: Center for Social Services Research, UC Berkeleyhttp://cssr.berkeley.edu/ucb_childwelfare

Slide 24Trainee Handout #7

July 1, 2008California:

Race/Ethnicity by Caseload Case Service %

Components (Missing Values & Other Race Excluded from Calculations, Children of all Ages Included)

44.558.5

45.5 38.2 31.046.5

23.9

19.424.6

25.723.0

30.2

11.4 11.18.4

16.7 10.6

13.4

8.0

11.3

19.315.211.330.7

4.34.03.43.52.93.4

0%

20%

40%

60%

80%

100%

Total (104,698)

Black (24,367)

White (25,248)

Hispanic (50,007)

Asian/ PI (3,408)

Native American(1,455)

Permanent Placement Family ReunificationFamily Maintenance Post-Placement Family Maintenance Pre-PlacementEmergency Response


<1 yr 1-2 yrs

3-5 yrs 6-10 yrs

11-15 yrs16-17 yrs

Black (113.2*) ALL

(49.2*) White (41.1*) Hispanic

(48.5*) Asian/PI(17.9*)

171

123130

121

102

8166

49505247

41

54

4042

4340

34

60

4749

5246

41

1815

1920

1815

2007California:

Referrals per 1,000 by Age and Ethnicity

*Ser ies Total

Slide 26

data outcomes have generated new questions.

Analyze age and especially ethnic disparities (slides #23–28/Trainee Handouts 6–11).

Look at differences between placement types (particularly kinship versus non-kinship care) in slide #29/Trainee Handout 12.

Examine the differences in exits to permanency (slide #30/ Trainee Handout 13).

Comment on the difference between the national goal and the data for Measure C3.3: In Care 3 Years of Longer (slide #31/Trainee Handout 14).

Note the measures for Permanency Composite 4: Placement Stability (slide #32/Trainee Handout 15).

Note the trend towards improvement for the percentage of all siblings placed together as reflected in slide #33/Trainee Handout 16 .

Consider the relationships between outcome measures. Recalling the “cycle of experiences” discussion and handout, a change in one performance measure may have a relationship with other outcomes. For example, diverting children from out-of-home care would decrease entry rates—but the children who still require placement may be more troubled, and less likely to be reunified, affecting reunification rates.

Identify factors that may be influencing performance. Some examples are:

Management information system issues (e.g., data cleanup, data entry, or interpretation issues that may have influenced performance measures.

Staff and provider training needs Case review system Service array and agency collaborations Recruitment, licensing and retention of

foster and adoptive parents


24

Trainee Handout #9

<1 yr 1-2 yrs

3-5 yrs 6-10 yrs

11-15 yrs16-17 yrs

Black (25.4*) ALL


(11.3*) Asian/PI(4.4*)

71

3330

25

20

13

25

1311

119

7

23

1110

97

5

24

1312

11

97

74

55

43

2007California:

Substantiated Referrals per 1,000 by Age and Ethnicity

*Ser ies Total


<1 yr 1-2 yrs

3-5 yrs 6-10 yrs

11-15 yrs16-17 yrs

Black (11.2*) ALL


(3.7*) Asian/PI(1.3*)

40

1513

99

6

13

54

33

2

12

43

22

2

12

54

33

2

31

11

11

2007California:

Entries to Foster Care per 1,000 by Age and Ethnicity

*Ser ies Total


<1 yr 1-2 yrs

3-5 yrs 6-10 yrs

11-15 yrs16-17 yrs

Black (31.7*) ALL


(6.7*) Asian/PI(1.7*)

2632

2725

3642

88

6689

77

5567

89

76

67

22

21

22

2007California:

Children in Foster Care per 1,000 by Age and Ethnicity

*Ser ies Total


0

10,000

20,000

30,000

40,000

50,000

60,000

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Point in Time

Place

ment

Fre

quen

cy

1998 to 2008Calif ornia:

Foster Care Caseload by Placement Type

Group/ Shelter

Kinship

FFA

Foster

TOTAL Caseload

68,475

104,300

Slide 30

Quality assurance system

Identify any county policies, regulations, or practices that may contribute to your performance on a given indicator.

Identify linkages between performance outcomes and practice processes.

Think of other outcome information that you may want to have in order to get a more complete picture of performance (e.g., no re-entry data is given in the handouts, but this information would be helpful to know with respect to children reunifying from care).

Conclude analysis by determining areas of strength and those in need of improvement.

Summary Points for Hands-on Exercise Handouts

Black children are referred for maltreatment proportionally more than any other group. Black children make up 6.2% of the total child population, but account for 14.9% of referrals. (Slide #2/Trainee Handout #6).

Once referred, children are equally likely to have substantiation. Across racial groups approximately 22–29% of children who are referred are substantiated.

Black children with open cases are the least likely to be served at home and the most likely to be in long term foster care. Only 8 % of the Black children with open cases are in the Family Maintenance Program, compared to 11.1% of White Children, and 13.4% of Hispanic children. (Slide #24/Trainee Handout 7 , for July 1, 2008)

Black children are the most likely to enter foster care. Black children enter care at a rate of 11.2 per 1,000—compared to 3.0 per 1,000 for White children, and 3.7 per 1,000 for Hispanic Children. (Jan. 1, 2007–Dec. 31, 2007, data)


25

Trainee Handout #132002 to 2008Calif ornia:

C3.1: Exits to Permanency (24m I n Care), by Exit Type

0369

121518212427303336

2002 2003 2004 2005 2006 2007

% Ex

iting

% Reunification

% Guardianship

% Adoption

% Exits to Permanency

TOTAL # in Care 24m+

27,696

44,435


0

10

20

30

40

50

60

70

80

90

100

2002 2003 2004 2005 2006 2007 2008

%


C3.3: I n Care 3 Years Or Longer (Emancipated Or Age 18)

National Goal (37.5%)

% I n Care 3+ Yearsgoal

# Emancipating or Age 18 in Care 4,6244,250



C4.1,2,3: Placement Stability

0

10

20

30

40

50

60

70

80

90

100

2002 2003 2004 2005 2006 2007

% W

ith 2

or F

ewer

Plac

emen

ts

24+ Months In Care

12 to 24 Months In Care

8 Days to 12 Months in Care



Sibling Placements

0

10

20

30

40

50

60

70

80

90

100

2002 2003 2004 2005 2006 2007 2008

% All Siblings Placed Together

Some or All Siblings Placed Together

Black children are the most likely to be in foster care. The rate of Black children in foster care is 29.1 per 1,000—compared to 5.2 per 1,000 for White children, and 6.0 per 1,000 for Hispanic Children. (July 1, 2008, count day)

Black children stay in care longer than children from other racial groups. The median length of stay is 553 days for Black children, 432 days for White children, and 487 days for Hispanic children. (Jan. 1, 2001–Dec. 31, 2006)

7. Report Back and Session Wrap-Up

The trainer should close the session by underscoring that managing for results necessitates creating a flow of information to support mid-course corrections and continuous improvements in outcomes. Some helpful ideas to accomplish this are:


26

Recognize that evaluation is a process, not an isolated event or a particular report.

Regularly monitor progress (e.g., expand management team meetings monthly to include regular updates on outcomes, create specific data/outcomes tracking committee that regularly meets and reports to management team and all agency staff).

Cultivate an office culture where everyone owns the outcomes: Prominently display outcomes throughout the agency Regularly update all staff on progress towards achieving outcomes

Finally, a homework assignment is provided in their Trainee Content (Handout #18). This series of steps is meant to review some of the key points covered in the seminar, and to reinforce the notion that supervisors can take what they have learned back to the office and begin applying it immediately. They can locate/download the latest copy of their county’s Quarterly Outcomes report, and refer to the UC Berkeley Child Welfare Dynamic Report System website for detailed underlying data (e.g., trends and relationships between outcome measures). Through the Dynamic Report System, they can track their agency's successes and areas needing improvement to help children and families more effectively.


27

supervisory core - calswec · web view(trainer's slides powerpoint [ppt] slide show) slide 1...

Documents