tennessee’s iv-e waiver demonstration project …...intervals 82 table 21. exits, average care...

TENNESSEE’S IV-E WAIVER DEMONSTRATION PROJECT

INTERIM EVALUATION REPORT

AUGUST 2017

Chapin Hall at the University of Chicago

TABLES OF CONTENTS

Acknowledgements................................................................................................................7

Introduction and Overview ....................................................................................................8 Background and Context ............................................................................................................. 8 Evaluation of the Waiver Demonstration .................................................................................... 9

Implementation Study ............................................................................................................. 9 Outcomes Study ...................................................................................................................... 9 Cost Study .............................................................................................................................. 10

The evaluation framework ........................................................................................................ 10 Theory of change/logic model ................................................................................................... 13

FAST ....................................................................................................................................... 14 KEEP ....................................................................................................................................... 15 R3 ........................................................................................................................................... 15 Implementation study ........................................................................................................... 15

Content Analysis: Planning and Implementation .............................................................. 15 Structured interviews, Focus Groups, and Online Surveys: Implementation Attitudes .. 16 Fidelity: Worker Time Use ................................................................................................ 16 Administrative data: Fidelity of EBI Implementation ........................................................ 17

Outcomes study ..................................................................................................................... 17 Analytic Approach ............................................................................................................. 17 Intent to Treat Design........................................................................................................ 18

Cost study .............................................................................................................................. 19 Sampling plan ............................................................................................................................ 20 Limitations ................................................................................................................................. 20 Evaluation timeframe and implementation status ................................................................... 20

Implementation Study ......................................................................................................... 22 Data Sources and Data Collection ............................................................................................. 22

Surveys and Questionnaires .................................................................................................. 22 Interview Data ....................................................................................................................... 23 Focus groups .......................................................................................................................... 24 Documentation review .......................................................................................................... 24

Data Analysis ............................................................................................................................. 24 General Implementation ....................................................................................................... 25

General Implementation: Year 1 ...................................................................................... 25 General Implementation: Year 2 ...................................................................................... 25

FAST ....................................................................................................................................... 28 Compliance ........................................................................................................................ 28 FAST “Findings” ................................................................................................................. 35 FAST: Summary ................................................................................................................. 44

KEEP ....................................................................................................................................... 45 Early impressions: Expectations for KEEP ......................................................................... 45 KEEP Implementation ........................................................................................................ 47

R3 ........................................................................................................................................... 47 Early impressions ............................................................................................................... 47 The use of R3 in practice ................................................................................................... 48

Time Use ................................................................................................................................ 49


Hypotheses ........................................................................................................................ 49 Organizing Framework ...................................................................................................... 49 Methods ............................................................................................................................ 51 Time Use Findings .............................................................................................................. 52

The General Staff Survey ....................................................................................................... 55 Participation ...................................................................................................................... 56 Scale Results ...................................................................................................................... 57 Implications of the GSS Findings. ...................................................................................... 60 Repeated Measures Analysis Results ................................................................................ 61 Summary ............................................................................................................................ 61

Outcomes Study .................................................................................................................. 63 Key Questions ............................................................................................................................ 63 Sample ....................................................................................................................................... 63 Data Sources .............................................................................................................................. 66 Data analysis .............................................................................................................................. 66

Re-reports and Recurrence ................................................................................................... 66 Admissions ............................................................................................................................. 67 Placement stability ................................................................................................................ 67 Permanency ........................................................................................................................... 67 Reentry .................................................................................................................................. 68 Care day count ....................................................................................................................... 68

Results ....................................................................................................................................... 69 Maltreatment Recurrence ..................................................................................................... 69

R3 and Re-Reports ............................................................................................................. 70 Admissions: Placement Rates ............................................................................................... 72 Placement stability ................................................................................................................ 73 Permanency ........................................................................................................................... 75 Reentry .................................................................................................................................. 81 Care Day Count ...................................................................................................................... 84

FY2015 Entry Cohort. ......................................................................................................... 84 FY2016 Entry Cohort .......................................................................................................... 87 In Care Group .................................................................................................................... 89

Cost Study ........................................................................................................................... 92 Background ................................................................................................................................ 93 Data Sources .............................................................................................................................. 93 Methodology ............................................................................................................................. 94 Findings...................................................................................................................................... 95

Total Child Welfare Spending ................................................................................................ 95 Foster Care Board and Maintenance Spending (FC B & M) .............................................. 97

Summary, Lessons Learned, and Next Steps ........................................................................ 104 Summary.................................................................................................................................. 104 Lessons Learned ...................................................................................................................... 104 Next Steps ................................................................................................................................ 104

Upcoming Data Collection ................................................................................................... 104 Ongoing analysis .................................................................................................................. 105

Furthering the Partnership ...................................................................................................... 105


Appendix A: Custodial Time Use Summary Tables ............................................................... 106

Appendix B: Non-Custodial Time Use Summary Tables ....................................................... 108

Appendix C: Detailed Results from the General Staff Survey ................................................ 110 TABLES

FAST completion, by Region (previous 12 months) ..................................................... 29 Table 1.

Family Safety, by Case Type ......................................................................................... 36 Table 2.

Discipline, by Case Type ............................................................................................... 38 Table 3.

Supervision, by Case Type ............................................................................................ 40 Table 4.

Caregiver Knowledge of Child/Family Needs, by Case Type ........................................ 41 Table 5.

Family Conflict, by Case Type ....................................................................................... 43 Table 6.

Interview Respondents, by Region and Role ............................................................... 46 Table 7.

Micro-Interview Participants, by Role and Region ...................................................... 47 Table 8.

The Eight Casework Processes of the CCfCS Method .................................................. 51 Table 9.

Regional Verification Figures ..................................................................................... 52 Table 10.

Response to General Staff Survey, by Region ............................................................ 56 Table 11.

Region and Role of GSS Participants, by Wave .......................................................... 57 Table 12.

Region and Program Area of GSS Participants, by Wave ........................................... 57 Table 13.

Admissions to Foster Care, by Region and Fiscal Year ............................................... 64 Table 14.

Children in care on October 1 of the year, by Region ................................................ 64 Table 15.

Recurrence of Maltreatment, by Region and Fiscal Year .......................................... 69 Table 16.

Cumulative Probability of Initial Move: First Ever Spell, by Fiscal Year of Entry ...... 73 Table 17.

Cumulative Probability of a Permanent Exit, by Fiscal Year of Entry and Age at EntryTable 18. 78

Median residual duration, in days, by in-care year and time in care ........................ 80 Table 19.

Conditional Probability of Reentry, by Exit Year and Age at Discharge: Six-Month Table 20.Intervals 82

Exits, Average Care Days, and Total Care Days: FY2015 Admissions Cohort (IHT), by Table 21.Age at Spell Start ........................................................................................................................... 85

Exits, Average Care Days, and Total Care Days: FY2015 Admissions Cohort (IHT), by Table 22.Performance/Waiver Year ............................................................................................................. 86

Exits, Average Care Days, and Total Care Days: FY2016 Admissions Cohort (IHT), by Table 23.Age at Spell Start ........................................................................................................................... 87

Exits, Average Care Days, and Total Care Days: FY2016 Admissions Cohort Table 24.(Demonstration), by Performance/Waiver Year ........................................................................... 88


Exits, Average Care Days, and Total Care Days: In Care on July 1 (Demonstration), by Table 25.Age and Time in Care ..................................................................................................................... 90

Exits, Average Care Days, and Total Care Days: In Care Groups, by Table 26.Performance/Waiver Year ............................................................................................................. 91

Total Child Welfare Spending in Thousands, by Major Category and Fiscal Year...... 96 Table 27.

Foster Care Board and Maintenance as Percent of All Child Welfare Spending, by Table 28.Fiscal Year 97

Foster Care Board and Maintenance Spending in Thousands, by Category and Fiscal Table 29.Year 99

Total Placement Days and Percent Change by Fiscal Year ....................................... 100 Table 30.

Average Daily Cost of Foster Care Placement .......................................................... 101 Table 31.

Care Day Usage by Placement Type and Fiscal Year ................................................ 101 Table 32.

Five Year Care Day Usage Change by Setting Type and Fiscal Year ......................... 102 Table 33.

Child Level Outcomes by Fiscal Year ........................................................................ 102 Table 34.

Eligibility and Response Rates for GSS1 and GSS2 ................................................... 110 Table 35.

Scale Results, by Wave ............................................................................................. 111 Table 36.

Scale Results from Repeated Measures Analysesa .................................................. 112 Table 37. FIGURES

Figure 1. Continuous Quality Improvement Evaluation Framework ......................................... 10 Figure 2. TNDCS Waiver Demonstration Logic Model ............................................................... 13 Figure 3. Implementation of the FAST, by Month and Case Type ............................................ 30 Figure 4. Timing of FAST Completion, by Region ....................................................................... 31 Figure 5. Distribution Across Classification Types, by FAST service rating: CPS-A Cases Only . 32 Figure 6. Distribution Across Classification Types, by FAST service rating: CPS-I Cases Only... 34 Figure 7. Actionable Scores on Family Safety item, by Region .................................................. 37 Figure 8. Actionable Scores on Discipline Item, by Region ........................................................ 39 Figure 9. Actionable Scores on Supervision Item, by Region..................................................... 40 Figure 10. Actionable Scores on Knowledge of Needs Item, by Region .................................... 42 Figure 11. Actionable Scores on Family Conflict Item, by Region ............................................. 43 Figure 12. Actionable Scores on Caregiver Substance Abuse, by Region .................................. 44 Figure 13. Age of children admitted to care, by fiscal year of admission .................................. 65 Figure 14. Age of children in care on in-care date (October 1), by year ................................... 66 Figure 15. Recurrence of Maltreatment, by Age and Fiscal Year .............................................. 70 Figure 16. Trends in Re-Reports: R3 and Non-R3 Areas ........................................................... 71


Figure 17. Placement Rate per 1,000, by Fiscal Year and Region .............................................. 72 Figure 18. Placement Rate per 1,000, by Fiscal Year and Age at Entry ..................................... 73 Figure 19. Cumulative Probability of Initial Placement Change, by Age at Entry ...................... 74 Figure 20. Cumulative Probability of Initial Placement Change, by Region (FY1516) ............... 75 Figure 21. Median Duration, by Fiscal Year of Entry and Region .............................................. 76 Figure 22. Median Duration, by Fiscal Year of Entry and Age at Entry ...................................... 77 Figure 23. Cumulative Probability of Permanent Exit in 12 Months, by Fiscal Year of Entry and Age at Entry 79 Figure 24. Probability of Permanent within 12 Months of Entry, by Fiscal Year of Entry and Region 80 Figure 25. Median Residual Duration, by Time in Care and In Care Group ............................... 81 Figure 26. Reentry Within 6 Months of Discharge, by Exit Year and Age at Discharge ............. 83 Figure 27. Total Care Days: FY2015 Admissions Cohort and Comparison Cohorts (Baseline), by Performance Year .......................................................................................................................... 86 Figure 28. Total Care Days: FY2016 Admissions Cohort and Comparison Cohort (Baseline), Year 1 89 Figure 29. Total care days: 2014 In Care Group and Comparison Cohorts (Averaged), by Year 91 Figure 30. Five Year Change in FC B & M and All Other CW Expenditures ................................ 98 Figure 31. FC B & M: Five Year Change in Private Provider and DCS FP Expenditures .............. 99


Acknowledgements

We would like to acknowledge the contributions made to this report by researchers at the Kempe Center at the University of Colorado in Denver, specifically John Fluke and Dana Hollinshead. The Kempe Center has been serving as a subcontractor on the evaluation of TNDCS’ IV-E Waiver Demonstration Project since the beginning of the effort. In particular, we acknowledge Dr. Fluke’s and Dr. Hollinshead’s work administering the General Staff Survey, producing the resulting analyses, and developing the relevant sections for this Interim Evaluation Report.


Interim Evaluation Report

Tennessee’s IV-E Waiver Demonstration Project August 2017

Introduction and Overview

Background and Context

The Tennessee Department of Children and Family Services (TNDCS) has a long and productive history of work with Chapin Hall at the University of Chicago. The work together has included the institution of performance-based contracting for TNDCS’ network of private providers; performance monitoring Chapin Hall has provided TNDCS as a part of its settlement agreement; ongoing research and analytic work; and, capacity building for staff. Chapin Hall was thrilled to have the opportunity to again partner with TNDCS, this time as the primary evaluator of their IV-E Waiver Demonstration Project.

Perhaps because of this long-standing partnership, Chapin Hall was invited to collaborate with TNDCS very early on in the process of identifying the best opportunities for taking advantage of the flexible funding offered by the IV-E Waiver mechanism. Using the products Chapin Hall has produced for TNDCS over time as well as some new, ad hoc analyses, Chapin Hall helped the Department identify variation in performance that spoke clearly to opportunities to invest in new practices. Ultimately, TNDCS decided to make investments in the following areas:

Family Advocacy and Support Tool (FAST). Beginning in 2014, staff working in all non-custodial program areas were trained to use the FAST as a decision/planning support tool and as an assessment of child and family well-being. The FAST is intended to help staff identify specific areas where children, youth, and families could use additional support, so that service determinations and referrals best fit families’ actual needs.

Keeping Foster Parents Supported and Trained (KEEP). KEEP is an evidenced-based support and skill enhancement model designed for foster and kinship parents. The program supports foster and kin families to promote child well-being and prevent placement disruptions. KEEP takes a group format, with foster parents meeting for 16 weeks to participate in a curriculum focused on teaching foster parents to increase the use of positive reinforcement, set consistent limits, and to utilize non-harsh discipline methods.

Nurturing Parenting Program (NPP). Nurturing Parenting Program is a family centered trauma informed curriculum designed to build nurturing parenting skills as an alternative to abuse and neglectful parenting and child rearing practices. The long-term goals of the program include the prevention of recidivism in families involved with the child welfare system and to stop the intergenerational cycle of abuse by teaching positive parenting behaviors. 1

1 NPP will not be included in this report because implementation is not scheduled to begin until July 2017, after the review period covered by this report.


R3. R3 is an evidence-informed implementation support model, rather than an evidence-based intervention. R3 provides guidelines for supervisors in their interactions with caseworkers and, subsequently, caseworkers interacting with parents/caregivers; the focus is on the improvement of engagement skills and techniques for motivating families to fully participate in their service/permanency plan. Specifically, R3 focuses on the value of reinforcing effort, reinforcing small steps towards goals, and reinforcing roles and relationships; the theory is that if each interaction with a caseworker and/or caregiver is infused with at least one of these principles – effort, small steps and role/relationship – engagement will be better, motivation higher, and positive outcomes (i.e., less recurrence of maltreatment, less time to permanency) more likely to occur.

The TNDCS demonstration project logic model was unveiled in the IDIR and the evaluation plan, both of which were submitted to the Children’s Bureau in the early Fall of 2014. TNDCS’ Waiver Demonstration Project has an official ‘start date’ of October 1, 2014.

Evaluation of the Waiver Demonstration

Broadly speaking, the purpose of the demonstration project is to reduce admissions to foster care, reduce lengths of stay in foster care, and improve well-being outcomes for children in foster care. The focus is on children placed in regular family foster homes licensed by TNDCS; that is, the demonstration project does not focus on children placed in foster homes licensed by any of their private providers of foster care.

Our key research questions are organized around the three facets of the evaluation:

Implementation Study

1. To what extent are Waiver strategies implemented with adherence to original Waiver-specific strategic plans?

2. To what extent are Waiver strategies implemented with fidelity (following model protocols)?

3. What associations exist between (a) staff attitudes about child welfare work, their jobs, and Waiver strategies, (b) adherence to Waiver plans, (c) implementation fidelity, and (d) worker time use?

Outcomes Study

1. What is the impact of demonstration project on the number of care days used, on average (both for children who enter placement after the implementation as well as children in-care at the time demonstration project begins)?

a. What is the impact of the demonstration on the likelihood that children will experience reunification within set periods of time?

b. For children who are not reunified, what is the impact of the demonstration on the rate and timing of alternative permanency options?

2. What is the impact of the demonstration project on the stability of children’s placements in care?


3. What is the impact of the demonstration project on the likelihood that children will experience a post-permanency maltreatment report, the likelihood that that report will be substantiated, and the likelihood a substantiated report will lead to placement (i.e., re-entry)?

4. To what extent do children receiving demonstration interventions (inclusive of new assessment practices and EBIs) exhibit improvements in their functional well-being (i.e., behavior problems, anxiety, depression, trauma symptoms, and adaptive behaviors) relative to their baseline status on these variables? For instance:

a. Do the scores on child level screening/assessment instruments (e.g., CANS) change over time and in the intended direction?

b. Do daily counts of child behavior problems or other child-related issues/needs change over time and in the intended direction?

Cost Study

1. What effect does the Waiver have on child welfare expenditures in TN?

2. What effect does the Waiver have on the average daily cost of foster care placement?

The evaluation framework

Chapin Hall’s general approach to prospective evaluation studies recognizes the need to blend rigorous methodology with the real-world contingencies of operating child welfare programs that directly touch the lives of vulnerable children and families. To meet those twin objectives, we have adopted what we call a Continuous Quality Improvement Evaluation Framework (CQI/EF). The evaluation framework stresses state-of-the-art methodology (the technical aspects of which are described in later sections), whereas the CQI component acknowledges the need to provide meaningful, formative feedback to stakeholders who are working with children and families. The evaluation framework overcomes the methodological weaknesses of many CQI models; the CQI framework manages the need for actionable knowledge well before the summative evaluation is complete.

Figure 1. Continuous Quality Improvement Evaluation Framework


Briefly, the CQI/EF dictates that the work of promoting better outcomes through system improvement is an iterative process (See Figure 1). The cycle begins when the agency identifies the target problems (for IHT, reducing admissions and shortening lengths of stay for children in care), the outcomes of interest (admissions to care, placement stability, permanency, reentry, and well-being), and the intervention/s needed to improve the outcomes (assessment and evidence-based interventions). The selected interventions must be grounded in a theory of change that clarifies the mechanisms that produce changes in the targeted outcomes.

The Waiver strategies selected by TNDCS can be thought of as investments in three areas: the process of care, the quality of care, and the capacity of the agency to deliver process and quality with fidelity. The process of care refers to the steps followed during the time family members/children are engaged with services. It is conceptualized as a series of activities that form the service pathways through the child welfare system (i.e., trajectories). Although the details that define the process of care may draw on a particular model of practice, a particular intervention, state regulation, or agency practice, any given process has common elements or requirements.

The process of care is initialized at the point of first contact (or inception). By necessity, the process consists of a referral mechanism that is used to manage the manner in which a service provider comes to know of a child or family; a procedure for bringing the client into services (i.e., the first contact); and, a procedure for conducting an assessment that is then used to inform what happens next. The referral, intake, and assessment phases lead to a plan that organizes how a service provider will engage the family. Ultimately, the plan has to link the clients to services/interventions that are designed to address the needs and strengths that were identified during the assessment. At this point, services are provided; that is, the process of care is defined by the requirements of a specific intervention. Preparation for drawing services to a close begins with a reassessment of needs, at which time the question at hand focuses on resolution—have the treatment objectives been met? If not, reassessment leads to review of the need/service match. If so, the process of care shifts to discharge planning, clinical follow up, and case closure.

In this case, TNDCS has made one major investment in the process of care: the FAST. With the FAST, TNDCS has put in place a routine and efficient process for assessing child and caregiver functioning across a range of domains for families who come to the attention of TNDCS via any of their non-custodial program areas. FAST assessments are expected to be completed within the first 10 days of a case, in 30-day intervals thereafter, and/or at other critical, clinical moments (changes in children’s or parent’s functioning).

In everyday language, the quality of care refers to how well care is delivered. In practice, process and quality are closely aligned in that adherence to the process of care is in and of itself an indicator of quality, especially if the underlying process protocols are supported by an evidence base that links the process to outcomes.2 As it relates to the demonstration project, TNDCS has made three major investments in improving the quality of care: providing support to foster parents (KEEP); providing support to parents receiving non-custodial services (NPP); and,

2 As noted earlier, the process of care, as a component of quality, may be outlined in law or regulation. However, in some cases, it is unclear how specific process and quality requirements are related to outcomes.


providing support to supervisors (R3). Each of these strategies is expected, in some way, to improve the quality of care provided to parents, foster parents, and/or children and youth.

The capacity of the system to implement the various demonstration interventions in a manner that is consistent with original outlines has human, structural, and organizational elements. For instance, with respect to the former, the capacity question is one of staffing. The number of staff required to implement the demonstration project at scale, to serve the number of children and families TNDCS expects to be placed in care, is the first issue. It is a necessary condition of implementation – necessary, but not sufficient. The ability of staff to do their work in a high quality way is just as important. Capacity is also a structural matter. There may be elements of the demonstration that require workers to have access to new or additional resources (i.e., adaptations to TFACTS to accommodate FAST data entry, tablets to facilitate the completion of the FAST in the field) in order to do their jobs well. From an organizational perspective, the successful implementation of the demonstration project will require adaptations at various system levels –within Central Office and/or across the regions. The faithful implementation of the demonstration project requires staff working in different roles to have the space to dedicate time and attention to the initiative.


Theory of change/logic model

The TNDCS demonstration project logic model is presented in Figure 2 (below). The model is intended to depict how the statistical modeling of effects will likely unfold.

Figure 2. TNDCS Waiver Demonstration Logic Model

Figure 2, above, lays out the theory of change underlying TNDCS’ Waiver project, as well as displaying the nature of each of the hypothesized statistical relationships. Note that the assessment-related changes (FAST), the practice motivated changes (implementation support program, known as R3), and the EBI motivated changes (KEEP, NPP) are each associated with changes in the process of care, the quality of care, and the capacity of individuals to do higher quality work (caseworkers) or provide higher quality care to children (foster parents, parents).3

The implementation of the FAST is represented on the bottom of the graphic. This investment is expected to have a direct effect on placement rates (i.e., safety) by (1) improving the quality of

3 The red lines in the graphic represent moderating effects, such that R3, the implementation support activity targeted at caseworkers, will boost the effects of the foster parent-mediated intervention (Project KEEP) and the parenting intervention, NPP.


information available to both investigation and assessment-track staff (2) improving the fit between families’ needs and the services offered and (3) enabling caseworkers to identify needed services faster. When families can remain together safely, children’s functional well- being is improved.

The evidence-based and evidence informed interventions (EBI) are represented toward the middle of the graphic, to the left; note that the box corresponding to R3 is shaded differently to reflect its status as an implementation support program rather than an evidence-based intervention. As antecedents to changes in well-being, the interventions depicted herein focus on improving the skills of caregivers (parents, foster parents), which will lead to a reduction in children’s problem behaviors, thereby increasing placement stability and, ultimately, permanency (which includes reducing reentry). More stable living arrangements are also associated with improved functional well-being in the medium to long run.

Caseworkers. Direct line staff will be trained in a casework model (R3), a supervisor-mediated intervention whose objective is to model engagement and positive reinforcement in the supervisory context, with caseworkers transferring these skills to the casework context. TNDCS is implementing R3 as a support to the other changes being introduced under the Waiver. The expectation is an improved quality of caseworkers’ day-to-day interactions with parents, foster parents, and youth (custodial and non-custodial), which should ultimately yield better outcomes for children.

Foster parents. The foster-parent focused model, Project KEEP, will be expected to influence the capacity of foster parents to care for children in custodial care. When foster parents have improved skills children’s placements are likely to be more stable, which will increase both the likelihood and the timing of permanent exits from care. These may be direct effects or effects mediated by reductions in children’s problem behaviors. That is, when caregivers’ capacity to work with children is improved, they can help children regulate their emotions and their behavior. When children’s problem behaviors are better managed, their placements are more stable; permanency outcomes are expected to improve as well.

Parents. Parents with children in custodial care in the select counties within the East Grand region, as well as select in-home cases, will also be eligible for an evidence-based parent training program, NPP. As noted previously, we will not offer any additional information about NPP at this time, as the program is just expected to begin rolling out in July 2017, after the review period covered by this Interim Evaluation Report.

The outcomes associated with the changes proposed under the demonstration project are explicated in the pages that follow.

FAST

The FAST will be used as one measure of well-being, the other measure being the CANS.4 The FAST is expected to help caseworkers better identify what clients need to resolve the safety and

4 The CANS has been in place in TN prior to the onset of the demonstration project and is not considered a Waiver intervention.


risk concerns that emerge at the beginning of a non-custodial event. As a result, service planning will be enhanced (quicker linkage to appropriate services, etc.).

KEEP

KEEP is hypothesized to improve foster parents’ skills. Specifically, KEEP focuses on helping foster parents learn strategies for managing children’s problem behaviors so as to (1) reduce foster parents’ stress levels in response to those problem behaviors and (2) help children thrive. This skill-building is expected to improve placement stability (fewer placement changes) and permanency (increased likelihood and timing of permanent exits).

R3

R3 is expected to improve the engagement of caregivers in the service/permanency planning process. In the non-custodial space the expectation is that the introduction of R3 will have the effect of a decrease in re-reports and a decrease in admissions following CPS-A, FSS, and FCIP service provision. In the custodial space the expectation is that R3 will lead to timelier exits to permanency.

In the sections that follow we provide details on the data sources, data collection methods, and analytic methods we have been using (and will continue to use) to answer the key research questions posed earlier in this report.

Implementation study

A range of qualitative methods will be used to monitor the development of TNDCS’ Waiver plan and to understand whether the FAST, KEEP, NPP, and R3 (as an implementation support) influence behaviors in the expected ways. These data collection efforts will be targeted at staff, for the most part. A brief overview of the potential respondents:

CPS-A. These are staff that respond to those maltreatment reports that are routed down the assessment track (non-investigative).

CPS-I. These are staff that respond to maltreatment reports in the traditional, investigative manner.

FSS. These are staff that provide Family Support Services – in home services that typically follow a CPS-A or CPS-I case, but this is not always the case. Some FSS cases do not come as a result of a maltreatment report.

FCIP. These are staff that work in the Family Crisis Intervention Program, which serves older children for the most part.

FSW. Family Service Workers are housed in the custodial space. They are the equivalent of case managers for foster care cases.

Content Analysis: Planning and Implementation

Chapin Hall staff will review written materials as it relates to pre-implementation decision-


making, such as key findings from focus groups and the case record review, data products, and any other written material that serves to document the process through which EBIs are selected, implemented, and monitored.

Structured interviews, Focus Groups, and Online Surveys: Implementation Attitudes

Senior regional staff, team coordinators (TCs), and supervisors are being interviewed at various points over the course of the Waiver demonstration period, during which the following topics are addressed:

1. Waiver decision-making: use of evidence, consideration of CQI concepts in the selection and implementation of EBIs

2. Staff morale

3. Shifts in supervisory responsibilities

4. Implementation of new practices (FAST, EBIs, implementation support models such as R3): practical concerns, utility in practice, etc.

Caseworkers and supervisors are also being surveyed on an annual basis related to the various changes taking place as a part of the Waiver demonstration project. The survey will be used to get a broad sense of implementation issues, such as:

1. Case skills (technical and interpersonal)

2. Job satisfaction, intent-to-leave, and workload

3. Supervision

4. Organizational culture and climate

5. Availability of services to meet client needs

6. Attitudes on reunification and the role of well-being in permanency decisions

7. Case closure decision-making

8. Attitudes on evidence-based practice

Fidelity: Worker Time Use

The process for estimating the amount of time workers spend on casework activities (all of the tasks associated with their job) started with a series of focus groups, in which caseworkers and supervisors provided estimates of the number of minutes it takes to complete various casework


tasks. Sixteen focus groups were held across the East Grand region in order to ensure each casework category was covered at least twice - and by staff working in different regions.5

The time use estimates that emerged from the focus groups informed the construction of a survey that was administered to all frontline staff and supervisors in the 10 counties within the East Grand region that served as pilot sites for the broader demonstration. In particular, the focus groups helped ensure the reasonableness of the answer options and the correct wording of questions. Ultimately, the survey allows for the production of time use estimates that are organized around the eight core processes or sections of casework that together make up the total set of case-specific activities for which workers are responsible.

Administrative data: Fidelity of EBI Implementation

The evaluation is fortunate to have at its disposal automated systems for tracking referrals to KEEP and, eventually, NPP. There is also an automated fidelity monitoring system in place for R3, which is operating in the IHT context as an implementation support. Those data can be linked to the seed analytic files (child spell data) that serve as the foundation for much of the analytic work Chapin Hall does on TNDCS’ behalf (see description in the following section). Ultimately, these linked files will allow for a full tracking of (1) children who would be considered eligible for KEEP (for example), given their age and their regional assignment, (2) the extent to which eligible children actually received the intervention, and (3) the timing of the intervention relative to other key events (beginning of placement, timing of placement changes.

The same holds true for KEEP, which uses the same fidelity monitoring system as is being used for R3 (FIDO).

Outcomes study

The available data allow for the development of a child-specific data file that extends as far back as 1992. From these files we are able to generate performance metrics related to safety (recurrence of maltreatment), placements into care, placement stability while in care, lengths of stay in care, and reentry. These data are included in this Interim Evaluation Report.

Analytic Approach

Using these seed databases, we are also able to develop person-period data files that records the time each child spends (a) receiving non-custodial services and/or (b) in care. Spells (or episodes) are divided into time intervals of a given length (3-month person periods are a starting point). Each person period has associated with it a series of flags indicating whether certain events occurred within the period, notably exposure to an evidence-based intervention, a re-report, admission to care, a placement move, or discharge from custody. The underlying

5 Harriet Ward and colleagues at Loughborough University in the UK originally developed the time use methodology being employed in this study. It has been adapted for use in the US, first in California and Oregon and now in Tennessee, New York City, and Michigan. For more details we would refer you to the following paper: Chamberlain, P., Snowden, L.R., Padgett, C., Saldana, L., Rolls, J., Holmes, L., Ward, H., Soper, J., Reid, J. & Landsverk, J. (2011). A strategy for assessing costs of implementing new practices in the child welfare system: Adapting the English Cost Calculator in the United States. Administration and Policy in Mental Health, Vol. 38, p. 24-31.


statistical model evaluates the log odds of the event of interest; the IHT effect is captured by whether person-periods that include IHT (i.e., during which the FAST was being used, KEEP was offered, R3 was in place, etc.) are more likely to end with either a re-report, an admission to care, a placement change, an exit to permanency, and so forth. The person period model can be extended to incorporate a multi-state, competing risk framework.6

There is the potential for the effect of the various demonstration components to be evaluated singly given that the components are being phased in separately – at least in theory. That is, the process of care change was introduced at the beginning of Waiver Year 1 (FAST); KEEP was introduced mid-way into the first year. R3 came later, with NPP scheduled to roll out in the beginning of Year 3, or thereabouts. However, the actual timing of the use of the FAST, of at-scale implementation of KEEP, etc., will be an important factor when considering the extent to which the roll-out of model components were really separate enough in time to allow for independent analyses.

Because children are clustered within regions, we will account for the nested structure with a multi-level model. In the unconditional model, the level-one intercept is the average rate of exit to permanency, as one example. The multilevel model produces properly weighted estimates of the exit rate (to account for the fact that regions contribute varying amounts of information). Addition of the demonstration effect shows the impact of the demonstration on the average rate. Adding time covariates (i.e., indicating the year during which the interval was observed), controls for any trends in the underlying data as well as other contemporaneous factors present in or affecting the child welfare system that are unrelated to the implementation of the demonstration project.

TNDCS’ demonstration project targets all children between the ages of 0 and 21 who receive non-custodial services (investigation, assessment, in-home family support services) or who are placed in non-specialty family foster homes licensed by the public child welfare system. The sample includes the children receiving in-home services or in care at the start of the waiver demonstration (the legacy caseload) and all new non-custodial cases and admissions involving children entering TNDCS-licensed family foster care. The person periods provide a concise way to introduce demonstration components at the specific time it occurs. For the legacy caseload, this method addresses the fact that children will be at different points in their placement history. Because the log odds of exit or placement change differ with respect to how long children have been in care (and the log odds of admission to care differs with respect to the chronicity of maltreatment), the person periods assess the treatment effects after controlling for the timing of the treatment.

Intent to Treat Design

As a general matter, the analysis will consider both intent-to-treat and per-protocol designs as a way to better understand treatment effects in practice. An Intent-to-Treat (ITT) approach requires that everyone assigned to an EBI-implementing region be included in the analysis of treatment effects, regardless of refusal, noncompliance, protocol deviations, withdrawal, or

6 Steele, F., Goldstein, H., & Browne, W. (2004). A general multilevel multistate competing risks model for event history data, with an application to a study of contraceptive use dynamics. Statistical Modeling, 4(2), 145–159.


anything else that interferes with post-selection uptake of treatment. ITT analysis avoids biased estimates of the efficacy of an intervention resulting from the removal of non-compliers by accepting that noncompliance and protocol deviations are likely to occur in actual practice. As a consequence, in ITT analysis, the estimate of the treatment effect is generally conservative because of dilution attributable to the non-compliance of individuals assigned to the intervention group.7

ITT approaches are often compared to what is termed a ‘per-protocol’ analysis. Per-protocol analyses would, using the present study as an example, exclude children (or the parents/foster parents of children) who deviated from the treatment protocol. One of the main limitations of the per-protocol design is that it can introduce a form of bias called attrition bias, in which the groups of children being compared no longer have similar characteristics. The results of per protocol analyses usually provide a lower level of evidence. Still, because per-protocol analyses tend to better reflect the effects of treatment when faithfully adhered to, they are a worthy complement to ITT analyses.

Cost study

The TNDCS Cost Study is a system level study that examines statewide spending patterns. The study will present the analysis of fiscal data collected from fiscal years 2012, 2013 and 2014 through state fiscal years 2015 and 2016.

The central task of the cost analysis was to create and populate a database including all statewide child welfare expenditures. The TNDCS Cost Study database represents all child welfare related expenditures for five full fiscal years. The database’s structure contains the flexibility to compare financial data within the state, across fiscal years, and within specific expenditure categories. The fiscal analysis began with a simple categorization of costs into seven major categories and then detail categories are layered on allowing for a more nuanced examination of trends over time.

The TNDCS Cost Study database was fully populated using information provided to researchers by TNDCS fiscal administrators. Using the data available to date, researchers examined the following dependent variables:

1. Total child welfare spending;

2. Total foster care expenditures;

3. Paid placement days; and

4. Average daily cost of foster care placement (total foster care board and maintenance expenditures divided by paid placement days).

For each dependent variable listed above, we present the indicator across five fiscal years. Available cost data covers 13 quarters prior to the Waiver and seven quarters since the waiver

7 Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., et al. (2008). Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug and Alcohol Dependence, 95, S74–S104.


was implemented. As such, waiver change is not reported at this stage and no significance testing is conducted at this point.

Sampling plan

The nature of the sampling plan varies somewhat depending on the component of the initiative under consideration. The FAST is being implemented statewide. KEEP and R3, however, are currently being implemented/studied in 10 select counties within the East Grand region (hereafter referred to as the “target counties”). For the most part, the sampling plan for the evaluation follows what was previously laid out above, in the section related to the methodology for the outcomes study. That is, for population-level/ITT analyses we will include in the ‘treatment condition’ the experiences of children in the IHT group (children age 0 to 18 across the state who were the subject of a maltreatment report and/or in-home services case; children age 0 to 21 placed in family foster care in any of 10 select counties within the four East Grand regions), giving consideration to whether a given child was already in care when the initiative began or whether a given child was admitted to care in the target counties on or after the date the initiative got underway. We can compare children over time, looking separately at entry cohorts (historical entry cohorts (2010 through 2012) compared to demonstration-period entry cohorts) and in-care groups (historical in-care groups (2010 through 2012) compared to the single demonstration-period in-care group).8

Limitations

Of course, in any major evaluation effort there are bound to be obstacles of one sort or another – some foreseeable, others less so. Because of Chapin Hall’s long history of working with TNDCS Chapin Hall researchers have, over the years, established strong working relationships with senior staff within Central Office and in many of the regions across the state - staff who tend to be designated as point-people for the implementation of initiatives such as the Waiver demonstration project. However, these individuals are almost always staff who have significant responsibilities outside of coordinating/overseeing the agency’s implementation of new initiatives and the relationship with the evaluation team.

TNDCS is, at its heart, a data-driven organization. Chapin Hall is not the only group collecting information from staff across program areas. We have confronted staff evaluation fatigue from time to time. Because leadership at both the regional and state level are aware of the demands being placed on staff we are often able to work around other data collection efforts to try and avoid fatiguing staff unnecessarily.

Evaluation timeframe and implementation status

Implementation of the evaluation plan follows from the implementation of the initiative.

The FAST was the first component of the demonstration to be rolled out. Initially, Chapin Hall had been working with data from REDCap (the FAST legacy system). About eight to nine months

8 When looking for demonstration-specific program effects we can also compare regions to each other during the same (demonstration) time period. In this scenario, we would compare admissions, stability, permanency, and well being outcomes for children placed in the East Grand focal counties to those placed elsewhere in the state where the EBIs introduced in the East Grand were not also available.


after the FAST was first introduced TNDCS moved over to their SACWIS, TFACTS, as the data management system for FAST assessment data. Data related to FAST implementation and descriptive information about children assessed using the FAST has been released quarterly since late 2014. The data presented in this interim report constitutes the tenth update of its kind.

As for the evaluation of activities related to the implementation of evidence-based models (KEEP) and implementation supports (R3), the evaluation team has been aggressive in collecting data since the onset of implementation: nuanced, qualitative data culled from structured interviews and focus groups; survey data from case planners and supervisors across the East Grand Region that has to do with the specific implementation of these models and other relevant issues; fidelity related data, based on an automated fidelity monitoring system (FIDO) and micro-interviews; and, implementation data, the source of which is TNDCS’ administrative data.


Implementation Study

The Implementation Study set out to address three key research questions:

1. To what extent are Waiver strategies implemented with adherence to original Waiver-specific strategic plans?

2. To what extent are Waiver strategies implemented with fidelity (following model protocols)?

3. What associations exist between (a) staff attitudes about child welfare work, their jobs, and Waiver strategies, (b) adherence to Waiver plans, (c) implementation fidelity, and (d) worker time use?

Data Sources and Data Collection

For organization purposes, it is important to keep in mind that some data collection activities were specific to one particular component of the demonstration. For example, we conducted focus groups with regional staff that were specific to the utilization of the FAST. Other data collection activities gathered information that cut across components of the demonstration. For example, the General Staff Survey and the Time Use Survey ask questions that speak to staff attitudes about and perspectives on their jobs (broadly) as well as specific questions related to the implementation of R3.

With this distinction in mind, in this section we detail the full range of data collection activities that have taken place over the last two and one-half years.

The main data collection activities undertaken as part of the process study fall into four overarching categories: surveys and questionnaires, interviews, focus groups, and documentation review.

Surveys and Questionnaires

Three instruments will be covered in this section: (1) a survey of worker time use, (2) a questionnaire related to organizations’ readiness to change, and (3) a survey of worker/supervisor attitudes about/perspectives on a range of topics related to their work and the demonstration project. For each data collection activity we will review our recruitment procedures and our success at recruiting participants in the various activities.

Time Use Survey (TUS). The TUS was administered to front line staff (CPS-A, CPS-I, FSW, FCIP and FSS staff) and supervisors from the 10 target counties within the East Grand region. Prior to the administration of the TUS, Chapin Hall coordinated with TNDCS staff from Central Office as well as regional leadership to coordinate the data collection effort to make sure senior staff were aware of the upcoming survey and to help senior staff communicate about the survey to front-line staff and supervisors. Chapin Hall provided regional leadership with language to send to staff to help encourage participation and answer any questions about the survey. Chapin Hall also instituted a “contest” of sorts: the region that was able to achieve the highest response rate rewarded with an office-wide pizza party. Staff had approximately three weeks to respond to the survey, during which time multiple reminders about the survey were sent.


A total of 288 staff members from across the 10 target counties within the East Grand region participated in the survey. This accounted for 89% percent of recruited staff. There was some variation in response rates across the regions, ranging from 98 percent (Northeast) to 90% (Knox).

Organizational Readiness for Implementing Change (ORIC). The ORIC was administered to front line staff (CPS-A, CPS-I, FSW, and FSS staff) and supervisors from the 10 target counties within the East Grand region as well as senior leaders from Central Office and the East Grand region. The ORIC was administered within the context of a broader interview. The respondents were asked to consider the implementation of R3 when responding to the items posed in the ORIC. Recruitment was handled in the typical way: Chapin Hall coordinated with TNDCS staff from Central Office as well as regional leadership to coordinate the data collection effort to make sure senior staff were aware of the upcoming request for interviews (that would include the administration of the ORIC). Staff, supervisors and senior leaders were recruited via email. Interviews were conducted in regional offices at the staff’s convenience. The Central Office senior leadership interviews were conducted by phone.

Response rates for the ORIC with front line staff and supervisors was high in the East and Northeast regions (average 93%). Response rates in the Smoky Mountain region were considerably lower (67%). Among senior leaders there was a high response rates for the ORIC (average 92%). In the East, Northeast and Smoky Mountain regions 100% of senior leaders responded.

General Staff Survey (GSS). The GSS was administered to front line staff and supervisors within the 10 target counties across the East Grand region. The first administration of the GSS (GSS1) occurred in July 2015; a total of 217 staff members participated in the GSS1, representing an overall response rate of 65 percent. The GSS2 was administered in January 2017; a total of 291 staff members participated in the GSS2, representing an overall response rate of 79 percent.

Recruitment methods for the GSS were essentially the same as those described just above with respect to the Time Use Survey. The Chapin Hall team coordinated the data collection effort with TNDCS and regional leadership. Staff were given ample time to respond to the survey (about three weeks), with multiple reminders sent out during that period. Again a contest was instituted to try and encourage staff participation in the survey.

GSS1 response rates varied from 47 percent to 89 percent; for GSS2, response rates varied from 75 percent to 81 percent.

Interview Data

Senior Leader Interviews (Fall/Winter 2014). Researchers conducted 27 interviews with senior staff from across the East Grand region and Central Office. The interviews covered topics such as the early stages of FAST implementation, perspectives on the investments the Department was planning to make as a part of its Waiver demonstration, and the implementation of evidence-based models. Participants were recruited via email. The majority of the interviews were conducted face-to-face with a few conducted via phone.


Senior Leader Interviews (Winter/Spring 2016). The evaluation team conducted 35 interviews with senior staff from across the Waiver implementation regions and Central Office. The interviews focused on the implementation of the FAST, Project KEEP and R3. Participants were recruited via email and the interviews were conducted face to face whenever possible.

Interviews with Case-Carrying Staff (Winter/Spring 2016). The evaluation team conducted a series of structured interviews with staff and supervisors to gather data related to the implementation of R3 and Project KEEP. Forty-two frontline staff members from across the East Grand region were invited to participate, as well as 25 supervisors. Multiple email invitations were sent to interview candidates. Senior leadership within the regions were engaged to help boost participation. Ultimately, a total of 56 interviews were completed with frontline staff and supervisors, a response rate of 83 percent.

Micro Interviews (Winter 2017). The evaluation team conducted brief (less than five minutes) interviews with front line staff across the 10 East Grand pilot counties to try and determine the extent to which caseworkers and supervisors were actually “doing” R3 in practice. Frontline workers were contacted and asked to recall specific, recent interactions with (a) a parent and (b) their supervisor, and to consider whether and how R3 principles were used in practice in each of those interactions. Seventy-one front line staff were invited to participate via email. Forty-six front line staff (CPSA, FSW and FSS/FCIP) were interviewed (response rate = 65 percent).

Focus groups

Time Use. The evaluation team conducted 16 focus groups with frontline staff and supervisors from across the 10 East Grand target counties (six to ten participants in each). Time use estimates gleaned from the focus groups were used to help construct the Time Use Survey. The purpose of these groups was to help researchers understand the process of care within each region and to gather time-use estimates that would be used to create response options that accurately reflect the on-the-ground experience of workers in TN.

FAST. Researchers conducted eight focus groups from across the state, 4 with supervisors and 4 with front line staff (six to ten participants in each). The focus groups covered topics related to the implementation of the FAST, perspectives of the FAST over time, the use of the FAST in the development of service plans, and the ability of the FAST to track changes in children and families’ well-being.

Documentation review

Since the onset of the demonstration project Chapin Hall staff have been reviewing planning materials, such as background information related to KEEP and R3, documentation related to the development and scoring of the FAST; and, background materials on NPP. The overarching purpose of this review was to clarify the intervention parameters and to assist TNDCS with the development of both the fidelity monitoring and evaluation strategies.

Data Analysis

In the following section we discuss the findings-to-date as they relate to the Implementation Study. First we present some qualitative data on the implementation of various Waiver strategies, gleaned from data collected from staff across role types during the first and second


demonstration years (Fall 2014/Winter 2015). We then report on the implementation of the FAST, KEEP, and R3, each in turn.9 We conclude with findings from our study of time use, the implications of which cut across the various demonstration strategies.

General Implementation

As detailed above, Chapin Hall conducted a series of 27 interviews with senior staff from across the East Grand region in the fall of 2014, during the early stages of implementation. The interviews covered topics such as the integration of well-being into casework practice; implementation of the FAST; and, how TNDCS is using various forms of evidence to think about permanency outcomes.

General Implementation: Year 1

Senior leaders were generally positive about the early implementation of the FAST. They reported that the new algorithm for determining children’s risk of maltreatment represented an improvement over previous assessments, and the revised timeframes go farther to encourage its use as a decision-support tool. However, interviewees had a mix of perspectives with respect to whether the FAST was being used as a decision-support tool or just being completed to be compliant with policy. As well, some senior staff envisioned the FAST as a decision-support tool for service planning, not for making decisions about whether or not to remove a child from home.

Still, most respondents reported that staff have been given the tools they need to make the most of the FAST. In fact, respondents described FAST training as more substantial than what is normally offered for similar kinds of roll-outs. Staff consistently pointed to TNDCS-sponsored training sessions and ongoing support from the mental health consultants at the Vanderbilt Center of Excellence. A smaller group of respondents focused on the limitation of the training for the FAST assessment tool, in that it covered what the FAST is, what the items mean, and how it should be scored, but stopped short of what caseworkers would do with that information (case practice). Another early and consistent complaint about the implementation of the FAST was that staff were required to enter data into a separate data management system (not the State’s SACWIS). Toggling between REDCap (the data capture system for the FAST) and TFACTS (the State’s SACWIS) was described as cumbersome and time-consuming.

General Implementation: Year 2

In the second year of the demonstration project (Spring, 2016) the evaluation team conducted interviews with three groups of staff from the four regions implementing evidence-based and evidence-informed interventions as a part of the demonstration project: senior leaders (n=35), supervisors (n=24), and frontline staff (n=34). The interviews covered topics related to the FAST, KEEP and R3.

Senior leaders were asked a series of questions related to the implementation of the FAST, such as the alignment between FAST risk ratings and case classifications, the timeliness of FAST

9 Note, NPP had yet to roll out at the time the Interim Evaluation Report was being written.


completion, and leaders’ ideas about improving the process and quality of the FAST implementation effort. The main themes that emerged are outlined below.

Explaining the misalignment of FAST risk ratings/service needs and case classifications. For some families, ongoing support in the form of case management is called for; however, frontline staff may not readily identify their own “service” (case management) via the FAST. This can lead to FASTs that have a final rating of ‘No Services Needed,’ while cases are classified as warranting ongoing attention.

Leaders identified a number of scenarios in which staff seem unsure how to score items on the FAST, such as: when families have service needs that are unrelated to the allegation under investigation, particularly when the maltreatment report will ultimately be unsubstantiated; when the FAST calls for historical data that staff feel is irrelevant to the family’s current situation; or, when a family needs services – but those services are already in place.

Lastly, a number of leaders (n=4) expressed concern that the algorithm that calculates the final risk rating/service need requires additional tweaking.

Timeliness of FAST completion. Leaders offered a number of ideas as it relates to gaps in performance on the timeliness of FAST completion. The leadership expected that the move from REDCap to TFACTS as the system of record for the FAST 2.0 would promote the more timely completion of the FAST, not only because of staff familiarity with TFACTS but because TFACTS allows regional administrators to pull real-time reports of children for whom a FAST is still outstanding.

A few senior leaders (N=3) pointed to a recent change in policy related to drug-exposed infants that has resulted in an increase in CPS-A cases. The increase in caseload has left staff struggling to complete the FAST within the desired timeframes.

Leaders also made note of the frequency of new staff hires: it takes a while for new staff to feel comfortable using the FAST and to get acclimated to the various rules and regulations around its usage.

Improving implementation. Front line staff may need continued training to make the connection between the FAST 2.0, service/case planning, and outcomes. If staff can better attach to the value of the FAST, they may be more invested in using it (and using it early on in their work). In a couple of regions, Vanderbilt COE consultants and regional leadership are working together to come up with creative approaches to continued training and case consultation around the implementation of the FAST 2.0.

Senior Leadership, supervisors and front line staff were also asked questions about the implementation of KEEP. The themes that emerged centered around community engagement and childcare, staff time, and the experience of foster parents. These findings are detailed below.

Community engagement and childcare. At the time these interviews were conducted, some of the communities in the target counties were struggling to find easily accessible community space and childcare providers. This meant that KEEP groups needed to be held at the DCS county offices, which was challenging for both staff and children. For staff, the challenge has to


do with finding space to accommodate KEEP groups given other work taking place (meeting with families, processing intakes). For children, the challenge has to do with returning to a place (the DCS county office) where they may have recently experienced trauma.

Staff time. TNDCS provides childcare for KEEP group participants as a way to encourage foster parents to participate. However, KEEP groups began before childcare provider contracts were in place, creating a situation where frontline staff were required to babysit. Senior leaders, front line staff and supervisors acknowledged the strain this put on the workforce.

As well, the majority of the group leads have reduced caseloads to accommodate KEEP-related responsibilities. However, the KEEP responsibilities have reportedly taken more time than projected, so the reduced caseloads are not having the anticipated effect.

Experience of foster parents. Staff were initially concerned about foster parent recruitment and retention because of the 16-week commitment required to participate in KEEP. In general, recruitment has gone well and retention has been fairly high. Foster parents are reportedly enjoying the groups. Staff report that foster parents are implementing the techniques learned in the groups in their homes. In fact, one KEEP group decided to continue meeting on their own even after the formal training ended because they found the support so useful.

The last topic discussed during this round of interviews was R3. Senior leaders, front line staff and supervisors were asked questions related to group supervision, R3 training and concepts, and their comfort level with using R3. The interviewees had a lot to say about R3 compared to the other interventions.

Group supervision. The majority of supervisors interviewed (n=16) view R3 supervision as positive for their teams, appreciating the learning and collaboration amongst team members. Similarly, a slight majority of frontline staff (n=15) described R3 group supervision as a positive experience, with the benefits of learning from colleagues and brainstorming solutions to difficult cases.

Prior to R3, most teams were not having group supervision, but rather team meetings during which general team information such as policy changes and case coverage were discussed. To be sure, supervisors and frontline staff both reported spending ample time discussing cases – just not typically in the context of these regularly scheduled team meetings. It was originally thought that R3 would be “done” during these meetings; however, since cases were not typically reviewed during this time, another group supervision meeting had to be added to the staff’s and supervisors’ calendars, the focus of which would be case review for the purpose of measuring supervisors’ fidelity to R3 concepts. Approximately one-third of the supervisors with whom we spoke (n=8) described this extra R3-specific meeting as a struggle for staff whose schedules were already full before R3. In part, supervisors’ discontent about the added meeting has to do with repeating case-related conversations so that they may be captured by the video camera.

Of the remaining frontline staff – those who were not explicitly positive about R3 – most (n=14) expressed ambivalence about group supervision, often saying they routinely discussed cases with colleagues and supervisors prior to R3. The addition of R3 is seen as bringing a new formality to those conversations, but the conversations were taking place beforehand.


Practice. Frontline staff and supervisors shared a mix of opinions about their comfort level using R3. On the one hand, staff and supervisors reported being more aware of how they speak to clients and each other since the implementation of R3, focusing now on making their language more positive. There were several examples of R3 being used as a verb, such as, ‘You would be proud of me, I just R3’d that parent.’ On the other hand, frontline staff and supervisors spoke in the interviews about the language of R3 not always feeling natural or fitting in with the personality of the supervisor. We heard from both frontline staff and supervisors that different personality types do not need or respond well to constant positive reinforcement.

Non-custodial teams reported the R3 model may not be as good of a fit for their work as for custodial cases. CPS-A staff who participated in interviews expressed concern that the volume of cases and the (relatively short) length of time many of their cases remain active does not leave them enough time to establish the type of relationships with families that are necessary for R3 to be implemented as intended. Furthermore, supervisory interactions within CPS-A teams often occur by phone, with staff calling in regularly from the field.

A small group of staff (supervisors and frontline staff combined) expressed some concern about the use of formal certificates to reward parents for making progress with small steps and how these certificates could be used in court proceedings. For example, if TNDCS does not feel it is safe for a child to return home yet the parent shows the judge certificates they have received for good progress, the certificates could be confusing to judges.

FAST

Our goal in this section is twofold: (1) to review FAST compliance (are workers completing a FAST for eligible children, within desired timeframes, and in alignment with case classifications?) and (2) to review FAST “findings” with respect to the presence of actionable needs in areas of particular importance for children at risk of being placed into out-of-home care.

Compliance

In this section we answer the following questions as it relates to compliance with FAST guidelines:

1. To what extent are children who are eligible for a FAST having at least one completed on their behalf?

2. To what extent are children who are eligible to be assessed using the FAST having a FAST completed within desired timeframes, allowing for the integration of the assessment information into decisions around services?

3. To what extent do FAST scores (level of service need) correspond with case classification decisions?

We address the first question in Table 1, below.


FAST completion, by Region (previous 12 months) Table 1.

MONTH/YEAR

REGION Apr

2016 May 2016

Jun 2016

Jul 2016

Aug 2016

Sep 2016

Oct 2016

Nov 2016

Dec 2016

Jan 2017

Feb 2017

Mar 2017

Northeast 97% 97% 96% 97% 97% 97% 96% 96% 96% 96% 96% 87% Knox 95% 95% 93% 92% 93% 92% 90% 91% 90% 91% 89% 76% East 94% 94% 94% 90% 95% 95% 94% 95% 94% 97% 96% 80%

Smoky Mt. 93% 94% 96% 92% 91% 93% 96% 94% 94% 93% 92% 86% Upper Cumb. 94% 92% 94% 90% 95% 94% 96% 93% 91% 94% 89% 79% South Central 96% 97% 97% 96% 96% 98% 96% 97% 97% 97% 96% 82%

Davidson 94% 93% 92% 94% 96% 96% 95% 94% 95% 94% 94% 69% Mid Cumb. 95% 95% 97% 96% 97% 94% 96% 93% 94% 94% 88% 76% Southwest 95% 97% 97% 95% 94% 96% 96% 93% 95% 93% 92% 76% TN Valley 94% 94% 94% 91% 94% 94% 93% 95% 95% 95% 92% 77%

Shelby 94% 94% 91% 92% 94% 94% 94% 94% 93% 94% 93% 80% Northwest 97% 96% 96% 94% 96% 96% 97% 94% 94% 92% 86% 72%

Table 1 displays the extent to which children who are eligible for a FAST (children with a new CPS-I, CPS-A, FSS, or FCIP case) have at least one FAST completed on their behalf. We look at the last year of implementation activity, and see that most regions have at least 90% completion rates month to month. The dip at the end of the observation period (March 2017) is likely an artifact of censoring; we have not yet had enough time to observe whether cases that opened in March 2017 will have a FAST completed. Figure 3 provides a different view: implementation levels over the past 12 months broken down by case type rather than region.


Figure 3. Implementation of the FAST, by Month and Case Type

Generally, the rate of FAST completion for CPS cases is hovering around 95 percent; for FSS and FCIP cases, the rate of FAST completion is between 80 to 90 percent, on average.


Figure 4. Timing of FAST Completion, by Region

Across regions, we see that the majority of FAST assessments are being completed within the desired time frames (within 10 business days of the event start date).

There remains the question of whether FAST assessments are factoring into case decisions in the manner intended and as articulated in the theory of change. The evaluation has been considering the question from two angles. One is through conversations with frontline staff and supervisors, where we ask this question fairly directly. We present that qualitative data in the General Implementation section, above. Another way to think about this is to compare FAST assessments (the level of service need, as indicated by the FAST score) with case classifications and look to see evidence of alignment between FAST scores and case classifications. In Figure 5 we provide this view, separately for CPS-A and CPS-I cases.


Figure 5. Distribution Across Classification Types, by FAST service rating: CPS-A Cases Only10

10 UNSUB refers to unsubstantiated, a designation typically used in the context of investigations (not CPS-A cases). However, a small portion of CPS-A events are given this designation, which is actually a composite of other, largely administration classifications. It does not necessarily speak to cases that moved from the CPS-A track to the traditional investigative track.


Overall, CPS-A cases that have higher service intensity ratings (per the FAST) are associated with classifications that indicate the need for services: Services Offered and Accepted, Services Offered and Refused, and Services Required. Of the CPS-A cases where the FAST indicated no service need, 70 percent had the corresponding classification of Services Not Needed. On the other hand, of the CPS-A cases where the FAST indicated a high service need, only 13 percent had a classification of Services Not Needed.


We see a similar situation play out for CPS-I cases (traditional investigations).

Figure 6. Distribution Across Classification Types, by FAST service rating: CPS-I Cases Only

In Figure 6 we see that the higher the service intensity the higher the likelihood that the case will be substantiated.


FAST “Findings”

Broadly, the FAST has two main purposes. The first purpose of the FAST is to inform service planning. It is a tool for workers to help them detect areas where children and caregivers are in need of support and the immediacy or urgency of the need. The second purpose of the FAST is to track changes in child and caregiver functioning over time. Because the FAST is expected to be completed upon initiation of non-custodial services and at regular intervals thereafter, there is the capacity to collect multiple data points on a given child/family, to see the extent to which progress is made in identified need areas. Looking at FAST scores in the aggregate – for example, looking at the set of FAST scores from all children’s initial FAST – can go a long way to help child welfare administrators understand the needs of children and families and to make sure that investments in improving the quality of care are properly targeted.

The tables below give snapshots of how children and caregivers receiving non-custodial services are doing in the following key areas: caregiver supervision, family safety, caregiver knowledge of child and family needs, family conflict, and discipline. We focus on these items because they will be critical to determining families’ eligibility for Nurturing Parenting Program services going forward. Further, they are items that are highly relevant for determinations about safety and risk and the potential need for placement in out-of-home care.

For each area we first focus on cases in which some type of service or need was indicated. For Assessment cases we focus on cases classified as Services Required, Services Offered and Accepted, and Services Offered and Refused; we excluded cases classified as Services Not Needed. We also provide information on the extent to which there is regional variability in the presence of actionable need in each area.

We are only including FASTs completed in the last 12 months (between April 2016 and March 2017).

Family Safety. The family safety item on the FAST gauges the extent to which children are considered safe in the home, with “safety” considered in fairly broad terms; that is, there is no specific mention to the type of injury (emotional, physical, etc.) a child could be at risk of incurring.

35

Family Safety, by Case Type Table 2.

Assessment Count Percent No Action Needed 11,483 72% Watchful Waiting 3,384 21% Action Needed 812 5% Immediate Action Needed 271 2% Total 15,950 100% Investigation SUB UNSUB No Action Needed 4,214 68% 15,163 84% Watchful Waiting 1,122 18% 2,432 14% Action Needed 480 8% 341 2% Immediate Action Needed 382 6% 67 0% Total 6,198 100% 18,003 100% FSS Count Percent No Action Needed 3,065 67% Watchful Waiting 1,136 25% Action Needed 315 7% Immediate Action Needed 93 2% Total 4,609 100%

Overall, most families are not scoring in the actionable range, regardless of the type of case (assessment, investigation, or in-home services (FSS)). Families are more likely to score in the ‘Watchful Waiting’ range in the FSS and assessment program areas compared to investigations, although the difference is by no means extreme.

Figure 7, below, looks at variability across regions in the extent to which families scored in the actionable range on the Family Safety item.

36

Figure 7. Actionable Scores on Family Safety item, by Region

Generally speaking, most families do not score in the actionable range on the Family Safety item. While scores across the board are low, we do notice some regional variability, with staff in Northwest using actionable scores (Action Needed or Immediate Action Needed) more regularly than staff in Shelby, Davidson, and South Central, most notably.

Discipline. This item on the FAST has to do with the kinds of discipline practices parents use; parents who score in the actionable range have a difficult time setting appropriate limits, may use harsh disciplinary techniques, and may have unrealistic expectations of their children.

Table 3 details the extent to which caregivers scored in the actionable range on the discipline item. As above (with Family Safety), we see little in the way of actionable scoring on the discipline item: between zero to five percent across the board. There is some evidence that issues with disciplinary practices are being picked up with greater regularity by the workers in the FSS program area; 20 percent of FSS cases from the past year involved a caregiver with disciplinary practices that resulted in a FAST score of ‘Watchful Waiting.’

37

Discipline, by Case Type Table 3.

Assessment Count Percent No Action Needed 12,548 79% Watchful Waiting 2,833 18% Action Needed 475 3% Immediate Action Needed 94 1% Total 15,950 100% Investigation SUB UNSUB No Action Needed 5,196 84% 15,866 88% Watchful Waiting 712 11% 1,912 11% Action Needed 170 3% 201 1% Immediate Action Needed 120 2% 24 0% Total 6,198 100% 18,003 100% FSS Count Percent No Action Needed 3,477 75% Watchful Waiting 941 20% Action Needed 169 4% Immediate Action Needed 22 0% Total 4,609 100%

Figure 8, below, looks at variability across regions in the extent to which primary caregivers scored in the actionable range on the Discipline item.

38

Figure 8. Actionable Scores on Discipline Item, by Region

Very few caregivers were scored in the actionable range on the Discipline item. There is very little regional variability, although again we see staff in Northwest using actionable scores (Action Needed or Immediate Action Needed) more regularly than staff in other regions.

Supervision. This item on the FAST has to do with ensuring age-appropriate supervision of children. Actionable scores are appropriate for caregivers who consistently demonstrate an inability to provide supervision for their children in line with their needs, given their age and/or development.

Staff are more likely to score caregivers in the actionable range on the Supervision item than we’ve seen so far with the Family Safety and Discipline items. Table 4 provides the details.

39

Supervision, by Case Type Table 4.

Assessment Count Percent No Action Needed 9,860 62% Watchful Waiting 4,520 28% Action Needed 1,159 7% Immediate Action Needed 411 3% Total 15,950 100% Investigation SUB UNSUB No Action Needed 3,815 62% 14,326 80% Watchful Waiting 1,479 24% 3,229 18% Action Needed 567 9% 372 2% Immediate Action Needed 337 5% 76 0% Total 6,198 100% 18,003 100% FSS Count Percent No Action Needed 2,658 58% Watchful Waiting 1,463 32% Action Needed 408 9% Immediate Action Needed 80 2% Total 4,609 100%

Figure 9, below, looks at variability across regions in the extent to which primary caregivers scored in the actionable range on the Discipline item.

Figure 9. Actionable Scores on Supervision Item, by Region

40

As with the other items that have been examined in this section, there is little regional variation here: in most regions, between five to nine percent of caregivers scored in the actionable range on the Supervision item. We see somewhat greater proportions in Northwest and Smoky Mountain.

Caregiver Knowledge of Child and Family Needs. This item on the FAST gauges the extent to which caregivers understand the needs of their children/family. It is fairly non-specific; “needs” and “knowledge” are very broad constructs that are not specifically defined in this context.

Caregiver Knowledge of Child/Family Needs, by Case Type Table 5.

Assessment Count Percent No Action Needed 10,369 65% Watchful Waiting 4,535 28% Action Needed 828 5% Immediate Action Needed 218 1% Total 15,950 100% Investigation SUB UNSUB No Action Needed 4,214 68% 15,077 84% Watchful Waiting 1,434 23% 2,659 15% Action Needed 366 6% 230 1% Immediate Action Needed 184 3% 37 0% Total 6,198 100% 18,003 100% FSS Count Percent No Action Needed 2,764 60% Watchful Waiting 1,430 31% Action Needed 357 8% Immediate Action Needed 58 1% Total 4,609 100%

Between six to nine percent of caregivers who come through the assessment and FSS route are determined to lack a fundamental understanding of their children’s/family’s needs. We see a similar proportion within the group of parents who have at least one child who is the victim of a substantiated investigation; for unsubstantiated investigations we see very little in the way of actionable need in this area.

Figure 10 displays variation by region.

41

Figure 10. Actionable Scores on Knowledge of Needs Item, by Region

The proportion of caregivers who score in the actionable range is low across the board – less than 10 percent in every region. There is some regional variability; again we see Northwest and Smoky Mountain towards the higher end and Shelby towards the lower end.

Family Conflict. This item on the FAST has to do with how much fighting and arguing takes place in the home. Actionable scores are given in situations where fighting and/or arguing is persistent or where fighting becomes violent and dangerous to family members.

As evidenced in Table 6, the likelihood of a family scoring in the actionable range on the Family Conflict item is higher in FSS and CPS-A event types than for investigations – approximately 14 percent in the former event categories compared to five percent in the latter.

Regional scores are not displayed; there is simply too little ‘actionable’ scoring on this item.

42

Family Conflict, by Case Type Table 6.Assessment Count Percent No Action Needed 1,073 54% Watchful Waiting 634 32% Action Needed 233 12% Immediate Action Needed 45 2% Total 1,985 100% Investigation SUB UNSUB No Action Needed 61% 71% Watchful Waiting 27% 24% Action Needed 7% 4% Immediate Action Needed 5% 1% Total 100% 100% FSS Count Percent No Action Needed 322 55% Watchful Waiting 180 31% Action Needed 78 13% Immediate Action Needed 5 1% Total 585 100%

Figure 11 displays the variation by region.

Figure 11. Actionable Scores on Family Conflict Item, by Region

43

Families in Shelby appear less likely to be scored in the actionable range than families in most of the other regions.

FAST: Summary

Compliance with FAST regulations has been very high – both across regions and over time. A solid majority (over 90 percent) of children are having at least one FAST completed on their behalf; 75 to 85 percent of the children for whom a FAST is completed are having one completed within the first 10 business days after a case is initiated.

Most caregivers assessed using the FAST are not scored in the actionable range in any of the major domains presented here. The Family Conflict item was the most likely to be scored in the actionable range; the Discipline item was the least likely. Workers seem more comfortable relying on the score that suggests either a history of a problem or the suspicion of a problem, rather than the scores that indicate a moderate-to-severe level of the problem. Whether this is because caregivers receiving non-custodial services truly do not have moderate to severe problems associated with these five areas (discipline, supervision, family safety, developmental, and knowledge of needs), at least as described by the FAST, or because workers need additional training to be able to identify these issues is unclear.

There is one caregiver issue that is not mentioned here that we do want to note: caregiver substance abuse. Because we chose to focus on those items that will inform, at least in part, whether parents get routed toward an upcoming Waiver strategy (NPP) we did not cover caregiver substance abuse above. However, this is most definitely an issue that staff (and families) are coping with in Tennessee.

Figure 12. Actionable Scores on Caregiver Substance Abuse, by Region

Of the families for whom there is at least one FAST and for whom there was some determination of service need, anywhere from five to 21 percent of them have an actionable

44

substance abuse problem. An additional 12 to 30 percent are scored as ‘Watchful Waiting’ on this item.

KEEP

In this section we look at findings related to KEEP, focusing on interview data that speaks to staff perceptions of KEEP.

Early impressions: Expectations for KEEP

Eighty-five interviews with front-line staff (CPSA, FSW, FSS/FCIP) and senior leadership (regional and Central Office) were conducted as a way to understand the beginning stages of the implementation of KEEP. Of interest was how the actual rollout of KEEP compared to expectations of the roll-out, including the level of foster parent participation at that time. The KEEP interview questions were part of a larger interview, which covered many aspects of the IHT implementation.

Table 7 details the interview participants, by role and region.

45

Interview Respondents, by Region and Role Table 7.Region CPSA FSW FSS/FCIP Leadership Total Knox 4 11 3 8 26 Smoky Mountain 3 7 2 3 15 East 2 7 3 5 17 Northeast 4 7 3 6 20 Central Office - - - 7 7 Total 13 32 11 22 85

The recruitment process was as follows. First, up-to-date staff contact information was obtained from regional leadership. Next, the evaluators prepared text for the regional leadership to send to the front line staff and supervisors via email, alerting them that a member of the evaluation team would be reaching out to them to participate in an interview and encouraging them to participate. Recruitment emails were then sent to potential participants, which explained the nature of the interview and how to contact a member of the Chapin Hall team to signal interest in participation. The initial email was followed by two follow up emails over the course of the next few weeks, each of which offered contact information and stated the dates the researcher would be available in their specific regional office for the 30-minute interview. All respondents who reached out to the evaluation team expressing interest in participating were ultimately interviewed. A total of 85 staff members were recruited for participation in interviews, with a response rate of 83 percent (85/103).

Three main themes emerged from the interviews with senior leadership, front line staff and supervisors related to KEEP: logistics (community engagement and childcare), staff time, and the experience of foster parents.

Logistics. At the time the interviews were conducted, some of the communities in the target counties were struggling to find easily accessible community space in which to hold the KEEP groups; the counties also struggled to secure childcare for foster parents’ children (during the KEEP group itself). Initially, this meant that KEEP groups needed to be held at DCS county offices, which was challenging for both staff and children. For staff, the challenge had to do with finding space to accommodate KEEP groups given other activities taking place (meeting with families, processing intakes). For children, the challenge had to do with returning to a place (the DCS county office) where they may have recently experienced trauma.

Staff time. Currently, TNDCS provides childcare for KEEP group participants as a way to encourage foster parents to participate. However, KEEP groups began before childcare provider contracts were in place, creating a situation where frontline staff were required to babysit. Senior leaders, front line staff and supervisors acknowledged the strain this put on the workforce. Further, while the majority of the KEEP group leads have reduced caseloads to accommodate KEEP-related responsibilities, KEEP responsibilities reportedly took more time than initially projected. As such, the reduced caseloads did not – at least initially - have the anticipated workload-balancing effect.

Experience of foster parents. Staff were initially concerned about foster parent recruitment and retention because of the 16-week commitment required to participate in KEEP. In general,

46

recruitment has gone well and retention has been fairly high. Foster parents are reportedly enjoying the groups. Staff also report that foster parents are implementing the techniques learned in the groups in their homes. In fact, one KEEP group decided to continue meeting on their own even after the formal training ended because they found the support so useful.

Generally speaking, our sense is the benefits associated with KEEP – in terms of workers’ capacity to support foster parents and foster parents’ capacity to support children in custody – outweigh the largely logistical challenges staff have encountered during the early phase of implementation.

KEEP Implementation11

Staff training in the KEEP model rolled out over the course of two months (late July 2015 through late September 2015). The target population for training was frontline staff from the ten target counties. Most of these staff members received foundational training in KEEP concepts. A small subset of staff were trained to facilitate KEEP groups. The target population for actual participation in KEEP groups was foster parents who had in their custody (or planned to have in their custody) children between the ages of 4 and 12 years. This includes non-kin and kinship foster parents as well as pre-adoptive parents.

R3

In this section we look at findings related to R3. Here we utilize data culled from micro-interviews held with staff to help understand the implementation of R3 in the field. Micro-interviews are short, five-minute interviews done with front line workers during which workers are asked to reflect on their most recent conversations with 1) their supervisor and 2) a parent of a child on their caseload. The evaluator then proceeds to ask them specific questions related to the use of R3 in those specific conversations to learn if and how R3 is happening.

Early impressions

Table 8 details the interview participants, by role and region.

Micro-Interview Participants, by Role and Region Table 8.Region CPSA FSW FSS/FCIP Total Knox 1 10 1 12 Smokey 3 7 2 12 East TN 4 5 0 9 Northeast 3 7 2 12 Total 11 29 5 45

11 In the next semi-annual update (covering the period through October 1, 2017) we will some descriptive information about the initial roll-out of KEEP, through June 30, 2017. We will look at the extent to which children in the target counties were touched by KEEP (by virtue of being in a foster home in which at least one foster parent participated in a KEEP group before or during their time in that home).

47

Recruitment for micro-interviews followed the same set of procedures outlined above with respect to staff interviews. The initial recruitment for the micro-interviews was fairly unsuccessful. The evaluation team waited about four weeks and then repeated the approach, enlisting regional leadership as partners to reach out to the staff first. The second round of recruiting for micro-interviews had better results. A total of 71 staff members were recruited for participation in a micro-interview, with a response rate of 63 percent (45/71).

As noted above, workers were asked to recall their most recent conversation with (1) a parent of a child on their caseload and (2) their supervisor. Participants were instructed to recall the most recent interaction, which could have been by phone, in-person, or via text/email. With respect to supervisors, workers were instructed to hone in on the most recent interaction in which any information was exchanged, outside the context of team meetings or group supervision.

Once workers identified the most recent interaction with (first) a parent and (next) their supervisor, workers were asked specific questions related to the use of the 3 R’s: reinforce small steps toward goals, reinforce their relationship and role and reinforce effort being made in their work. If the worker indicated that they or their supervisor had used one of the above strategies, they were then asked to relay exactly how they did it.

Importantly, when conducting these micro-interviews the interviewer went to some length to help reduce the extent to which responses would be influenced by what is known as the “social desirability bias,” which refers to individuals’ desire to ‘sound good’ to the interviewer. The evaluator made explicit that the questions were not an evaluation of the worker or their supervisor but rather, our way of understanding if and how R3 is being used in the field.

The majority of conversations upon which participants reflected had occurred within the previous 24 hours. A few of the conversations occurred within the previous few days. Just over 20 percent of the interactions occurred four to seven days prior to the micro-interview; two of the supervisor conversations were more than 10 days prior to the micro-interview.

The use of R3 in practice

Four main themes emerged during these micro-interviews. First, in general we did not pick up on a strong distinction between R3, which requires the identification of a specific observable behavior, and general strengths based practice. Some examples of strengths-based practice that seem to fall short of the R3 standard (identifying specific, observed behaviors) include, “my supervisor gave me praises and asked if there was anything she could do to help me,” and “[my supervisor] is always saying ‘good job’ and sending us smiley faces and encouraging emails” and “[my supervisor] often reiterates I am a good worker.”

Second, when staff were asked if their supervisors reinforced their role and relationship with their clients the examples indicated that supervisors are more likely to emphasize workers’ role of administrating the tasks necessary to move the case along, versus the value of the workers’ role/relationship with parents. Workers provided examples of the R3 principle of “reinforcing role and relationship” including, “[my supervisor] reminded me that I am the caseworker and

48

need to be on top of everything,” and, “my supervisor outlined what I am supposed to do during an initial meeting,” and, “[my supervisor] reminded me that I am supposed to return calls to clients in a timely manner.”

Third, staff do not appear to see all interactions as ripe for the use of R3. Rather, we heard about several types of interactions for which staff indicated R3 would not apply; that is, staff believed R3 would not be indicated for those types of interactions. Examples include when workers have their first meeting with a newly referred family; when talking to a parent whose rights have been terminated; when supervising visits; and other case interactions that, “…are not really a reinforcing conversation but more of a telling her what needs to be done.”

Lastly, although the R3 model and fidelity monitoring protocol allow for the possibility that a single interaction can satisfy the requirement to both “reinforce small steps towards goals” and “reinforce effort,” our sense from staff is they do not necessarily see the distinction between these two components of the R3 model. This was more readily apparent when micro-interview participants were reflecting on a recent interaction with their supervisor.

Time Use

The premise underlying the study of time use is the inherent potential in understanding the value, in terms of improved outcomes, in changing the amount of time workers spend doing certain tasks (FAST) and changing the quality of the time workers spend doing certain tasks (R3, KEEP).

Hypotheses

The strategies TNDCS is employing under IHT are hypothesized to shift workers’ time use patterns in various ways. For example:

1. Workers may spend more time communicating with biological parents and children as a result of enhanced relationships derived from R3. Generally, more direct contact with children and families is desirable, in that it promotes better assessment and greater capacity of workers to reinforce gains in permanency planning (or provide additional support).

2. Workers may spend less time making referrals throughout the life of a case if workers are better able to identify families’ strengths and needs at the beginning of a case (FAST). Less time making service referrals could mean more time spent having important direct contact with children, caregivers, and families.

The preceding examples consider how outcomes might improve as a result of changes in the way workers use their time; namely, less time on indirect casework activities and more time on direct casework activities. However, better outcomes can be achieved without a noticeable shift in time use patterns. That would suggest a change in the way workers are spending their time – as opposed to the amount of time they are spending.

Organizing Framework

49

In order to carefully track shifts in time use, the Cost Calculator for Children’s Services method of studying time use separates casework into eight processes (or categories of activity). The eight processes are well established and have been shown to have applicability across jurisdictions.12

Table 9 details the eight processes of the CCfCS method.

12 Holmes, L., McDermid, S., Padley, M., & Soper, J. (2012). An exploration of the costs and impact of the Common Assessment Framework. Department of Education: London.

Holmes, L., & McDermid, S. (2012). Understanding costs and outcomes in child welfare services: A comprehensive costing approach to managing your resources. Jessica Kingsley Publishers: London.

Ward, H., Holmes, L., & Soper, J. (2008). Costs and consequences of placing children in care. Jessica Kingsley Publishers: London.

50

The Eight Casework Processes of the CCfCS Method Table 9.

Process 1 Making the decision to place a child in out-of-home care

Includes all activities up to and including the decision to place a child in foster care.

Process 2 Developing the initial permanency plan

Includes all activities that contribute to the development of the initial permanency plan. Usually occurs within the first 30-45 days of a case.

Process 3 Maintaining the case Includes all activities related to the ongoing maintenance of the case except for non-court case reviews (such as conferences) and court related activities. “Starts” after the initial service (or permanency) plan is developed.

Process 4 Ending a case Includes activities specifically related to ending a case. Does not include tasks covered in Process 3.

Process 5 Placement change Includes activities related to a placement change (from one foster home to another, or from one level of care to another). Does not include activities related to the ongoing maintenance of a case (Process 3).

Process 6 Non-court case reviews Includes the tasks required to hold a single non-court case review, such as child family team meetings (CFTM). Includes such tasks as scheduling the meeting, attending the meeting, and debriefing with case stakeholders following the meeting.

Process 7 Legal activities Includes activities related to court hearings, such as traveling to court, preparing for court, and appearing in court.

Process 8 Independent living services Includes activities related specifically to the provision of (or connection to) services related to preparing adolescents for adulthood.

Methods

To review, time use data is collected via a series of focus groups and a widely administered online survey. The Time Use Survey is administered at two points in time (second administration projected for mid-2018); comparing reported time use from one time period to the next – before/early in the implementation process and then after the implementation has occurred – allows for an examination of the extent to which time use patterns have changed as a result of the new strategies.

51

Once estimates are gathered from child welfare staff those estimates are verified; that is, the estimates are determined to be reasonable (or not). Recall the time use data gathered from workers reflects the time they spend on direct and indirect casework activities – not in supervision, on paid time-off, in staff trainings or so on. That is, we would not expect the time use data gathered from workers to account for 100 percent of their time. Research shows that workers tend to spend between 65 to 80 percent of their time on casework activities (direct and indirect), with the remaining time absorbed by time off, training, supervision, and other non-casework activities.13

The verification process used in this study involved the use of the administrative data to re-create caseloads on a given point in time for 40 randomly selected workers (10 from each participating region). Then, each child’s administrative record was examined to understand which of the eight CCfCS “processes” (see above) were operating within the span of one month. For example, in one month a worker may have to handle three placement changes for three separate children (Process 5); may have two new cases that require the intense attention that new cases tend to require (Process 2); have two cases that are preparing for reunification (Process 4); and so on. The administrative data gives us a lot of insight into what is required, from a case management perspective, of a worker in a given month period. Once those determinations are made, the relevant time use estimates are attached to each child’s record.

The TN time use verification analysis revealed that the forty workers in our sample spent, on average, between 69 and 82 percent of their time on case-specific work during the sample month. This is consistent with what we would expect, given the literature (referenced above) regarding workers spending 65 to 80 percent of their time on caseload management tasks.

Regional Verification Figures Table 10.

Region Average hours spent on casework specific

tasks in a given month Percent of available time

Smoky Mountain (n=10) 103.90 69%

Northeast (n=10) 123.50 82%

Knox (n=10) 115.40 77%

East Tennessee (n=10) 103.9 69%

With time studies of this nature, there are understandable concerns about overestimation and double counting. This method incorporates a system of checks and balances to identify issues of this nature.

Time Use Findings

13 New York State Office of Children & Family Services. New York State Child Welfare Workload Study: Final Report. Submitted by Walter R. McDonald & Associates, Inc. and American Humane Association. November, 2006

52

In July 2015, case-carrying workers and their supervisors from the 10 pilot counties within the East Grand region were invited to participate in a Time Use Survey. Overall, we heard from 100 percent of the custodial workers invited to participate and 84 percent of the non-custodial workers invited to participate.

The full set of time use data tables are available in Appendices A and B. Below we provide general observations related to Time 1 of the TUS. Later, after a second administration of the TUS, we will assess whether there have been changes in the way workers in TN allocate their time.

Developing the Initial Service (Permanency) Plan. On average, caseworkers spend approximately 44 hours over the course of the initial 30 days developing the permanency plan for one child in foster care. Team Leaders, on the other hand, spend 5 hours (on this same case) during this early phase of the case.

1. Of the 44 hours the caseworker spends developing the initial permanency plan for a child in foster care, approximately 7.5 hours are spent scheduling, supervising, and documenting family visits.

2. The caseworker spends approximately 10 hours in face-to-face contact with the child and caregivers during initial permanency plan development phase. This is separate from the time caseworkers spend on family visits that occur within the first 30 days. Generally, this time is broken down as follows:

a. 2 hours with the child

b. 5 hours with the biological family

c. 3 hours with other relatives and foster parents

3. Setting family members up with services during the first 30 days of a case also takes time. Caseworkers report spending, on average, about 2 hours on researching, making, and documenting referrals.

4. Travel takes up about 16 percent of the time (7 hours) caseworkers spend on casework related tasks during the first 30 days of a case. This includes travel to the biological home, foster home and school.

When a child is receiving non-custodial services, the development of the initial service plan takes about 12 hours of caseworker time and 2.5 hours of supervisor time. About 40 percent of this time is spent on activities related to the CFTM or in direct contact with family members.

Maintaining the Case. There is a set of ongoing activities that go into maintaining a case while a child is placed in foster care or when a child is receiving in home services. These are routine case management responsibilities, many of which happen each month. The time use data indicate that it takes 21 hours of caseworker time and .5 hours of supervisor time - on a monthly basis - in order to maintain a child's placement in foster care. A non-custodial case takes about 6 hours monthly to maintain. Looking closely at figures related to children in foster care, we see that:

53

1. Almost a quarter of monthly maintenance time is dedicated to casework related to family visits (supervising, scheduling, documenting). That is, caseworkers in TN are spending, on average, 5 hours handling family visits for a single child in a one-month period.

2. During a standard month, caseworkers report spending an additional 6 hours in direct contact with children and their parent(s). The amount of time a worker spends maintaining the case monthly increases if the child has special needs (an additional 5.25 hours per month).

Ending a Case. On top of the time foster care caseworkers spend monthly maintaining one case, they report to spend an additional 8 hours on a case if it is exiting to reunification. Team Leaders are spending an additional 3 hours (on average) on activities related to the closing of that same case. When a non-custodial case closes successfully, workers spend a little over 2 hours on tasks related to the case closure.

Placement Changes / Case Transfer. Each time a child in foster care requires a change of placement (and it is unplanned), in addition to monthly time spent managing the case, workers indicate that there are 14 additional hours of caseworker time spent tending to tasks specifically related to the change of placement. There is an additional 2 hours of Team Leader time required.

1. If a child is moving to a congregate care setting or has special needs, there will be an additional 6-8 hours, on average, dedicated to executing the placement change.

2. Of the 14 caseworker hours dedicated to a placement change, nearly 4 of those hours (27 percent) are spent traveling to and from the new foster home.

3. Arranging, attending and documenting the Permanency Stability Meeting takes 2 hours of a caseworker’s time (related to one placement change).

When a child receiving non-custodial services is in need of extended in-home services, the case is transferred to FSS. The frontline CPSA worker reports spending about 4 hours on tasks related to executing this transfer while the FSS worker receiving the case will spend about 3 hours as well.

Non-Court Case Reviews (CFTMs). Executing one CFTM – including scheduling, documenting, de-briefing and attending – takes a caseworker about 4 hours (custodial and non-custodial). Team Leaders spend nearly 1.5 hours on that same meeting. Nearly one fifth of that time is spent on administrative tasks such as scheduling and securing a room.

Legal Activities. Each permanency hearing consumes a total of 7 hours of a caseworker’s time and nearly 1 hour of a Team Leader’s time. On the non-custodial side, a status review hearing takes nearly 7 hours with an additional 3 hours of supervisor time involved.

1. Other types of hearings such as Permanency Plan Ratification hearings and Status Review hearings take even more time.

54

2. Preparation of testimony and the court report (for one permanency hearing) takes about 4 hours.

3. One worker will spend nearly 2 hours waiting outside of court for a permanency hearing to begin.

The time use data collected here provides great opportunity for TNDCS and regional leaders. First, these data give TNDCS and regional leaders greater insight into how workers are currently using their time on the job. There are valuable opportunities for action such as knowing how time use patterns vary by region and understanding how, if at all, this variation maps onto regional-level variation with respect to outcomes for children and families.

There are also advanced opportunities for the evaluation team. Time use data can be merged with other data sources – such as information on staff characteristics – in order to examine the relationship between worker level characteristics, time use patterns and, potentially, child level outcomes. In Year 4 of the Waiver, Chapin Hall will re-administer the TUS, which will allow us to see how time use patterns change, if at all, as a result of the changes made under IHT.

The General Staff Survey

The purpose of the GSS is to use field-tested scales to identify community-, staff-, work unit-, and agency-level factors that may affect implementation of waiver interventions. Research suggests that the topics covered via the GSS may be important considerations with respect to the interventions’ implementation context (Baumann, Kern & Fluke, 2011). By implementing the survey near the beginning of the waiver and repeating it over the course of the waiver, we can identify the baseline for these scales, gain insight into any changes over time, and capture the perspectives of newer staff who joined the agency since the first round of the survey.

To date, two rounds of a General Staff Survey (GSS) have been implemented in the four waiver regions; the first (GSS1) occurred in June 2015, and the second (GSS2) in January, 2017. Briefly, the scales used in the survey cover such topics as:

4. Role type

5. Program area (in-home services, foster care, investigations)

6. Years working in child welfare or in current position;

7. Perceptions of workload and job satisfaction;

8. Self-assessment of case skills;

9. Perceptions of organizational culture and climate (administrative leadership, work unit cohesion, supervision);

10. Confidence in local service availability and quality;

11. Proclivity towards family preservation or child safety;

55

12. Attitudes regarding placement, reunification, and case closure;

13. Demographic characteristics (age, education, race/ethnicity).

Results from the GSS1 were used to make alterations and additions to the survey prior to the implementation of the second wave of the GSS. The factor analyses and reliability analyses indicated some items could be dropped, so for the sake of parsimony, they were eliminated on the GSS2.14 For the GSS2, additional questions about job satisfaction were added (to make the measure more robust) and also added scales concerning caseworkers’ perceptions of support or liability should something egregious happen on one of their cases. Further, since R3 implementation had begun by the time of the GSS2, questions were added asking caseworkers and supervisors for reflections on R3-related behavior and fit.

Response Rates. Table 11 presents the GSS1 and GSS2 response rates overall, and by region. The overall response rate across the four regions was 65 percent on the GSS1 (n = 217) and 79 percent on GSS2 (n = 291).15 In general, the response rate increase for the GSS2 appears to be driven largely by greater participation of staff in the East, Knox and Smoky regions.

Response to General Staff Survey, by Region Table 11. GSS1 Total GSS2 Total Region Invited16 Participated Response Invited Participated Response

East 58 36 62% 59 51 81% Knox 95 51 54% 120 91 76% Northeast 107 95 89% 119 94 79% Smoky 75 35 47% 73 55 75% Total 335 217 65% 370 291 79%

Participation

Table 12 presents the profile of respondents by region and role for each wave of the survey. As expected, given structure and staffing patterns, caseworkers and CM3s represented the vast majority of participants in both rounds of the survey. Twice as many Team Leaders (supervisors) participated in the second round of the GSS, compared to the first.

14 GSS1 results, presented here reflect revised versions of scales that were adapted to mirror those used in the GSS2. 15 Denominators shifted between the two waves as more staff were targeted for inclusion in the GSS2. 16 The numbers for staff who were invited and participated include the CPS-I staff assigned to that region.

56

Region and Role of GSS Participants, by Wave17 Table 12. Caseworkers & CM3s Team Leaders Team Coordinators

Participated % of Row

Total Participated % of Row

Total Participated % of Row

Total GSS1 Total 166 80% 26 13% 15 7%

East 24 71% 6 17% 4 12% Knox 41 85% 5 10% 2 4%

Northeast 77 82% 13 14% 4 4% Smoky 24 77% 2 7% 5 16%

GSS2 Total 202 69% 62 21% 16 5% East 36 74% 10 20% 3 6%

Knox 63 74% 18 21% 4 5% Northeast 66 73% 18 20% 6 7%

Smoky 34 65% 16 31% 2 4%

Table 13 presents the distribution or surveys from staff by wave, region and program area. As some staff, such as Team Coordinators, CQI, or training staff do not have a single program area affiliation, they are excluded from the frequencies below and any program area analyses.

Region and Program Area of GSS Participants, by Wave Table 13.

CPS-A CPS-I FSS & FCIP FSW Overall

Region Number % of Row Total

Number % of Row Total



Total Number

GSS1 Total 55 28% 30 16% 18 9% 91 47% 194 East 8 26% 7 23% 4 13% 12 39% 31

Knox 13 28% 2 4% 2 4% 30 64% 47 Northeast 28 31% 16 18% 9 10% 37 41% 90

Smoky 6 24% 5 20% 3 12% 12 46% 26 GSS2 Total 63 23% 53 20% 27 10% 127 47% 270

East 11 23% 8 17% 5 11% 23 49% 47 Knox 18 21% 17 20% 3 4% 47 55% 85

Northeast 24 28% 15 17% 15 17% 32 37% 86 Smoky 10 19% 13 25% 4 8% 25 48% 52

Scale Results

The select results from the surveys presented here provide a sketch of the staff and implementation environment in the 10 target counties. Details on the descriptive and the

17 Excludes surveys from staff not affiliated with these roles (e.g., CQI, training, etc.) and includes the centrally-managed CPS-I staff the Team Leaders overseeing investigators who are assigned to cases in the region.

57

bivariate analyses for each of the waves and the overall cross-wave scores are available in Appendix C.

The results of the bivariate analyses, presented below, reflect analyses conducted on data from both waves of the survey in order to examine the broadest representation of staff. As Appendix C indicates, overall, 76 percent (n = 377) of the 496 unique potential participants targeted for inclusion at one or both waves of the survey have submitted GSS data. In order to utilize data from both waves, a composite overall score variable was calculated. If a respondent participated only in the GSS1 or only in the GSS2, the score from that survey was utilized. For those individuals who participated in both rounds of the GSS a repeated measures analysis (the details of which are presented below) were also run in order to determine if there were changes in scores over time. If significant differences were observed between the GSS1 vs. GSS2 scale averages, the mean score of the two was calculated and used in the analysis. Where no differences over time were detected in the aggregate, the score from the GSS2 was used.

ANOVAs, chi-square analyses, and t-tests were run to test for significant differences in scores by region, program area, or role; only the statistically significant contrasts are presented below. Analyses regarding associations between survey results and case-level outcomes are in process.

Key findings from the General Staff Survey are included, below.

Caseworkers’ concerns about workload varied by region. Although levels of concern about workload were fairly moderate overall, East staff consistently indicated less concern about workload compared to staff from other regions. Knox, Smoky and Northeast staff expressed similar and higher levels of concern than East staff with respect to caseloads and their influence on job performance. Further, Northeast staff indicated more concern about keeping up with agency policies and guidelines, and both Smoky and Northeast staff expressed more concerns than East staff about being able to spend enough time with clients.

Staff with more positive views of administrative leadership reported fewer concerns about workload and higher levels of job satisfaction. Concerns about workload had a stronger, more negative association with staff perceptions of administrative leadership in the East region compared to other regions, suggesting it has more of an effect on perceptions in that region. The relationship between job satisfaction and perceptions of administrative leadership was stronger in Smoky than in the other regions suggesting the two factors are more closely tied to each other in that region.

Staff reporting less cohesion between staff in their work unit or more concerns about workload reported lower satisfaction with their job. The relationship between work unit cohesion and job satisfaction was stronger in the Knox and Northeast regions, suggesting that an increase in work unit cohesion may have a stronger effect on job satisfaction in those regions compared to East and Smoky. Workload concerns had the strongest association, or largest effect, on job satisfaction in East and the weakest effect on reports of job satisfaction by Smoky staff.

58

Perspectives about family preservation and child safety differ according to one’s role. On a scale assessing the degree to which one’s beliefs favor family preservation versus child safety (Dalgleish, 2010), scores varied by role. Scores from Team Leaders and Team Coordinators indicated they were more oriented towards family preservation, while caseworkers and CM3s favored child safety, on average.

The length of time one had worked in child welfare had an association with their perspectives about key aspects of their job. Casework staff with more years working in child welfare were associated with: less positive views on the supervision they receive and on administrative leadership; a slight leaning towards family preservation (vs. child safety); and less confidence in the support they might receive from supervisors or administrative leadership should a child on their caseload be harmed.

Caseworkers’ confidence in services meeting family needs varied by region. Knox staff consistently indicated greater confidence than staff from other regions on every service considered. Overall, staff expressed the most confidence in food pantry and parenting services and the least confidence in housing and utilities services. Compared to Knox staff: Northeast and Smoky staff were less confident about substance abuse and mental health services; Northeast and East staff were less confident in food pantry services; Smoky staff were less confident in housing services, and; each of the three other East Grand regions were less confident than Knox staff in their DV, parenting, utilities, and household management-related services.

Caseworkers anticipating higher levels of support from their supervisor and administrative leadership in a circumstance where a child on their caseload experienced harm also reflected more positively on a number of measures. Staff indicating they felt higher levels of support were associated with:

1. More job satisfaction;

2. Fewer concerns about workload;

3. More positive reflections on supervision and administrative leadership;

4. Stronger cohesion in their work unit; and

5. Having fewer years of experience with child welfare work.

The following reflections are from participants in the GSS2.

Caseworkers’ concerns about workload differed by program area. CPS-A staff expressed the most worry about workload and CPS-I the least. Looking at the questions that make up the scale, the difference between CPS-A and CPS-I staff was largely due to CPS-A staff concerns about caseload size, its effect on the quality of their work, and not having enough time with clients. FSW staff also expressed more concern than CPS-I about having enough time to spend with clients and keeping up with policies and guidelines.

59

Caseworkers’ confidence in services meeting family needs varied by program area. FSS & FCIP staff consistently indicated lower confidence than staff affiliated with the other programs on every service considered. FSW and CPS-I staff were more confident about most services compared to CPS-A and FSS & FCIP staff. Staff across the programs expressed the most similar levels of confidence with respect to substance abuse, housing, and parenting services.

Caseworkers who indicated their supervisor’s behavior during videotaped R3 group supervision was consistent with their typical supervision practice had more positive attitudes about organizational culture and climate at the agency. Caseworkers who rated their supervisor’s videotaped R3 behavior as more authentic were more likely to report: more positive reflections on their supervision; more positive reflections on their administrative leadership (including Team Coordinators and regional leadership); higher job satisfaction; and more confidence that they would receive support from their supervisors and agency leadership should a child on their caseload be harmed.

Caseworkers who indicated that their own behavior during the R3 videotaped sessions reflected their typical behavior with clients reflected more positively on their work environment. Staff indicating a closer fit between R3 principles and their actual casework practice were also more likely to report more positive scores on the supervision they receive and more positive scores on administrative leadership.

We did not find any significant variation across staff in the four East Grand regions for a number of questions on the surveys. Overall, caseworkers across the regions reported similar average scores for:

1. Their case skill self-assessments, the overall quality of the supervision they receive, and job satisfaction;

2. The amount of cohesion they feel with their work unit;

3. The degree to which they use personal experience to guide case closure decisions;

4. The degree to which they consider family and cultural dynamics prior to closing a case;

5. Any overall proclivity towards preventing removal or fostering reunification;

6. Any overall proclivity regarding family preservation or child safety;

7. The amount of support they believe they will receive or liability worry they might feel if a child on one of their cases is harmed.

Implications of the GSS Findings.

The lack of variability or differences between regions on the scales reflecting perspectives on family preservation versus child safety, preventing removal, or reunifying children to their families suggest that while there may be individual staff who think differently, on average, staff across the regions hold fairly similar perspectives on these topics. Moreover, on average, staff

60

from each of the regions indicated similar levels of satisfaction with their jobs, similar perceptions of work unit cohesion, and similar overall tenure with child welfare work. Staff in the East region indicated slightly lower concerns about workload. While statistically significant, the magnitude of the difference is not large.

Differences between staff according to the role they have in the agency (caseworker, Team Leader, Team Coordinator) or program area in which they work (investigations, assessments, custodial, etc.) may reflect both the reality of their day-to-day concerns and, perhaps, that with tenure and experience, perspectives can change.

Overall, differences between staff (whether at the region, role, or program area level) may or may not be problematic depending on the topic and its alignment with system or regional goals. Still, individual level differences reflecting extreme scores on single or multiple scales may be of concern, depending on the question in focus.

Repeated Measures Analysis Results

Because we have administered multiple rounds of the GSS2, we are able to track changes over time. Still, when considering changes in individual staff our sample is limited to those staff for whom we have a survey at both time points. Therefore, we further examined eligibility and participation across the two waves, determined by: 1) presence or absence from distributions lists supplied by administrators from the waiver regions at each survey time point, and 2) completion of a survey. As Table 1 in Appendix C indicates, fifty-five percent (n = 118) of staff eligible to participate in both surveys elected to complete both the GSS1 and GSS2 and only 15 people (7%) of those eligible to participate both times failed to participate in both rounds.

With close to half the potential wave 1 and 2 respondents missing in the data (since they did not participate in one or both surveys), results presented here should be considered preliminary and subject to response bias. That is, people who participated may not represent the views, experiences, or other characteristics of those who did not. Response bias analyses (examining caseload volume, demographics and outcome, as well as participant demographics, scale scores, etc.) are underway. Still, preliminary results suggest that, for the group that participated in both rounds, scores on most scales were not associated with significant changes over time. On the other hand, scores on some scales did. Process-oriented case skills, administrative leadership, and work unit cohesion shifted more positively and staff inclination to reunify children with their families and confidence in specific services shifted negatively between the two waves of the survey. Further details on these findings can be found in Appendix C.

Further analyses are underway to understand reasons for the shifts that could account for these findings. Whether these findings reflect aggregate shifts, or reactions to events or local issues that may be meaningful to waiver implementation at the site level is currently unknown. Additional analyses are underway to better understand changes over time, to the extent such change has been observed.

Summary

61

The results of the two rounds of the General Staff Survey have been provided to state and regional leadership in Tennessee. Feedback from leadership suggests that these findings are illuminating areas where training, support, and other interventions might strengthen staff skills, knowledge, and elevate ongoing support efforts in the regions or across work units. We are currently in the process of examining the relationship between these scores and case decisions (e.g., placement into foster care, reunification, etc.), work unit trends, and other outcomes of interest in the waiver regions. We also intend to explore with the state how unit staffing decisions are made in order to understand the extent to which staff are able to drive the type of work they do (i.e., chose the program area in which they work) in order to better understand some of the differences observed in the results.

62

Outcomes Study

At the heart of the outcomes study are seven research questions:

Key Questions

1. What is the impact of the Waiver demonstration project on the rate of maltreatment recurrence (a substantiated investigation within 12 months of a previous substantiated investigation) as well as re-report?

2. What is the impact of the Waiver demonstration project on the rate of admission to foster care for children who are the subject of a maltreatment report?

3. What is the impact of the Waiver project on the likelihood that children will experience a permanent exit within set periods of time?

4. What is the impact of TNDCS’ IV-E Waiver demonstration project on the likelihood that children in out-of-home care will experience a movement from one foster home to another?

5. What is the impact of TNDCS’ Waiver project on the likelihood that children will experience reentry following a permanent exit from care?

6. What is the impact of TNDCS’ IV-E Waiver demonstration project on the number of care days used, on average, both for children who enter placement after the implementation of the project as well as children in-care at the time TNDCS rolled out its IV-E Waiver demonstration project?

7. To what extent does the functional well being of children and families improve over the course of the IHT period?

Sample

The TNDCS Waiver demonstration project targets all children between the ages of 0 and 18 who receive non-custodial services (investigations, assessments, family support services, family crisis intervention services) and children between the ages of 0 and 21 who are placed in DCS-licensed family foster homes. The sample includes both children in care at the start of the demonstration (the legacy caseload) and all admissions involving children entering family foster care. The demonstration project officially began on October 1, 2014. We use the experiences of children who belong to the 2010 through 2013 entry cohorts as a comparison group.

TFACTS was the primary source of data to help determine whether children would be included in the demonstration (treatment) or comparison group. Table 14 enumerates the breakdown of the admissions sample (demonstration and comparison) across regions.

63

Admissions to Foster Care, by Region and Fiscal Year Table 14.

Region FY12 FY13 FY14 FY15 FY16 FY17 Davidson 526 513 475 429 457 371 East Tennessee 500 535 484 449 487 260 Knox 586 511 509 508 465 438 Mid Cumberland 839 890 744 749 788 649 Northeast 674 617 588 520 511 482 Northwest 340 325 358 353 397 246 Shelby 1,148 1,066 671 644 755 524 Smoky Mountain 736 641 559 573 609 432 South Central 459 476 356 415 541 388 Southwest 327 345 355 302 259 257 Tennessee Valley 684 680 611 568 578 457 U. Cumberland 636 691 660 635 637 395 STATE 7,455 7,290 6,370 6,145 6,484 4,899

In terms of raw totals, it appears that the total number of children coming into care has, over time, generally trended downward. We see a very modest rise in the number of children admitted to care from FY15 to FY16; FY17 is censored (censor date is March 31, 2017). We have not yet observed all FY17 entries into care.

In Table 15 we look at children in the legacy or “in-care” groups – both the demonstration group (the 2014 in-care group) and the comparison group (in-care groups 2009-2013). The designation ‘IC’ refers to children who were IN CARE on July 1 of each of the named years.

Children in care on October 1 of the year, by Region Table 15.

Region IC2009 IC2010 IC2011 IC2012 IC2013 IC2014 Davidson 453 387 396 398 460 426 East Tennessee 493 425 534 498 492 408 Knox 498 506 574 676 630 656 Mid Cumberland 794 691 726 825 825 758 Northeast 515 516 640 706 756 721 Northwest 201 232 260 305 350 384 Shelby 935 807 1,008 1,036 979 807 Smoky Mountain 594 674 790 754 782 730 South Central 490 524 521 450 491 412 Southwest 294 295 304 298 367 336 TN Valley 636 687 710 681 720 680 U. Cumberland 437 465 605 703 742 758 Grand Total 6,340 6,209 7,068 7,330 7,594 7,076

64

In contrast to what we see in terms of admissions to care, the number of children in care on October 1 of the year has generally gone up since 2009, although we do see the trend line dip down for the 2014 in care group.

Generally speaking, the make-up of children in the demonstration-period admissions groups is similar to that of the comparison group (entry cohorts FY12 through 2014; see Figure 12).

Figure 13. Age of children admitted to care, by fiscal year of admission

Children admitted to care in fiscal years 2016 and 2017 are slightly more likely to be teens than had been the case in earlier entry cohorts. Overall, children coming into care are more likely to be 4 years old and older. Babies make up the smallest proportion of entrants, year to year.

As for the children in the in-care groups, we see very stable breakdowns by age over the five years represented below, in Figure 13.

65

Figure 14. Age of children in care on in-care date (October 1), by year

The 2014 in care group has a greater proportion of 4 to 12 year olds than years past. The proportion of children that are on the younger side (three years old and younger) has remained fairly stable. Although there are slightly more teens in the 2014 in care group than the 2013 in care group, the proportion of each in care group that is teens has generally gone down over time.

Data Sources

The primary source of data for the outcomes study is TFACTS, which includes information related to children’s maltreatment history (maltreatment reports, investigations, assessments), other non-custodial services (Family Support Services, Family Crisis Intervention Program), and placements in out-of-home care. Tables from TFACTS were used to identify the group of children that would be included in the sample: (1) children age 0 to 21 who were placed in DCS-custody on or after October 1, 2014 as well as children in care ON January 1, 2014 and (2) children who are the subject of a non-custodial event on or after October 1, 2014.

Data analysis

The basic monitoring of core performance outcomes vis-à-vis the Waiver focuses on the following:

Re-reports and Recurrence

Re-reports are a key outcome associated with R3, particular as implemented in non-custodial settings. That is, R3 is hypothesized to reduce the rate of re-reports for children receiving non-

66

custodial services. As well, it is hypothesized to reduce the rate of maltreatment recurrence, which is defined as a substantiated investigation within 12 months of a previous substantiated investigation.

The Department is also looking to NPP to address issues of re-reporting and recurrence. NPP has yet to be rolled out and will be the focus of study in coming months.

Admissions

We use the placement rate to consider changes in admissions to care. The placement rate is calculated by determining the number of children placed into out of home care per 1,000 children in the population. The placement rate is stratified by fiscal year of admission, by region, and by children’s age at entry.

Placement stability

The method we use to measure placement stability is the cumulative probability, here in accumulating 30-day intervals. The cumulative probability answers the question, what is the probability that a child will experience an initial placement move in the first 30 days of their foster care spell, the first 60 days of their spell, the first 90 days of their spell, and so on. We focus on the first-ever placement change because the best way to prevent children from serial moves during their foster care spell is to avoid the first-ever move. Understanding when the probability of that first-ever move is highest gives leadership information on which they can act to try and get ahead of those experiences and avoid them altogether. Data on the cumulative probability of an initial placement change is organized around children’s age at the time their spell began.

Note the data presented here is on a subset of the full sample: children experiencing their first-ever placement in out of home care.

Permanency

Quartile duration is a measure of permanency that describes the number of days it takes 25, 50, and 75 percent of an entry cohort to leave care. The 50 percent quartile duration is referred to as the median. We use median duration as the first measure of permanency.

The cumulative probability of a permanent exit is the second measure used to understand changes in permanent exits for children admitted to care. Probabilities are considered in six-month intervals. The cumulative probability is organized around the age of children in the sample at the time their child spell began. For permanent exits we are including reunification, adoption, and discharges to relatives.

For the in-care group, we use the median residual duration as a measure of permanency. The median residual duration tells you how long, in days, it takes 50 percent of a group of children to leave foster care. We organize these data according to the length of time members of the various in-care groups (2010 through 2013 for the comparison; 2014 for the IHT group) had

67

been in care as of July 1 of the given in-care year: zero to six months; six to 12 months, 12 to 18 months, 18 to 24 months, and 24+ months.

Reentry

To measure reentry we use the conditional probability of reentry, organized in six-month intervals and presented by exit year and age at exit. Here, the question we are answering is, what is the probability that a child will reenter care in the first six months after their exit from care? For children who have not yet reentered after six months, what is the probability they will reenter in the next six-month interval (and so on)? Using small intervals of time makes it easier for agencies to consider what might be driving the reentries, when they do occur.

To assemble the reentry-specific analytic file we focus on children who exit from foster care; that is, we use exit cohorts rather than entry cohorts. As noted above, we re-categorized the age variable as well, so that here we are focusing on the child’s age at the time of their discharge from care rather than their age at the time of their entry into care.

Care day count

At the heart of the Waiver demonstration model is the goal of reducing the number of foster care days systems use, which is, of course, simply the aggregate form of reducing the number of foster care days individual children are using. We provide counts of care days for the FY2015 entry cohort, the FY2016 entry cohort, the FY2017 entry cohort and the 2014 in-care group. For each group we also include a historical comparison of care day usage.

Conceptually, what we do with each entry cohort and for the single in-care cohort is identify the group of children who entered care in the relevant time period (within each fiscal year for the entry cohorts, and in-care on July 1 of the year for the in-care groups) and who were eligible for IHT and then watch them flow out of care over the course of the Waiver period (until September 30, 2019). Each performance year starts with a group of children. For the entry cohorts, the first performance year starts with all of the eligible children admitted to care during that fiscal year. For each subsequent performance year, the performance year starts with all of the children (of those admitted in the first performance year) still in care at the beginning of the next performance year. A similar approach holds for the in care group. The first performance year starts with all of the children in eligible placements/agencies on July 1 of the given year. Each subsequent performance year starts with the children still in care (of those who started out in the first performance year) at the beginning of the next performance year.

For the in-care group we structure the strata (sub-categories) a little differently. Typically, the strata are focused on the age of the child at the time they entered placement. We use this organizational frame for the in care group (as we do for the entry cohorts) but with an additional strata that tells us how long children between the ages of 4 to 12 (a specific target of the IHT demonstration) had been in care as of July 1 of the given year: 4 to 12 year olds who had been in care for less than two years as of July 1 of their in-care year and those who had been in care for more than two years as of July 1 of their in-care year. We do this out of

68

recognition that longer-stayers tend to have a different set of trajectories than children who have not yet accumulated as much time in care.

Results

Because the demonstration project is still somewhat early on in the implementation of core evidence-based interventions and practice models, we offer only preliminary findings with respect to specific program effects. We also offer general performance trends over time with a focus on the outcomes of particular interest to TNDCS, given the goals of the demonstration.

As described above, the core outcomes of central concern under the demonstration project include admission to foster care, placement stability, permanency, reentry, and care day utilization. We take each one in turn in the sections that follow.

Maltreatment Recurrence

The tables and figures below provide information on the extent to which children who are the subject of a substantiated investigation have another maltreatment experience (again the subject of substantiated investigation) within 12 months of the initial event. Table 16 provides the regional view over the last several fiscal years.

Recurrence of Maltreatment, by Region and Fiscal Year Table 16. FY12 FY13 FY14 FY15 FY16 FY17 STATE 5% 5% 7% 6% 5% 2% Davidson 4% 4% 5% 5% 4% 2% East Tennessee 5% 8% 8% 7% 5% 3% Knox 6% 4% 6% 7% 7% 2% Mid Cumberland 5% 6% 8% 6% 6% 1% Northeast 7% 6% 7% 8% 6% 2% Northwest 5% 6% 6% 6% 5% 2% Shelby 4% 5% 6% 7% 4% 1% Smoky Mountain 3% 8% 8% 7% 6% 4% South Central 6% 5% 6% 7% 4% 2% Southwest 7% 5% 3% 4% 3% 3% Tennessee Valley 5% 5% 7% 6% 3% 1% Upper Cumberland 6% 4% 8% 5% 4% 2%

The shaded columns reflect censoring: not all children who were the subject of substantiated investigation in FY16 have had 12 months within which to experience another substantiated investigation. The FY17 cohort is still incomplete (censor date = March 31, 2017).

Overall, recurrence rates are fairly stable, ranging between three and eight percent (excluding FY17, which is heavily censored). At the same time, there is discernible regional variability.

Figure 15, below, shows recurrence rates by age.

69

Figure 15. Recurrence of Maltreatment, by Age and Fiscal Year

The dip in FY16 is likely due to censoring; not all children who were the subject of a substantiated maltreatment report in FY16 have had 12 months within which to experience a subsequent substantiation. Otherwise, it appears that recurrence has been on the rise for toddlers and school-age children. Babies also saw a significant increase in recurrence rates from FY12 to FY14; those rates came down a bit in FY15. For teens the picture is a bit uneven, with spikes and dips over the years. Still, in FY14 and FY15 teens were the least likely to experience maltreatment recurrence.

R3 and Re-Reports

The purpose of the R3 impact analysis is to gauge whether the likelihood of re-report changed after the R3 project was implemented. To measure the pure impact of the R3 project, it is crucial to separate confounding factors which can also influence re-report outcomes. Multiple empirical analysis frameworks have been in place to deal with confounding factors that come from multiple sources: child, supervisor, county, and region. First, child clinical differences and the nested structure of child-county-region were adjusted. Second, an intent-to-treat analysis was conducted to overcome potential supervisor selection bias. Third, comparative interrupted time series (ITS) was implemented to address any remaining selection bias.

The final model chosen to evaluate the impact of R3 was a comparative ITS model. However, before the comparative ITS model was built, the difference-in-difference (DID) model was built for comparison. The DID model is pretty similar to the comparative ITS model; it is largely a simplified version of comparative ITS model. The DID model observes the average difference before and after treatment periods in both treatment and control groups without considering differential trends in treatment and control groups. The key difference between the DID and the

70

comparative ITS is that DID does not consider growth patterns because it considers only average gains or losses after a treatment in both the treatment and control groups. However, the comparative ITS design assumes different growth patterns in both groups. Since the comparative ITS model incorporates a more complex model design, it is considered more rigorous.

In Figure 16 we display patterns in re-reports for the R3 target counties (‘R3 Region’) and the comparison areas (‘Non-R3 Region’). Rates of re-report were on the decline in both groups; children in the R3 counties have had higher rates of re-report rates than children in the non-R3 counties.

Figure 16. Trends in Re-Reports: R3 and Non-R3 Areas

After R3 implementation (to the right of the bold line), both groups show lower rates of re-report. After examining the results, it is not clear whether the deviation from previous re-report trends in the R3 region is greater than that observed in the non-R3 region. Interrupted Time Series analyses are used to formally test these hypotheses.

The results of the ITS model indicate that R3 had the effect of decreasing rates of re-report by six percent (Estimate=-0.06). However, the corresponding p-value was non-significant (p=0.155). The conclusion at this point is that the R3 impact of a six percent drop in rates of re-report needs further investigation, with additional data (i.e., more time), before we can determine whether there is, indeed an R3 effect on re-reports.

71

Admissions: Placement Rates

The focus in this section is on admissions to care. Recall that part of the IHT demonstration project involves the use of the FAST to better (and more quickly) identify families’ needs so as to make the necessary linkages with community resources, thereby reducing the need for placement in foster care. Two views are presented here. The first looks at differences in placement rate by region between FY12 and FY16 (FY17 is censored).

Figure 17. Placement Rate per 1,000, by Fiscal Year and Region

The black dotted line represents the statewide placement rate. Figure 17 makes clear the variation in admissions by region, with Upper Cumberland well above average (more placements per 1,000 children in Upper Cumberland compared to the state) and Mid Cumberland below average (fewer placements per 1,000 children in Mid Cumberland compared to the state).

There is also considerable variation in admission practices by age.

72

Figure 18. Placement Rate per 1,000, by Fiscal Year and Age at Entry

Year after year, the placement rate is considerably higher for babies compared to children age 1 and older. Four to 12 year olds have the lowest placement rate. Placement rates have been fairly stable over time with the exception of children under 1 year of age, who experienced a significant drop in the placement rate around FY15.

Placement stability

In this section we provide data related to the stability of children’s placements. The focus is on children in the admissions groups: the demonstration-period entry cohorts and the comparison entry cohorts. We further hone in on children experiencing their first placement in care. We do this because the relative risk of placement disruption is often different for children in their subsequent spell.

In Table 17 (below) we look at the cumulative probability of an initial placement change for children in their first-ever spell. The rows represent the cumulative probability that children within each fiscal year entry cohort experienced an initial move within particular windows of time (columns).

Cumulative Probability of Initial Move: First Ever Spell, by Fiscal Year of Entry Table 17.

1 Day 1 - 7 Days

1 - 30 Days

1 - 60 Days

1 - 90 Days

1 - 180 Days

1 - 180 + Days

FY12 8% 15% 23% 28% 30% 38% 48% FY13 9% 16% 24% 28% 31% 38% 48% FY14 8% 17% 24% 28% 31% 38% 48% FY15 8% 14% 23% 28% 31% 39% 50% FY16 11% 21% 27% 33% 36% 43% 51%

73

Generally, the probability of a child having their first ever placement experiencing a placement change within 180 days of their entry in care has gone up slightly over the past few years. However, the likelihood of a very early placement change is up quite a bit in the most recent fiscal year for which we have full data (FY16): 21 percent of children in their first ever foster care spell experienced a move within the first week of custody.

In the figure below we see how the likelihood of experiencing a placement change varies depending on how old children are at the time they enter care.

Figure 19. Cumulative Probability of Initial Placement Change, by Age at Entry

In Figure 19 we are looking at just one fiscal year: FY1516. Teens have the highest probability of experience a move within 180 days; toddlers, the lowest probability. Approximately 55 percent of teens who entered care in FY1516 experienced a move within 180 days of their placement; for toddlers, the figure is just above 30 percent.

We might also expect there to be regional variation in the likelihood that a child will experience an initial move – and within distinct intervals of time. Figure 20 provides the regional view looking just at FY1516.

74

Figure 20. Cumulative Probability of Initial Placement Change, by Region (FY1516)

The bold black line represents the statewide values. As with the placement rate, there is clear indication of regional variability in the likelihood that a child will experience an initial placement change in 90 days. Children experiencing their first ever placement in Mid Cumberland appear to be more likely, on average, to experience an initial move within the first 90 days of their placement (48 percent of the children who entered in FY1516); children experiencing their first ever placement in Northwest appear relatively less likely to experience an initial move within the first 90 days of their spell (25 percent of the children who entered in FY1516).

In general, this measure of placement stability is useful in that it points fairly clearly to the potential value in revisiting placement policies, in that a considerable proportion of moves are happening very early on – within the first 30 days of placement. Moreover, it may be worth focusing those efforts in particular regions to start – places where there is evidence that children are at increased risk of an early move.

Permanency

In this section we look separately at children in the admissions/entry groups and children who were in care at the time the demonstration project took effect (the in-care group).

Admissions/Entry Cohorts. Permanent exits are typically defined as either reunification, discharge to relatives, or adoption. All other exit types are generally considered non-permanent exits.

75

The first measure of permanency is the median duration, or the number of days it takes 50 percent of each entry cohort to leave care. Figure 21 provides the regional view over the last five fiscal years.

Figure 21. Median Duration, by Fiscal Year of Entry and Region

Like with admissions and movements while in care, there is a fair amount of regional variability in the length of time it takes children to leave foster care. The regions also vary quite a bit year to year. For example, it took 452 days for 50 percent of the FY16 entry cohort from Smoky Mountain to exit care – longer than in any of the other regions, by far. However, it took just 350 days – under one year - for 50 percent of the FY12 entrants in Smoky Mountain to leave care; this was still above average (average = 305 days) but not the longest (Knox, 418 days for the FY12 cohort).

There is also important variability in the number of days it takes children to leave care with respect to age; that is, how old children are at the time they enter care.

76

Figure 22. Median Duration, by Fiscal Year of Entry and Age at Entry18

Babies have historically taken the longest to leave care (FY12 through FY15) although we see that trend shifting in the most recent fiscal year for which we have data (FY16): toddlers and, even more so, four to 12 year olds are taking longer to leave care. Teenagers consistently have the shortest median duration compared to the other age groups.

The second method used for understanding changes in lengths of stay over time is the cumulative probability. Data is organized around two main strata: age at entry and region.

Table 18 presents the cumulative probability of a permanent exit in accumulating six-month intervals.

18 The shading for 4 to 12 year olds in the FY16 cluster is different to reflect the fact that 50 percent of 4 to 12 year olds admitted in FY16 have not yet exited care.

77

Cumulative Probability of a Permanent Exit, by Fiscal Year of Entry and Age at Entry19 Table 18. Under 1 1 to 3 Years

Entry 0-6 m. 0-12 m. 0-18 m. 0-24 m. 0-30 m. 0-36 m. 0-6 m. 0-12 m. 0-18 m. 0-24 m. 0-30 m. 0-36 m.

Fiscal Year FY2012 35% 52% 68% 81% 88% 92% 36% 55% 70% 78% 84% 89% FY2013 38% 56% 69% 78% 86% 92% 37% 58% 71% 79% 86% 87% FY2014 37% 56% 70% 81% 90% 37% 52% 67% 78% 85% FY2015 31% 45% 63% 31% 52% 65% FY2016 31% 34%

4 to 12 Years 13 to 17 Years Entry

0-6 m. 0-12 m. 0-18 m. 0-24 m. 0-30 m. 0-36 m. 0-6 m. 0-12 m. 0-18 m. 0-24 m. 0-30 m. 0-36 m. Fiscal Year

FY2012 39% 57% 64% 68% 70% 71% 30% 70% 81% 85% 87% 88% FY2013 38% 56% 64% 68% 70% 70% 31% 73% 84% 89% 90% 90% FY2014 35% 51% 59% 65% 66% 31% 71% 85% 89% 91% FY2015 37% 55% 62% 31% 73% 84% FY2016 34% 33%

On average, between 50 to 60 percent of children age 0 to 12 have a permanent exit within 12 months of entry. For teens the probability is even higher – between 70 to 75 percent. Figure 23 provides a clearer view.

19 Shaded cells denotes censoring.

78

Figure 23. Cumulative Probability of Permanent Exit in 12 Months, by Fiscal Year of Entry and Age at Entry

In the most recent fiscal year for which we have complete data, babies had a reduced probability of a permanent exit within 12 months of entry, compared to FY12 through FY14. For children age 1 to 12 years the probability of a permanent exit within 12 months dropped a bit in FY14; while the trend line is pushing back up for children age 4 to 12 in FY15 it has remained mostly flat for toddlers (children age 1 to 3 at entry).

Figure 24 provides the regional view.

79

Figure 24. Probability of Permanent within 12 Months of Entry, by Fiscal Year of Entry and Region

Statewide, between 42 to 49 percent of children have a permanent exit within a year of their entry. There is considerable variation across regions and over time. In the most recent fiscal year for which we have data (FY15) the proportion of the entry cohort that had a permanent exit within 12 months ranged from a low of 30 percent (South Central) to 63 percent (East Tennessee). However, just three years prior South Central was closer to the top of the range (55 percent of the FY12 entry cohort had a permanent exit in 12 months).

In-Care Groups. Children in the in-care groups have, of course, a varied set of experiences with regards to the “amount” of foster care they have accumulated as of July of the given in-care year. The median residual duration analysis, the results of which are presented below, take this reality into account.

Median residual duration, in days, by in-care year and time in care Table 19.

In Care Year 0-6 Months 6-12 Months 12-18 M h

18-24 M h

2+ Years 2010 2,824 3,457 3,487 2,665 3,448 2011 2,966 3,364 3,409 3,782 3,888 2012 3,331 3,807 3,231 3,306 3,286 2013 3,324 3,420 3,850 3,944 3,296 2014 3,512 3,416 3,537 4,065 3,991

80

Comparing the 2014 (demonstration-period) group to the preceding in-care years, we see a mixed set of findings in the number of days it took 50 percent of the in-care group to leave foster care, given the amount of time they had spent in care prior to July 1 of the in care year. The graphic view, in Figure 25, brings the narrative out a bit more clearly.

Figure 25. Median Residual Duration, by Time in Care and In Care Group

As just described, we see in Figure 25 the rising trend in the number of days it takes 50 percent of longer-stayers within each in-care cohort to leave foster care. The same holds true, albeit modestly, for children who were fairly new to foster care when July 1 hit (the 0-6 month group). At the same time, it has generally taken the same/less time for 50% of children who had been in care for 18 months or less as of July 1 of the given year to exit foster care.

Reentry

A single measure was used to monitor the extent to which children who exit from foster care reenter at some future date: the conditional probability of reentry, in six-month intervals. This is a useful way to think about reentry because it tells you not only how likely reentry is but when it is most likely to occur, so that TNDCS and its regional leadership can be strategic in the nature and timing of the support they offer. Table 20, below, displays the conditional probability of reentry for children in the series of admissions cohorts at the heart of the analysis (three historical comparison cohorts and the FY2015 and FY2016 IHT cohorts).

Overall we see fairly low levels of reentry across the board. Children who exit as teens are consistently the most likely to reenter care.

81

Conditional Probability of Reentry, by Exit Year and Age at Discharge: Six-Month Intervals Table 20.Under 1 1 to 3 Years

Exit Year 0-6 6-12 12-18 18-24 24-30 30-36 0-6 6-12 12-18 18-24 24-30 30-36

FY2012 3% 4% 1% 2% 2% 1% 2% 3% 4% 2% 1% 2%

FY2013 3% 3% 2% 0% 2% 0% 4% 3% 2% 1% 2% 1%

FY2014 4% 2% 3% 2% 1% 1% 3% 3% 3% 2% 1% 0%

FY2015 3% 3% 1% 0% 3% 3% 2% 1%

FY2016 3% 1% 2% 3%

4 to 12 Years 13 to 17 Years

Exit Year 0-6 6-12 12-18 18-24 24-30 30-36 0-6 6-12 12-18 18-24 24-30 30-36

FY2012 4% 4% 4% 1% 1% 2% 8% 9% 6% 5% 3% 1%

FY2013 4% 4% 2% 2% 1% 1% 6% 9% 6% 5% 3% 2%

FY2014 3% 3% 3% 2% 1% 1% 8% 7% 4% 5% 3% 2%

FY2015 4% 3% 3% 2% 9% 9% 6% 4%

FY2016 4% 3% 7% 8%

82

Figure 26, below, shows the probability of reentry in 6 months, by exit year and region.

Figure 26. Reentry Within 6 Months of Discharge, by Exit Year and Age at Discharge

Again, we see low levels of reentry within six months. We also see the familiar regional variation. There are regions that have seen big improvements in the rate of reentry within six months (Northwest, down from 14 percent in FY12 to 4 percent in FY15); for the most part, regions are fairly stable in their performance on this measure over time.

83

Care Day Count

The purpose of this section is to detail the utilization of care days by children eligible for IHT and a historical comparison group of children (children from previous entry cohorts who also meet the eligibility criteria for IHT). As noted above, we do this separately for each IHT cohort: IHT admission cohort FY2015 compared to historical entry cohorts; IHT admission cohort FY2016 compared to historical entry cohorts, and the single IHT in-care group compared to historical in-care groups.

FY2015 Entry Cohort.

We compare care day utilization for the FY2015 entry cohort with the average of three historical entry cohorts: FY2010, FY2011, and FY2012. Table 21, below, delineates the starting population – the number of children admitted in FY2015 (and the average number of entrants in three consecutive historical entry cohorts), broken down by the age at spell start. Starting with Year 2, the starting population represents the number of children from the original entry cohort still in care at the beginning of the performance (Waiver) year. For each group we report the percent that exited within the given Waiver year. The percent exited is always the percent of the original. Last, we report the total number of care days and the average number of care days.

84

Exits, Average Care Days, and Total Care Days: FY2015 Admissions Cohort (IHT), by Age at Spell Start Table 21.

Starting Population Percent Exited

Average Care Days Total Care Days (% of Initial Pop)

Age Categories

Waiver /Perf. Dates Baseline Actual Baseline Actual

(th. 3/31/17) Baseline Actual (th. 3/31/17) Baseline Actual

(th. 3/31/17) Year

Under 1

Year 1 7/1/14 - 6/30/15 664 664 27% 23% 145 150 96,389 99,722

Year 2 7/1/15- 6/30/16 484 510 37% 40% 267 275 129,067 140,322

Year 3 7/1/16 - 6/30/17 234 242 22% 19% 237 201 55,429 48,555

1 to 3 Years

Year 1 7/1/14 - 6/30/15 825 825 31% 25% 137 135 112,641 111,362

Year 2 7/1/15- 6/30/16 561 615 35% 36% 262 271 146,965 166,915

Year 3 7/1/16 - 6/30/17 268 315 19% 17% 244 204 65,388 64,210

4 to 12 Years

Year 1 7/1/14 - 6/30/15 1,705 1,705 32% 26% 138 143 235,837 243,245

Year 2 7/1/15- 6/30/16 1,166 1,259 34% 35% 262 272 305,884 342,096

Year 3 7/1/16 - 6/30/17 580 669 17% 14% 264 218 152,934 145,946

13 to 17 Years

Year 1 7/1/14 - 6/30/15 1,474 1,474 32% 29% 126 130 185,884 191,445

Year 2 7/1/15- 6/30/16 865 909 28% 30% 217 213 188,079 193,584

Year 3 7/1/16 - 6/30/17 283 292 7% 5% 230 201 65,124 58,744

The grey shading in Year 3 reflects the fact that Year 3 is censored; we have not yet had the opportunity to observe all Year 3 activity, just activity through March 31, 2017. Generally, we see while Year 1 performance (with respect to the proportion of the population that experienced a permanent exit within the year window) fell short of meeting baseline expectations, Year 2 performance was notably better across age groups. The actual proportion of permanent exits in Year 2 was higher than baseline expectations across the board.

85

Table 22 and Figure 27 provide additional views, collapsed across age categories.

Exits, Average Care Days, and Total Care Days: FY2015 Admissions Cohort Table 22.(IHT), by Performance/Waiver Year



Baseline Actual Baseline Actual

(th. 3/31/17)

Baseline Actual

(th. 3/31/17)

Baseline Actual

(th. 3/31/17)

Year 1 6,011 6,011 31% 27% 137 140 823,660 840,737

Year 2 3,928 4,138 36% 37% 235 238 922,510 984,330

Year 3 1,519 1,632 13% 11% 229 202 347,982 329,397

Children admitted in FY2015 (Year 1 of the demonstration project) were slightly less likely to have a permanent exit within the first year as children in the comparison/baseline condition; similarly, they used slightly more care days, on average, than had historically been the case. Children admitted in FY2015 who were still in care at the beginning of FY2016 (Year 2 of the demonstration project) were just slightly more likely to have a permanent exit in FY2016; they used, on average, slightly more care days during the year.

Figure 27 displays the picture of care day utilization for children admitted in SFY2015.

Figure 27. Total Care Days: FY2015 Admissions Cohort and Comparison Cohorts (Baseline), by Performance Year

The total number of care days being used year to year is a bit higher than had been the historically. However, as we saw above in the section on admissions, the number of children being admitted to foster care is also higher now than it had been historically.

86

FY2016 Entry Cohort

We have a smaller observational window for the FY2016 entry cohort than we did for the FY2015 entry cohort: just 1.5 years.

Exits, Average Care Days, and Total Care Days: FY2016 Admissions Cohort (IHT), by Age at Spell Start Table 23.



Age Categories Perf. Year Dates Baseline Actual Baseline Actual (th.

3/31/17) Baseline Actual (th. 3/31/17) Baseline Actual (th.

3/31/17)

Under 1

Year 2 7/1/15 - 6/30/16 711 711 27% 26% 145 159 103,212 113,069

Year 3 7/1/16 - 6/30/17 517 525 37% 26% 267 223 137,974 117,204

Year 4 7/1/17 - 6/30/18 - - - - - - - -

1 to 3 Years

Year 2 7/1/15 - 6/30/16 801 801 31% 29% 137 149 109,364 119,561

Year 3 7/1/16 - 6/30/17 549 565 35% 19% 262 234 143,700 132,061

Year 4 7/1/17 - 6/30/18 - - - - - - - -

4 to 12 Years

Year 2 7/1/15 - 6/30/16 1,697 1,697 32% 28% 138 143 234,730 243,409

Year 3 7/1/16 - 6/30/17 1,157 1,209 34% 23% 262 219 303,398 265,153

Year 4 7/1/17 - 6/30/18 - - - - - - - -

13 to 17 Years

Year 2 7/1/15 - 6/30/16 1,733 1,733 32% 28% 126 130 218,547 224,527

Year 3 7/1/16 - 6/30/17 1,010 1,080 28% 20% 217 193 219,735 208,214

Year 4 7/1/17 - 6/30/18 - - - - - - - -

87

Across all age categories, the proportion of FY2016 admits who had a permanent exit within FY2016 was a bit lower than had been the case historically. These children also used slightly more care days, on average (and more care days, in total). It is yet too soon to remark on Year 3 performance (i.e., the proportion of the FY2016 cohort who experiences an exit in FY2017).

As with the FY2015 entry cohort, a condensed view that looks at performance years (no longer broken down by age) helps make clear the difference in care day utilization (the FY2016 entry cohort compared to the historical comparison groups; see Figure 27).

Exits, Average Care Days, and Total Care Days: FY2016 Admissions Cohort (Demonstration), by Performance/Waiver Year Table 24.



Baseline Actual Baseline Actual (th. 3/31/17) Baseline Actual

(th. 3/31/17) Baseline Actual (th. 3/31/17)

Year 1 4,942 4,942 31% 29% 174 179 861,060 885,907

Year 2 3,928 4,138 35% 25% 244 208 957,876 858,959

Year 3 - - - - - - - -

Similar to the view above, we see lower than expected rates of permanency and slightly higher care day utilization.

88

Figure 28. Total Care Days: FY2016 Admissions Cohort and Comparison Cohort (Baseline), Year 1

In Care Group

Here we are comparing care day utilization by children in care on July 1, 2014 to children in care on July 1 of three consecutive historical in-care groups (2010 through 2012). Because all children in the starting population (all children in care on July 1) are all already in care we would expect to see higher overall care day utilization for this group compared to the entry cohorts described just above, where children are entering care at different times throughout the year.

As Table 25 (below) displays, the starting population for each age subgroup is larger than what we observed for the entry cohorts. We also see, as expected, a higher overall accumulation of care days by the 2014 in care group and the historical in care groups (on average).

89

Exits, Average Care Days, and Total Care Days: In Care on July 1 (Demonstration), by Age and Time in Care Table 25.



Age Categories

As of 1/1 Perf. Yr Dates Baseline Actual Baseline Actual (th.

3/31/17) Baseline Actual (th. 3/31/17) Baseline Actual (th.

3/31/17)

Under 1 Year All prior histories

Year 1 7/1/14- 6/30/15 969 969 56% 55% 252 257 243,708 248,617

Year 2 7/1/15-6/30/16 425 435 28% 31% 228 222 97,055 96,430

Year 3 7/1/16-6/30/17 153 136 11% 7% 207 193 31,676 26,300

1 to 3 years All prior histories

Year 1 7/1/14- 6/30/15 1,036 1,036 55% 54% 251 257 259,666 266,284

Year 2 7/1/15-6/30/16 469 476 27% 29% 238 239 111,688 113,689

Year 3 7/1/16-6/30/17 190 176 12% 9% 229 189 43,473 33,285

4 to 12 Years

LT 2 Years

History

Year 1 7/1/14- 6/30/15 1,866 1,866 51% 47% 261 270 486,421 504,186

Year 2 7/1/15-6/30/16 913 985 24% 27% 262 269 239,264 265,253

Year 3 7/1/16-6/30/17 455 486 12% 11% 264 216 120,211 104,752

GT 2 Years

History

Year 1 7/1/14- 6/30/15 740 740 43% 40% 266 278 196,504 205,903

Year 2 7/1/15-6/30/16 399 424 19% 40% 274 279 109,365 118,410

Year 3 7/1/16-6/30/17 232 243 9% 15% 280 230 65,042 55,975

13 to 17 Years

All prior histories

Year 1 7/1/14- 6/30/15 1,460 1,460 42% 40% 217 233 317,408 340,015

Year 2 7/1/15-6/30/16 523 558 11% 11% 223 231 116,664 128,787

Year 3 7/1/16-6/30/17 176 202 3% 2% 223 178 39,202 35,961

90

By and large, performance for the 2014 in care group is somewhat similar to what was observed previous (particularly the FY2015 entry cohort): less-than-baseline permanency in the first performance year, and better than baseline performance in the second performance year.

Table 26 provides the collapsed view.

Exits, Average Care Days, and Total Care Days: In Care Groups, by Performance/Waiver Table 26.Year

All Children Baseline Actual Baseline

Actual (th.

3/31/17) Baseline

Actual (th.

3/31/17) Baseline

Actual (th.

3/31/17)

First Year 7,189 7,189 52% 51% 238 245 1,713,154 1,763,112

Second Year 2,965 3,073 19% 21% 244 247 722,340 758,656

Third Year 1,350 1,289 8% 6% 245 205 330,600 264,366

Year-to-year, the total number of care days is a bit higher for the 2014 in care group compared to the historical in care groups. Average care day utilization is also a bit higher. The magnitude of the difference in total care day utilization between the 2014 in care group and the historical comparison groups (on average) is laid out in Figure 29, below.

Figure 29. Total care days: 2014 In Care Group and Comparison Cohorts (Averaged), by Year

^Third year is censored

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

1,600,000

1,800,000

2,000,000

First Year Second Year Third Year^

Care

Day

s

Baseline

Actual

91

Cost Study

As discussed earlier in this report, the general purpose of the federal government granting Title IV-E Waivers is to provide jurisdictions the opportunity to use Title IV-E funds more flexibly on behalf of children and families. As part of its Waiver Initiative, TNDCS agreed to replace fee-for-service Title IV-E reimbursement with a fixed payment. Like other Waiver-involved jurisdictions, TN is trading guaranteed, unlimited, fee-for-service federal contributions for IV-E eligible children in exchange for a fixed amount of money that can be used for all child welfare services for any child, regardless of their eligibility. The fixed amount of money provided is intended to be the same amount as what TN would have received under normal Title IV-E reimbursement rules (i.e., in the absence of the Waiver). The allocation amount is based on historical average gross expenditures for foster care maintenance and foster care administration.

The basic thesis underlying TN’s Waiver project is that the constellation of interventions made possible through the flexible use of Title IV-E funds will improve child safety, permanency and well-being. Theoretically, these outcomes would reduce overall foster care spending. There are several ways to achieve a reduction in foster care spending: by reducing the number of children coming into care, by moving children out of care more quickly, and/or by reducing the cost of placements, most likely through the utilization of less expensive types of placements. Whether this reduction in foster care spending is happening and the impact of any observed changes are having on other child welfare spending is the basis of the discussion below.

As detailed earlier in this report, TN agreed to implement three interventions (and one implementation support, R3) as part of its Waiver initiative, which went into effect on October 1, 2014:

1. FAST: A standardized risk assessment protocol administered to all families coming into contact with non-custodial program areas

2. KEEP: An evidence based program that provides support and education to foster and kinship parents

3. NPP: An evidence based parenting education program

4. R3: An evidence-informed implementation support model that provides guidelines for supervisors in their interactions with caseworkers and, subsequently, caseworkers interacting with parents/caregivers

It is important to note that all interventions are not being rolled out throughout the entire state. KEEP is implemented in six regions, and R3 is currently being implemented in 10 select counties within four of the twelve regions. The FAST, on the other hand, has been rolled out statewide for staff working in all non-custodial program areas. NPP is scheduled to go into effect in July 2017, following the submission of this report.

This cost study will explore questions related to spending patterns before and during the Waiver. This is a system level study (as opposed to a regional or county based view) that will present the analysis of fiscal expenditure data collected from state fiscal years 2012 – 2016. First, the cost study provides a description of TN’s funding structure and gives some background on how the Waiver is intended to impact spending. Then, we provide an overview of the data sources used for this portion of the study along with an explanation of our analytic methods. This is followed by findings related to overall child welfare spending, foster care spending and the unit cost of foster care. Lastly, we provide summative observations and detail ongoing work and next steps as it relates to the TN Cost Study.

92

Background

Tennessee’s child welfare system is state-administered by the Department of Children‘s Services (DCS). Child welfare services are provided by a combination of state and private entities while DCS maintains sole responsibility for child protection investigations. Under a Performance Based Contracting system that was phased in over three years starting in FY07, private providers in TN receive base contract payments on top of which they are paid incentives (or are charged penalties) for their performance on select measures. The contracts are intended to incentivize reductions in average care days provided these are achieved through exits to permanency that are maintained over time. While this arrangement is not tied to the Waiver directly, it is important to keep in mind because it demonstrates TN’s familiarity with the benefits of flexible funding, and their early interest in developing financing structures that incentivizes the outcomes state leadership was seeking for children entering foster care – namely timely permanency.

According to their most recent Annual Report, TNDCS managed a budget of $736,194,600 in SFY16. Funding for TNDCS is derived from both federal and state sources; no local dollars are used. State appropriations made up 43 percent of total budgeted funding in SFY16. Federal funds, including Title IV-E, Title IV-B, and Social Services Block Grant made up 21 percent of total budgeted revenue. TennCare, which is the state of Tennessee’s Medicaid program, contributed 34 percent of revenue while the remaining 2 percent was covered by other sources (i.e.: non-governmental, education funding)20.

This phase of the cost study will answer the primary question of whether the fiscal stimulus of the Waiver and the associated service interventions had an effect on statewide expenditure patterns. To answer this question, the evaluators collected expenditure data from TNDCS dating back five full fiscal years. The cost study presents the analysis of fiscal data collected from state fiscal years 2012, 2013 and 2014 (13 quarters prior to the beginning of the Waiver) through state fiscal years 2015 and 2016 (7 quarters into the Waiver). Data collected to date will allow for the evaluation of child welfare spending patterns prior to and during the early part of the Waiver demonstration period. As part of the system-level expenditure analysis, the cost evaluation will identify and examine spending associated specifically with foster care. Since a goal of the Waiver is to increase permanency and reduce reentry, it would follow that foster care spending would be reduced as well.

Data Sources

The central task of this analysis was to create a database that includes all child welfare expenditures. The database constructed for the cost study represents all child welfare related expenditures for the past five full fiscal years, SFY 2012 through SFY 2016. The database’s structure has the flexibility to compare financial data within TN, across fiscal years, and within specific expenditure categories.

Expenditure patterns are the focus of this phase of the cost study. The revenue data is in need of additional disaggregation and so the focus for this phase of work is on spending only. Revenue data will be incorporated into the database and will contribute to the analysis for the Final Report.

There is typically a set of qualitative data collection activities that precede the collection of fiscal data for a cost study of this nature. In this circumstance, the need for that work was minimal. Through

20 Tennessee Department of Children’s Services Annual Report, State Fiscal Year June 2015 – June 2016. Issued December 2016 by Tennessee Department of Children’s Services.

93

collaborative work on TN’s Performance Based Contracting initiative as well as other projects, Chapin Hall has a rich history of involvement with TNDCS and, in particular, the fiscal department. As such, members of the evaluation team have a deep understanding of the financing structure in TN and the need to collect qualitative data, as is customary in these types of evaluations, was minimal.

For the cost study, the evaluators received comprehensive expenditure information from fiscal administrators at TNDCS, which had been downloaded directly from the financial management system TNDCS relies on, known as Edison. The information was deemed to be highly reliable as it is used for claiming purposes and also serves as the basis of payment. Data dictionaries were provided as well and utilized for categorization purposes.

Methodology

The fiscal analysis began with a simple categorization of costs. For the current study, seven major categories were established:

Out of Home (Foster Care Board and Maintenance).21 When children are removed from their homes due to abuse or neglect, TN manages the case in one of two ways: 1) they contract with private agencies to provide foster care services, including residential care or; 2) they provide foster care services directly through DCS-licensed foster homes. Foster care board and maintenance costs include payments provided to foster parents or private providers, depending on how the case is being managed, for each day a child is in their care. It covers child related costs such as food, clothing, shelter, daily supervision, school supplies, personal incidentals and travel.

Out of Home (Other). This category includes all Out of Home expenditures not captured under Foster Care Board and Maintenance. These expenses are related to children served in juvenile justice (JJ) facilities, agency grants and contracts, and employee-related training and benefit costs.

In Home Purchased Services (Preventive). Broadly speaking, the goal of in-home services is to either avert the need for foster care placement or to expedite the discharge of children from foster care and reunification with their families. These services are provided to children and family members of children, who are at highest risk of coming into state custody. These services include child advocacy center services, family support services, relative caregiver services, and child abuse prevention services. Services in this category are provided exclusively by private providers. Other in home costs – related to children who are receiving non-custodial services provided by DCS – are captured in the case management category (below).

Case Management. This category captures expenditures for regional office staff and field staff providing services to custodial children, adoption services, non-custodial case management, and child protective services investigations.22

21 This category includes spending in Custody Services (35930) with Account Description “Pymnts Foster ChildC&AdoptnAst”.

22 For this phase of work, researchers were unable to disaggregate case management expenses and apportion among out of home, in home and the adoption categories. We proceed with the analysis with the caveat that some expenses in each of those three categories are captured herein.

94

Adoption. Adoption expenditures represent payments for special needs adoption assistance, adoption recruitment and placement services, and pre- and post-adoption support services.

Administration. Includes expenditures associated with TNDCS overseeing and managing the entire child welfare system. For example, the Commissioner’s Office and the administration of the divisions of Child Programs, Child Safety, Child Health and Juvenile Justice. Other central office administrative functions are included as well such as Human Resources, Facilities Management, Information Technology, Legal Services and Finance and Budget.

Other. This category includes Major Maintenance, Needs Assessment and Unknown expenses.

Using the data available to date, the research team examined the following dependent variables:

1. Total child welfare spending

2. Total foster care expenditures

3. Paid placement days

4. Average daily cost of foster care placement (total foster care board and maintenance expenditures divided by paid placement days).

For each dependent variable listed above, we present the indicator across five fiscal years. Since TN’s Waiver went into effect on October 1, 2014, available cost data covers a little more than three years prior to the Waiver and a little less than two years since the Waiver was implemented. As such, Waiver change will not be reported, and no significance testing is conducted at this stage.

Findings

As discussed previously, under the Waiver agreement, TN is now able to retain IV-E funding after covering traditional IV-E expenditures and use it for other child welfare purposes. As a result, the expectation is that TN will take action to reduce foster care expenditures in ways that improve outcomes for children and families. By making programmatic changes and investing flexible funds, TN can potentially reduce the length of stay in foster care, reduce reentry, and reduce the use of high-cost placements. The savings generated as a result of the Waiver are meant to be reinvested in child welfare services other than foster care, resulting in a continued decline in the need for foster care. Below we begin to explore this theory of change and the fiscal implications.

Total Child Welfare Spending

Table X displays all child welfare spending in TN for the past five full fiscal years. As detailed above, the data are organized around seven major categories used to separate spending types.

95

Total Child Welfare Spending in Thousands, by Major Category and Fiscal Year Table 27. Pre Waiver Years In Waiver Years

FY12 FY13 FY14 FY15 FY16

Out of Home - FC B & M $206,748 $221,858 $243,328 $262,584 $273,935 Out of Home - Other $64,434 $68,706 $67,282 $66,418 $55,217 In-Home Purchased $38,547 $36,487 $40,893 $43,309 $37,708 Case Management $218,133 $213,234 $234,197 $239,311 $246,693 Adoption $84,812 $79,557 $87,403 $88,739 $94,206 Administration $46,913 $51,011 $52,703 $55,577 $52,819 Other $6,429 $6,802 $6,185 $5,059 $5,226 TOTAL CW SPENDING $666,015 $677,655 $731,991 $760,997 $765,805

Proportion by Major Category Out of Home - FC B & M 31% 33% 33% 35% 36% Out of Home - Other 10% 10% 9% 9% 7% In-Home Purchased 6% 5% 6% 6% 5% Case Management 33% 31% 32% 31% 32% Adoption 13% 12% 12% 12% 12% Administration 7% 8% 7% 7% 7% Other 1% 1% 1% 1% 1% TOTAL CW SPENDING 100% 100% 100% 100% 100%

Annual Change by Major Category Out of Home - FC B & M

7% 10% 8% 4%

Out of Home - Other

7% -2% -1% -17% In-Home Purchased

-5% 12% 6% -13%

Case Management

-2% 10% 2% 3% Adoption

-6% 10% 2% 6%

Administration

9% 3% 5% -5% Other

6% -9% -18% 3%

TOTAL CW SPENDING

2% 8% 4% 1%

Total child welfare spending in FY16 has increased fifteen percent from FY12. It is important to note that this spending includes the costs of Waiver interventions. The largest increase in spending took place from FY13 to FY14, right before the Waiver was being initiated, and it appears to be leveling off from FY15 to FY16. Out of Home expenditures have also experienced an increase across the full five years. On the other hand, spending related to In-Home purchased services declined in FY16 after increases in FY14 and FY15 (an 8 percent reduction from FY14 levels). TN fiscal administrators note that there has been a deliberate effort on the part of DCS to ensure that Behavioral Health Organizations (BHO) are appropriately absorbing costs for eligible children for eligible services. As such, the notable decrease in preventive spending from FY15 to FY16 is not reflective of a decrease in services; rather, it reflects a shifting of costs from the state to BHOs for eligible in-home expenditures.

96

However, when we look at spending proportionally, there have not been major shifts in any of the categories across the past five fiscal years. Out of home spending accounts for 41 to 44 percent of child welfare spending while case management accounts for the next largest percentage (31 to 32 percent). Interestingly, when we look at both types of out of home spending, we see that board and maintenance has increased while other out of home costs have decreased. This dynamic, and the impact it is having on the unit cost of foster care, is explored below.

Foster Care Board and Maintenance Spending (FC B & M)

Here we look specifically at spending related to foster care board and maintenance, which is a portion of overall out of home expenditures. Foster care board and maintenance captures payments made on behalf of a child in foster care to cover expenses such as food, clothing, shelter, daily supervision and personal incidentals. Payments are made directly to foster parents or, in other cases, to private provider agencies.

The question here is the extent to which the proportion of foster care board and maintenance expenditures as a percent of total child welfare expenditures, changed as a result of the Waiver. The relative proportion of FC B & M spending could decrease by increasing other child welfare expenditures, decreasing FC B & M expenditures, or some combination of both. Although it is too soon to identify if changes are happening as a result of the Waiver, these baseline figures offer a glimpse into pre-Waiver trends, which may or may not contribute to observed shifts taking place during the Waiver years.

Foster Care Board and Maintenance as Percent of All Child Welfare Spending, by Fiscal Table 28.Year

Pre Waiver Years In Waiver Years


Foster Care Board and Maintenance $206,748 $221,858 $243,328 $262,584 $273,935

All Other CW Expenditures $459,267 $455,797 $488,663 $498,413 $491,869

Total CW Expenditures $666,015 $677,655 $731,991 $760,997 $765,805

% FC B & M (of Total CW Expenditures) 31% 33% 33% 35% 36%

% Change FC B & M Spending

7% 10% 8% 4%

TN has experienced an increase in both the proportion of spending related to FC B & M (up from 31 percent in FY12 to 36 percent in FY16) and actual spending related to FC B & M (up from $206M in FY12 to $273M in FY16). While overall child welfare spending and FC B & M spending have both increased over the five-year period, the increase in FC B & M has outpaced the increase in other categories (see Figure 30).

97

Figure 30. Five Year Change in FC B & M and All Other CW Expenditures

With the increase in FC B & M spending, it is worth examining sub-categories of spending as well. The overarching question here is whether there have been changes in how foster care dollars are being spent and, more specifically, on what types of children and services.

The cost study database breaks FC B & M spending into three main categories: 1) private provider placement expenses (includes residential care as well as family based foster care settings); 2) DCS foster parent payments which are payments made directly to foster parents; and 3) other expenses such as independent living and adoption/guardianship payments as well as specialized programs.. Here we will focus in on the first two, which account for 100 percent of spending for FY12, FY13, FY14 and FY15 and more than 90 percent of spending for FY16. It is worth noting that ‘Other’ expenses experienced a sharp increase from FY15 ($3,240) to FY16 ($21,019,197). This category includes programs called Child and Family Well Being as well as a Child Permanency Program. Future analysis will disaggregate this category further and explore explanations for the shift.

$0

$100,000

$200,000

$300,000

$400,000

$500,000

$600,000

$700,000

$800,000

$900,000


All Other CW Expenditures (In Thousands)

Foster Care Board and Maintenance (In Thousands)

98

Foster Care Board and Maintenance Spending in Thousands, by Category and Fiscal Year Table 29. Pre Waiver Years In Waiver Years


Private Provider Payments $182,663 $189,110 $209,974 $227,164 $215,638 DCS FP Payments $24,084 $32,745 $33,349 $35,417 $37,278 Other $1 $2 $5 $3 $21,019 Total $206,748 $221,858 $243,328 $262,584 $273,935 Proportion by Major Category Private Provider Payments 88% 85% 86% 87% 85% DCS FP Payments 12% 15% 14% 13% 15% Other 0% 0% 0% 0% 8% Total 100% 100% 100% 100% 100% Annual Change by Major Category Private Provider Payments

4% 11% 8% -5%

DCS FP Payments 36% 2% 6% 5% Other 133% 101% -34% 648,701% Total 7% 10% 8% 4%

As Figure 31 depicts (below), across the five fiscal years, spending on DCS foster parent payments has increased by a total of 55 percent ($24M to $37M) while spending related to children placed with private providers has increased by a total of 18 percent ($182M to $215M). In the following section, we explore these spending shifts alongside information around care day usage.

Figure 31. FC B & M: Five Year Change in Private Provider and DCS FP Expenditures

99

Placement Days and Unit Costs

As mentioned earlier, in order for TN to reduce foster care spending overall, they would need to reduce the number of paid care days, reduce the average cost of a care day, or both. This can be accomplished by any one of the following (or by some combination): a decrease in the number of children coming into care; an increase in the number of children leaving care; a decrease in the length of time children remain in care; increased use of less expensive types of care (i.e.: kinship care, family foster care); and/or decreased use of more expensive types of care (i.e.: congregate care). Below we present data on trends in care day use and unit costs in TN.

Table 30, below, details the number of care days used in TN in total for each fiscal year.23 This count includes all children who were in care at the start of the fiscal year as well as all children admitted to care during the fiscal year. The count includes all of the days that children spent in any type of foster care (including kinship care, treatment foster care, and detention) during the associated fiscal year. The number of care days being used has been fluctuating since FY12. There was an increase followed by no change and then a drop followed by a slight increase. When we look across all five years, the change is an overall decrease of 1 percent from FY12 to FY16.

Total Placement Days and Percent Change by Fiscal Year Table 30. Pre Waiver Years In Waiver Years24


Count of Paid Placement Days 2,982,611 3,066,802 3,069,260 2,930,286 2,958,781

% Change from Prior Year 3% 0% -5% 1%

Utilizing the figure related to total foster care board and maintenance spending for each fiscal year, we examine the average daily cost of foster care placement (total FC B & M spending divided by number of care days used).

23 Data on placement days by placement type comes from the Chapin Hall Multistate Foster Care Data Archive (FCDA). The FCDA is a longitudinal data warehouse developed and maintained by the Center for State Child Welfare Data at Chapin Hall. The FCDA utilizes child-level data from TFACTS. 24 Since the care day counts for In Waiver Years include days used by children who were in care at the start of the fiscal year as well as children admitted during the fiscal year, some of these days will be for children who were in care well before the Waiver went into effect. As such, we are not making any connection between the Waiver interventions and the overall count of paid placement days used here.

100

Average Daily Cost of Foster Care Placement Table 31. Pre Waiver Years In Waiver Years


Five Year

Change25

Foster Care Board and Maintenance

$206,747,835 $221,857,523 $243,328,192 $262,583,993 $273,935,358 32 %

Annual Count of Paid Placement Days

2,982,611 3,066,802 3,069,260 2,930,286 2,958,781 -1 %

Average Daily Cost of Foster Care

$69.32 $72.34 $79.28 $89.61 $92.58 34 %

As highlighted in Table 31, TN is experiencing a decrease in care day usage while experiencing an increase in FC B & M spending. One potential reason for a shift of this nature is the use of fewer, yet more expensive, care days. For example, more restrictive settings such as primary treatment centers and hospitals are far more expensive than less restrictive settings such as foster homes and kinship homes. Below we look at care day usage by placement type to begin the process of exploring this dynamic.

Care Day Usage by Placement Type and Fiscal Year Table 32. Pre Waiver Years In Waiver Years FY12 FY13 FY14 FY15 FY16 Family Foster Care 1,548,034 1,624,730 1,621,200 1,593,910 1,623,073 Kinship Care 463,383 466,861 464,497 399,659 387,622 In Home 261,650 269,455 267,980 237,773 233,742 Congregate Care 565,155 560,273 568,681 545,764 549,848 Detention 32,149 30,281 28,013 35,041 36,311 Emergency Shelter Placement 16,604 17,372 17,737 17,189 18,642 Hospital 5,554 5,242 3,800 4,546 5,741 Other (IL, Runaway, Unknown) 90,082 92,588 97,352 96,404 103,802 Total Care Days Used 2,982,611 3,066,802 3,069,260 2,930,286 2,958,781

While all types of care days experienced a slight decline between FY12 and FY16, the table below highlights that the use of more restrictive types of care days (i.e.: more expensive) increased by 25 percent from FY12 to FY16 while the use of less restrictive types of care days (i.e.: less expensive) declined by 22 percent from FY12 to FY16. This may be contributing to the increase in the average unit

25 The five year change is calculated by subtracting the FY12 total from the FY16 total and dividing by the FY12 total.

101

cost of foster care. Another consideration worth noting are the rate increases that TN has undertaken in recent years. For example, regular FY16 board rates in TN for 0 – 11 year olds are 1.6 percent higher than the FY15 rate.26 Private provider contract rates have increased as well.

Five Year Care Day Usage Change by Setting Type and Fiscal Year27 Table 33. LESS RESTRICTIVE SETTINGS

Family Foster Care 5% Kinship Care -16% In Home -11%

Overall Change in Use of Less Restrictive Care Days (FY12 – FY16) -22% MORE RESTRICTIVE SETTINGS

Congregate Care -3% Detention 13% Emergency Shelter Placement 12% Hospital 3%

Overall Change in Use of More Restrictive Care Days (FY12-FY16) 25%

When we look at child level outcomes as a way of exploring the care day usage patterns, we see that between FY12 and FY16, fewer children are now coming into care, fewer children are exiting care and the length of time children are remaining in care has gone up, albeit slightly (Table 34). As such, it would appear that the children in care are staying slightly longer than they were in FY12 and FY13 while simultaneously utilizing more expensive types of placements.

Child Level Outcomes by Fiscal Year28 Table 34.

FY12 FY13 FY14 FY15 FY16 Five Year Change

All Entries 7,454 7,290 6,370 6,145 6,480 -13% All Exits 7,046 7,119 6,773 6,184 6,621 -6% Median Duration (in days)

274 276 301 293 281 2%

Conclusions

The baseline view of pre-Waiver spending patterns offered here highlights some key changes that may impact spending patterns under the Waiver. Total child welfare spending has been increasing steadily over the past five full fiscal years. In the first full fiscal year of the Waiver (FY16) total child welfare spending increased by one percent. In addition, expenses related to purchased preventive services declined in FY16 following two years of increases.

TN has experienced an increase in both the proportion of spending related to FC B & M (up from 31 percent in FY12 to 36 percent in FY16) and actual spending related to FC B & M (up from $206M in FY12 to $273M in FY16). While overall child welfare spending and FC B & M spending have both increased

26 https://www.tn.gov/assets/entities/dcs/attachments/Tennessee_Foster_Care_Board_Rates_FY_7-1-2016.pdf 27 Change is calculated by subtracting FY12 totals from FY16 totals and dividing by FY12 totals. 28 Tennessee Cross Regional Workbook, delivered to TNDCS May 2017 by Chapin Hall Center for Children.

102

over the five-year period, the increase in FC B & M has outpaced the increase in other categories. Overall, the use of foster care placement days have declined since FY12 (after increases in FY13, FY14 and FY15). The average daily cost of foster care placement has increased by 34 percent from FY12 to FY16. This is due to higher spending alongside a steady use of days.

As the observational window expands, researchers will be in a position to make more concrete assessments related to the relationship between the fiscal stimulus and child welfare spending. This interim study examines just under two years of spending since the Waiver went into effect. It will take more time for the impact of the Waiver to be examined. Next steps for the Cost Study include the continued collection of expenditure information, gathering additional information related to revenue streams, and gaining a complete understanding of intervention unit costs by incorporating information around staff time related to intervention delivery. Finally, we expect to join the fiscal data with data on outcomes, which will allow for a cost-per-outcome view.

103

Summary, Lessons Learned, and Next Steps

Summary

TNDCS has achieved a great deal since the demonstration program began two and one-half years ago. The FAST, a decision/planning support and assessment tool has been implemented across the state with a completion rate of 95% for incoming CPS cases. A new casework model, R3, has been implemented in the 10 target counties, the objective of which is to improve the caseworker’s engagement skills and boost families’ motivation to fully participate in their service/permanency plan. KEEP, an evidence-based foster parent support program, was implemented in the 10 target counties in the summer of 2015. One year later, KEEP was rolled out to the remainder of the counties in the East Grand region. In April of 2017, KEEP was implemented in the Davidson, South Central regions and Sumner County of the Mid-Cumberland region.

TNDCS has supported the implementation of the demonstration project by putting structures in place that allow for regular communication between key staff members across the state. Early on in the implementation each region selected a lead staff member for each program area – FAST, KEEP and R3. These ‘leads,’ as they have come to be called, are responsible for disseminating information about implementation (new activities, associated data collection efforts, etc.) to their respective regions. The leads, along with other regional leadership, participate in regularly scheduled implementation calls to review implementation plans and processes. Central Office staff facilitate regularly scheduled calls with regional leads and other senior staff to review implementation plans and processes.

Importantly, TNDCS has always welcomed the evaluation team’s participation – from the earliest planning days through present. Indeed, TNDCS has, for a long time, held the view that research/evidence should be a central part of the conversation. The support the evaluation has had from the most senior leaders in the Department – support for data collection efforts, support for the utilization of evaluation findings – has been a critical factor in the success the evaluation team has had in implementing the evaluation plan over these past 2.5 years

Lessons Learned

Truth be told, the Chapin Hall evaluation team has not been faced with many challenges as far as the evaluation of the demonstration project is concerned. As noted previously, Chapin Hall has a long-standing partnership with TNDCS. Relationships (and trust) has been built over the years, the value of which cannot be overstated.

Next Steps

The evaluation team’s next steps can be organized into two groups of activities: upcoming data collection to support the evaluation and ongoing data analysis.

Upcoming Data Collection

In the next six to nine months, the evaluation team will focus on the following data collection activities:

104

Foster parent experiences of KEEP. In late summer/fall of 2017 focus groups will be conducted in the East Grand region with foster parents who have participated in KEEP groups. Foster parents will be asked to reflect on their experience participating in the KEEP training, to consider the utility of the various lessons taught during the training, to contemplate the relative value of the support structure of KEEP versus the concrete skill-building at the heart of the model, and to talk about the extent to which their actual parenting practices have changed in line with what they learned in KEEP training.

The evaluation team will also contact foster parents that chose to not participate in KEEP to understand why they chose to not participate and how they are handling any externalizing behavior problems from the children in their home.

Nurturing Parenting Program. The Nurturing Parenting Program is beginning in the summer/fall of 2017 in the East Grand region as well as the Shelby and Northwest regions. The evaluation team will be working with the Department to gather completed AAPI forms from a group of families that will serve as the comparison group, to enable the Department to see the extent to which certain parenting attitudes and behaviors shift in the absence of NPP. Evaluators will also conduct focus groups (currently scheduled for January 2018) to gain insight into staff perspectives related to the administration of NPP.

Ongoing analysis

General Staff Survey 2. The evaluation team – including subcontractors from the Kempe Center - will continue working with the data from both the GSS1 and GSS2.

FAST. The evaluation team will analyze the qualitative data collected from participants in the FAST focus groups conducted earlier this year. Of central interest is understanding whether and how staff actually uses the FAST in case decision making and service planning efforts.

Furthering the Partnership

The evaluation team looks forward to a continued successful partnership with Tennessee Department of Children’s Services throughout the end of the Waiver demonstration project.

105

Appendix A: Custodial Time Use Summary Tables

The full set of time use data is available upon request. It is a multi-tab Excel workbook which details these figures as well as the sub-tasks captured within each category. It also details the additional (or reduced) time spent for case variations.

Developing the Initial Permanency Plan during the first 30-45 days following placement in foster care FSW Team Leader Task Hours Percent Hours Percent Getting acclimated to the new case 2.6 6%

Direct communication 10.4 23% Conferences 5.6 13% Administration 9.1 19% Connecting with services 2.0 4% Family visits 7.5 17% Travel time 7.2 16% TOTAL (in HOURS) 44.3 99% 4.6 100%

Child Family Team Meetings: Values reflect ONE 6-month permanency conference

FSW Team Leader Task Hours Percent Hours Percent Preparing parties for the CFTM 1.2 27%

Administration 0.9 21% Conference itself 1.4 31% Following the conference 0.9 20% Total (in HOURS) 4.4 100% 1.4 100%

Managing an unplanned placement change FSW Team Leader Task Hours Percent Hours Percent Direct communication 3.3 24%

Conferences 2.3 17% Administration (docu., assess., mtgs.) 2.6 19% The new placement 2.0 14% Travel time 3.8 27% Total (in HOURS) 14.0 100% 2.4 100%

106

One Permanency Hearing (and all associated tasks) FSW Team Leader Task Hours Percent Hours Percent Preparing for hearing 3.8 52%

Travel time 0.8 12% At the court 2.6 36% Total (in HOURS) 7.3 100% 0.8 100%

Ending a case for a child reunifying with parent FSW Team Leader Task Hours Percent Hours Percent Direct communication 0 0%

Discharge conference 2.3 30% Assessments 1.3 17% Communicating with service providers 1.9 25% Administration 2.2 28% Total (in HOURS) 7.7 100% 3.4 100%

Connecting older youth with Independent Living Services

FSW Team Leader Task Hours Percent Hours Percent IL assessment 1.2 22%

IL service referrals 0.4 8% Refer to agency IL staff 0.3 5% Transition Planning CFTM 3.5 65% Total (in HOURS) 5.4 100% 0.2 100%

Ongoing case management responsibilities (times reported monthly)

FSW Team Leader Task Hours Percent Hours Percent Face-to-face contacts 3.8 19%

Other direct communication 2.4 11% Family visits 4.7 23% Assessment 0.3 2% Other case management tasks 7.8 38% Travel 1.7 8% Total (in HOURS) 20.7 100% 0.5 100%

107

Appendix B: Non-Custodial Time Use Summary Tables

Developing the Initial Permanency Plan during the first 30-45 days following placement in foster care

FSW Team Leader Task Hours Percent Hours Percent Getting acclimated to the new case 2.6 6%

Direct communication 10.4 23% Conferences 5.6 13% Administration 9.1 19% Connecting with services 2.0 4% Family visits 7.5 17% Travel time 7.2 16% TOTAL (in HOURS) 44.3 99% 4.6 100%

Child Family Team Meetings: Times reflect ONE 6-month permanency conference

FSW Team Leader Task Hours Percent Hours Percent Preparing parties for the CFTM 1.2 27%

Administration 0.9 21% Conference itself 1.4 31% Following the conference 0.9 20% Total (in HOURS) 4.4 100% 1.4 100%

Managing an unplanned placement change

FSW Team Leader Task Hours Percent Hours Percent Direct communication 3.3 24%

Conferences 2.3 17% Administration (docu., assess., mtgs.) 2.6 19% The new placement 2.0 14% Travel time 3.8 27% Total (in HOURS) 14.0 100% 2.4 100%

One Permanency Hearing (and all associated tasks)

FSW Team Leader Task Hours Percent Hours Percent Preparing for hearing 3.8 52%

Travel time 0.8 12% At the court 2.6 36% Total (in HOURS) 7.3 100% 0.8 100%

108

Ending a case for a child reunifying with parent

FSW Team Leader Task Hours Percent Hours Percent Direct communication 0 0%

Discharge conference 2.3 30% Assessments 1.3 17% Communicating with service providers 1.9 25% Administration 2.2 28% Total (in HOURS) 7.7 100% 3.4 100%

Connecting older youth with Independent Living Services

FSW Team Leader Task Hours Percent Hours Percent IL assessment 1.2 22%

IL service referrals 0.4 8% Refer to agency IL staff 0.3 5% Transition Planning CFTM 3.5 65% Total (in HOURS) 5.4 100% 0.2 100%

Ongoing case management responsibilities (times reported monthly)

FSW Team Leader Task Hours Percent Hours Percent Face-to-face contacts 3.8 19%

Other direct communication 2.4 11% Family visits 4.7 23% Assessment 0.3 2% Other case management tasks 7.8 38% Travel 1.7 8% Total (in HOURS) 20.7 100% 0.5 100%

109

Appendix C: Detailed Results from the General Staff Survey

Eligibility and Response Rates for GSS1 and GSS2 Table 35.

GSS1 Only GSS2 Only Both GSS Surveys

Neither Survey Total

Eligible for GSS1 Only

Number 64 0 0 61 125

% within Eligibility 51% 0% 0% 49% 100%

% of Total 13% 0% 0% 12% 25%

Eligible for GSS2 Only

Number 0 114 0 43 157


% of Total 0% 23% 0% 9% 32%

Eligible for both GSS1 & GSS2

Number 27 54 118 15 214


% of Total 5% 11% 24% 3% 43%

Count 91 168 118 119 496

% of Total 18% 34% 24% 24% 100%

110

Scale Results, by Wave29 Table 36.

Data Source N Mean Std. Deviation

Administrative Leadership GSS1 197 4.58 1.04 GSS2 269 5.22 1.01

Overall 360 5.00 1.00

Shared Vision GSS1 166 3.73 1.52 GSS2 194 5.25 0.85

Overall 349 4.73 1.24

Supervision Aa GSS1 166 5.05 1.14 GSS2 182 5.37 1.22

Overall 274 5.29 1.16

Supervision Ba GSS1 166 4.31 1.95 GSS2 181 5.40 1.27

Overall 273 4.92 1.61

General Confidence in Services GSS1 196 4.91 1.09 GSS2 271 5.24 0.92

Overall 362 5.13 1.00

Confidence in Specific Services GSS1 196 4.81 1.15 GSS2 271 4.49 1.00

Overall 362 4.61 1.02

Process Case Skillsa GSS1 165 5.55 0.81 GSS2 184 5.63 0.67

Overall 272 5.57 0.73

Interpersonal Case Skillsa GSS1 165 5.70 0.78 GSS2 184 5.64 0.65

Overall 272 5.67 0.71

Workload Concernsa GSS1 166 4.83 1.68 GSS2 185 4.78 1.54

Overall 274 4.78 1.62

Job Satisfaction GSS1 209 4.76 1.29 GSS2 273 5.22 1.40

Overall 366 5.07 1.40

29 For the overall score, we calculated a composite overall score variable for use with future outcome measures analysis. For those participating in both rounds of the survey, a repeated measures analysis was conducted. If significant differences were observed between GSS1 and GSS2 responses, an average score was calculated. Where no differences over time were detected in the aggregate, the score from the GSS2 was used. If a respondent participated only in the GSS1 or only in the GSS2, their scores reflect the answers they provided on that survey.

111

Data Source N Mean Std. Deviation

Family Preservation vs Child Safety GSS1 196 2.17 12.70 GSS2 271 3.47 12.75

Overall 362 3.14 12.68

Inclination to Prevent Removal GSS1 196 3.65 1.37 GSS2 271 3.47 1.31

Overall 362 3.56 1.27

Inclination to Reunify GSS1 196 2.89 1.67 GSS2 271 2.31 0.86

Overall 362 2.56 1.23

External References Guide Case Closurea GSS1 161 5.45 0.92

GSS2 197 5.38 0.99 Overall 281 5.43 0.98

Internal References Guide Case Closurea GSS1 161 2.93 1.29

GSS2 197 2.87 1.22 Overall 281 2.90 1.28

a These scales were only administered to staff carrying a caseload

Scale Results from Repeated Measures Analysesa Table 37.

Scale Paired Nb

GSS1 Mean(SD)

GSS2 Mean(SD) F(df) p

Administrative Leadership 106 4.53 (1.01) 5.07 (1.09) 20.688(1) 0.000 Shared Vision 86 3.80 (1.51) 5.04 (0.87) 38.855(1) 0.000 Supervision A 74 4.92 (1.28) 5.07 (1.29) 0.707(1) 0.403 Supervision B 74 4.30 (1.92) 5.19 (1.32) 13.026(1) 0.001 General Confidence in Services 105 5.00 (1.02) 5.17 (1.07) 2.398(1) 0.124 Confidence in Specific Service Types 105 4.87 (1.11) 4.46 (1.10) 14.726(1) 0.000 Process Case Skills 77 5.57 (0.70) 5.75 (0.68) 5.953(1) 0.017 Interpersonal Case Skills 77 5.66 (0.74) 5.64 (0.75) 0.032(1) 0.858 Workload Concerns 77 4.88 (1.56) 5.07 (1.45) 1.024(1) 0.315 Job Satisfaction 116 4.86 (1.26) 4.99 (1.52) 0.788(1) 0.377 Child Safety vs. Family Preservation 105 2.17 (12.95) 1.92 (13.55) 0.049(1) 0.825 Inclination to Prevent Removal 105 3.63 (1.35) 3.36 (1.31) 2.748(1) 0.100 Inclination to Reunify 105 2.80 (1.54) 2.23 (0.78) 12.885(1) 0.001 External References Guide Case Closure 77 5.37 (0.89) 5.43 (0.89) 0.268(1) 0.606 Internal References Guide Case Closure 77 2.88 (1.16) 2.95 (1.28) 0.261(1) 0.611 a Analyses were conducted examining scores from just the staff who participated in both rounds of the GSS. b Sample size varies according to whether scale was administered to casework staff only, or all staff.

112

tennessee’s iv-e waiver demonstration project …...intervals 82 table 21. exits, average care...

Documents