1

Children and FamilyResearch Center

University of Illinois at Urbana-Champaign

School of Social WorkTM

Making the Case for Safe Reduction

What Does the Evidence SayMark Testa

Director & Professor

Tampa, Florida

November 5, 2009

2

Desired Outcomes

Safety – Ensuring the safe reduction of the number

of children in foster care.

Continuity – Conserving family continuity and

community connections.

Stability – Investing in the stability and quality of

the least restrictive, substitute care.

Permanence – Expediting the timely achievement

of family permanence.

Well-Being – Safeguarding the well-being of the

children who remain and preparing them for

productive and responsible adulthood. .

3

U.S. Foster Care Corrected Trend

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

Pe

r 1

,00

0 C

hild

ren

0-1

8

Survey Estimates

Westat

CDF

U.S. Total Foster Care

IV-A/ IV-E Claims

4

U.S., CA, & IL Foster Care Trends

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

Pe

r 1

,00

0 C

hild

ren

0-1

8

U.S. Foster Care

IL Foster Care

CA Foster Care

3.3 per 10002020 Goal

5

IL Caseload Reduction Dynamics

6

Maryland’s Place Matters

7

Illinois Programs

Front-End Reforms• Alternative Response – Extended Family Support

(EFS) for children in pre-existing kinship care.

• In-Home Services – Child Endangerment Risk

Assessment Protocol (CERAP).

Back-End Reforms• Adoption and Guardianship – Subsidized

Guardianship Waiver Demonstration.

8

PICO Components of Well-Built Question

P – Conditions of the target population for

which one wishes to draw inferences.

I – Intervention or program whose efficacy

and effectiveness one is interested in

evaluating.

C – The alternative course of action with

which one wants to draw a comparison.

O – Intended outcome one hopes to achieve.

9

Examples of Well-Built Questions

• Does the offer of subsidized guardianship (I) in addition

to the usual permanency options of reunification and

adoption (C) increase overall family permanence (O) for

children in stable kinship foster care (P)?

• Is the rate of short-term recurrence of indicated

maltreatment reduced (O) for children and young persons

under 18 years old (P) when structured safety assessments

are conducted (I) compared to investigation processes that

do not feature structured assessments(C)?[

10

The Problem of Selection Bias

δ + bias

Outcome

εrror component (unobserved systematic & random influences)

Intervention v. Comparison group

11

Selection Bias

• Selection of intervention & comparison groups.

• Is this a valid comparison?

vs.

Intervention Group Comparison Group

12

Balanced Groups

• Selection of intervention & comparison groups.

• This is a much better comparison

vs.

Intervention Group Comparison Group

13

Approximating the Counterfactual

Randomized controlled experiments

Quasi-experiments

-Instrumental variables

-Regression discontinuity

Observational studies

-Propensity score matching

-Regression analysis

Historical cohort comparisons

Stronger

Weaker

Illinois SG Waiver Demonstration

• In 1997, Illinois received a IV-E waiver to offer

subsidies to families who assumed private

guardianship of children who otherwise would have

remained in foster care.

• Available to all children who have been in state

custody for one year, resided continuously with a

relative for one year and for whom reunification and

adoption have been ruled out as permanency plans.

• Also available for children aged 12 and older who

reside with an unrelated foster parent.

14

15

Randomized Controlled Experiments

δ

Outcome

εrror component (unobserved systematic & random influences)

Intervention v. Comparison group

∆ = 0

Statistical Equivalence at Baseline

Intervention Comparison Difference

Child characteristics

Age at interview 9.9 10.1 -0.2

Age at removal 4.8 4.8 0.0

Female 49.5% 49.7% -0.2%

White 9.9% 9.4% 0.5%

Black 83.6% 85.3% -1.7%

Caregiver characteristics

Age at interview 51.2 51.8 -0.7

White 10.7% 10.8% -0.1%

Black 82.5% 83.2% -0.8%

Married 32.5% 32.2% 0.3%

Less than high school 40.0% 39.9% 0.1%

High school graduate 17.2% 19.3% -2.1%

Some college 28.5% 24.8% 3.7%

Full-time employment 34.8% 34.2% 0.6%

Not in labor force 47.6% 48.7% -1.1%

Intend to raise child to adulthood 78.7% 79.6% -0.9%

Caregiver-child relationships

Grandparent-grandchild 43.4% 48.3% -4.9%

Aunt/Uncle-niece/nephew 18.0% 18.1% -0.1%

Foster parent-foster child 18.5% 17.2% 1.3%

Matched ethnic backgrounds 2.8% 3.3% -0.5%

Sample N 1,197 1,228

16

17

Availability of SG boosted legal permanence …

At wave I of the evaluation (1998)

there was a 8% permanency

advantage for children in the

intervention group who were offered

the choice of subsidized guardianship

compared to children in the

comparison group.

Increase in both adoptions and

guardianships

Experimental Control

0%

20%

40%

60%

80%

100%

AlreadyReunified

Already SG Already AA

}8.3%

Illinois

18

Internal Validity

δ = 8 per 100 children

Permanence

εrror component (unobserved systematic & random influences)

SG v. Comparison group

∆ ≈ 0

External Validity

• Concern: Do these findings generalize beyond

the state’s unique historical circumstances and

distinctive policies on kinship foster care?

• Fact: Two replications of Illinois’ waiver

demonstrations in Tennessee and Milwaukee

find that subsidized guardianship increases

overall family permanence.

19

Waiver Findings

Intervention Comparison

0%

20%

40%

60%

80%

100%

Guardianship

Reunificatioon

Custody to Kin

Adopted

} 6.6%

Illinois

(June, 2007)

Intervention Comparison

0%

20%

40%

60%

80%

100%

Guardianship

Reunificatioon

Custody to Kin

Adopted

Intervention Comparison

0%

20%

40%

60%

80%

100%

Guardianship

Reunificatioon

Custody to Kin

Adopted

}19.9%

Wisconsin

(November, 2007)

} 15.0%

Tennessee

(November, 2008)

No Adverse Impact on Reunifications

Concern: SG will discourage CW agencies from

reunifying children with their parents.

Fact: All three randomized experiments find that

reunification rates were not significantly

different between families offered subsidized

guardianship vs. families denied this choice.

21

No Adverse Impact on Reunifications

Intervention Comparison Difference

Illinois (June 2007) 5.2% 7.7% -2.6%

Tennessee (December 2008) 13.2% 13.7% -0.5%

Wisconsin (November 2007) 9.6% 8.6% 1.0%

Differences in Reunification Rates, Intervention vs. Comparison Groups

22

Cost-Effective

Concern: SG will end up costing more money in the

long-run.

Fact: In Illinois after 10 years, offering families the SG

option reduced the average length of foster care by

269 days or 22% compared to what would have

happened in the absence of the offer. In Wisconsin

after 3 years, offering family the SG option reduced

the average length of foster care by 133 days or

24%. In Tennessee, analysis is underway.

23

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01

Pe

r 1

00

0 C

hil

d P

op

ula

tio

n

Fiscal Year Entry Cohorts

Sexual Abuse

Physical Abuse

SEI

Lack Supervision

Other Neglect

Risk of Harm

Extended Family Support

Alternative Response

24

Safety Assessment: CERAP

• Secular trend study examines recurrence rates

before and after the point in time (December 1,

1995) when the implementation of CERAP

occurred.

• This is the date that all Department of Children

and Family Services (DCFS) workers and

private providers had been trained in the use of

the protocol and over 99 percent had been

successfully certified.

25

60-day recurrence rates for first reports in

time period: Sequence A, Excluding PCs,

2.63 2.652.79

2.23

1.77 1.80 1.84 1.79

2.30

1.73

1.47

1.26

1.08 1.11

0.850.73

0.0

0.5

1.0

1.5

2.0

2.5

3.0

12/85 -

11/86

12/86 -

111/87

12/87 -

11/88

12/88 -

11/89

12/89 -

11/90

12/90-

11/91

12/91 -

11/92

12/92 -

11/93

12/93 -

11/94

12/94 -

11/95

12/95 -

11/96

12/96 -

11/97

12/97 -

11/98

12/98 -

11/99

12/99 -

11/00

12/00 -

11/01

12/01 -

11/02

Implementation

Year

26

27

Recurrence by Race

28

Investigator Rotational Assignment

• Most families are effectively randomized to

investigators.

• Different investigators have different

removal propensities (strict to lenient).

• What happens to cases when

investigators disagree about removal

recommendations?

29

Approximating the Counterfactual

The key variable is how lenient or strict an investigator is when it comes to making removal decisions.

In the language of economics, we call this variable an ―instrumental variable‖ if it meets two primary conditions:

(1) investigator removal differential predicts the likelihood of removal from from the home; and

(2) the differential itself doesn’t have a direct impact on the outcome, except through its effect on removal.

30

Instrumental Variable Model

δ

Outcome

Population conditions

Intervention (v. Comparison group)

β

Z (Instrumental variable)

π

εrror component

Here, Z = Investigator Removal Differential: In

cases other than a particular family’s case, what

fraction of children investigated by that family’s

investigator are placed, relative to placement rates

of other investigators on the same team in a given

year.

31

Marginal Cases

High Safety Risk

Always placedNever placed

SR1 SR2

Inv’gator 1 Inv’gator 2

Strict Investigator 1 removes if SR > SR1 Lenient Investigator 2 removes if SR > SR2

Low Safety Risk

Area of

Disagreement

Children at the margin of removal perform worse when

they are removed from home1:

– 3x Higher Arrest Rate

– 3x Higher Delinquency

– 2x Higher Teen Motherhood

– 40% Lower Employment

– No Effects for Childhood Burns / Broken Bones;

– But 3x more likely to receive well-child visits.

32

Detrimental Impact of Foster Care

1Doyle, J. J. (2007a). Child protection and child outcomes: Measuring the effects of

foster care. The American Economic Review, 97(5), 1583—1610. Doyle, J. J. (2008).

Child protection and adult crime: Using investigator assignment to estimate causal

effects of foster care. Journal of Political Economy, 116(4), 746—770.

Conclusions

• Strong evidence that subsidized guardianship is a

cost-effective program for reducing the number of

children in long-term foster care with relatives.

• Weaker but compelling evidence that alternative

response to children in pre-existing kinship care and

the use of structured safety assessment can safely

reduce the number of foster care removals.

• Moderately strong evidence that children at the

margin of removal fare better when they remain

home.

33