Measuring Sustainment of Multiple EBPs Fiscally Mandated in Children’s Mental

Health Services: Knowledge Exchange on Evidence-Based Practice Sustainment

(4KEEPS) Study Lauren Brookman-Frazee

Southern California Regional Dissemination, Implementation and Improvement Science Webinar Series

June 1, 2016

Objectives

1. Provide an overview of system driven EBP implementations and community context and the 4KEEPS Study.

2. Describe study methods and initial findings: • Characterizing sustainment outcomes

• Characterizing potential inner context determinants of sustainment outcomes

Public Mental Health Care System (Local, State)

Organizations contracted to provide MH Care

Therapists providing direct services to consumers

Exploration Phase

Preparation Phase

Implementation Phase

Sustainment Phase

Phases and Levels of EBP Implementation

Adapted from Aarons, Hurlburt & Horwitz, 2011

Public Mental Health Care System (Local, State)

Organizations contracted to provide MH Care

Therapists providing direct services to consumers

Exploration Phase

Preparation Phase

Implementation Phase

Sustainment Phase

Sustained Delivery of Multiple EBPs in System-Driven Implementation Efforts

Adapted from Aarons, Hurlburt & Horwitz, 2011

Statewide Reform: California’s Mental Health Services Act

Prevention and Early Intervention

(PEI)

Los Angeles County

• Nation’s largest county mental health department • 75 directly operated sites • 288 contracted agencies

(120 with child MH services)

• 8 Service Provision Areas range from rural to inner city communities

• Serves an ethnically diverse, disadvantaged population

Hispanic/Latino

Asian/Pacific Islander

African American

Applying Aarons et al (2011) Model to LACDMH PEI Timeline

Applying Aarons et al (2011) Model to LACDMH PEI Timeline

Characteristics of Practices Practice Age Range

(years) Target Problem(s)

Triple P 2-12 conduct

CPP 0-6 trauma; attachment

MAP 0-21 anxiety; trauma; depression; conduct

TF-CBT 3-18 trauma

CBITS 11-15 trauma

Seeking Safety 13-18 trauma; substance use

Applying Aarons et al (2011) Model to LACDMH PEI Timeline

Applying Aarons et al (2011) Model to LACDMH PEI Timeline

4KEEPS Study Aims

• A neutral, observational study investigating the sustainment of original 6 practices with implementation support.

• Aim 1: Characterize sustainment outcomes • EBP Concordant Care • Volume/penetration of each practice over time

• Aim 2: Use mixed methods to characterize inner context factors and early implementation conditions that potentially predict EBP sustainment.

• Aim 3: Identify inner context and early implementation conditions that determine sustainment outcomes

Funded by NIMH Grant # R01 MH100134 MPI Anna Lau and Lauren Brookman-Frazee

Multiple Sources of Data

LACDMH Administrative

Claims Data

LACDMH Site Visit

Documents

Online Surveys of Program Leaders and Therapists

Semi-Structured Interviews

Session Recordings

Sustainment outcomes and potential inner context determinants of outcomes

Today

2014 2015 2016 2017 2018

4KEEPS Study

launch

1/1/2014

Online survey 1

launch

3/1/2015

In-depth sample

launch

5/1/2015

Online survey 1

supplement

2/1/2016

In-depth sample wrap-up

Spring 2017

Online survey 2 launch

Spring 2017

4KEEPS Study Timeline

4KEEPS Events Timeline

Sustainment outcomes

• # & % agencies continuing to be reimbursed for the practice

• # & % Therapists continuing to claim to the practice

• # & %Unique clients served by the practice

• # $ % Units of services being provided within each practice

Practice Volume/Penetration

• Degree to which a therapist’s practice resembles the essential strategies one would expect within an evidence-based protocol for a given problem focus.

EBP Concordant Care

Practice Volume/Penetration

94

agencies

8,514 providers

87,100 children

2,331,000 psychotherapy claims

3,014,353 total claims

2009 2015

Cumulative # of Claims Per Practice • MAP = 905,395

• TF-CBT = 662,184

• Seeking Safety = 515,208

• Triple P = 140,147

• CPP = 105,231

• CBITS = 2835

Volume Over Time

Initial increases in raw volume for most practices in the first quarters with peak and leveling off after the initial few years Highlight the rapid impact of a fiscal policy change restricting reimbursement to specific practices and training in these practices

Gross Penetration Over Time

During the initial ramp-up period, some practices ramped up more quickly than others. Reordering and stabilization after 2 years MAP, TF-CBT, SS > Triple P, CPP, CBITS

Characteristics of Practices

Higher Penetration (MAP, TF-CBT, CPP) • Apply to a broad age range of

clients

• Can be delivered in multiple settings

• Addresses common presenting problem -trauma

• MAP also covers a range of presenting problems

• Train-the-trainer capacity (MAP, SS)

• Minimal training required for billing (SS)

Lower Penetration (Triple P, CPP, CBITS) • Apply to a narrower age

range of clients (CPP, CBITS)

• Restrictions on settings/format • Group (CBITS, Triple P) • School setting (CBITS) • Caregiver directed (CPP,

Triple P)

• Additional requirements to deliver such as MOAs between programs and schools (CBITS)

Sustainment outcomes

• # & % agencies continuing to be reimbursed for the practice

• # & % Therapists continuing to claim to the practice

• # & %Unique clients served by the practice

• # $ % Units of services being provided within each practice

Practice Volume/Penetration

• Degree to which a therapist’s practice resembles the essential strategies one would expect within an evidence-based protocol for a given problem focus.

EBP Concordant Care

Measuring Sustainment: EBP Concordant Care

Addresses concerns about the feasibility and

appropriateness of using traditional fidelity instruments to

asses ongoing delivery of multiple EBPs

EBP Concordant Care Assessment (ECCA) Development Process Reviewed existing practice inventories.

Adapted/selected/generated items (strategies).

Collected data from practice experts about how essential/interfering strategies are to their practice.

Delphi rating system used to determine item selection for therapist-report ECCA (alpha version).

Collected data from large sample of therapists using ECCA (alpha version).

Examine the properties of the therapist-report ECCA (alpha version).

Validate therapist-report ECCA with observational coding system.

Finalize therapist-report ECCA based on #6 and 7.

Current ECCA

Purpose Assesses the extent to which a therapist delivered individual psychotherapeutic strategies considered essential for a given EBP Target

Online Versions

Session (therapist report, observer rated) and Bi-monthly (therapist report)

Items 38 psychotherapeutic content and techniques EBP Targets #Content #Techniques

Conduct 18 9 Trauma 10 5 Anxiety 7 7 Depression 9 6

ECCA Data Collection To Date

• 710 therapists from 54 agencies completed a Bi-Monthly ECCA

Full Sample

• 71 therapists submitted 459 Session ECCAs with corresponding audio recordings of sessions as well as Bi-Monthly versions for each client

Validation Sub-Sample

ECCA Preliminary Findings

High internal consistency for each EBP Target Composites for Anxiety, Conduct, Depression and Trauma (alphas ranged from .86 to .95; M=.90).

Discriminant validity of the scales supported by significant differences in EBP Target composites by EBP delivered When therapists delivered an EBP that targeted Conduct or Trauma,

their ratings on the content items from the corresponding composite were significantly higher than for other EBP targets

Construct Validity supported by Item Response Theory Analyses indicating that items with high item difficulty appeared to include strategies that reflect high integrity EBP concordant care (Ignoring/Differential Reinforcement of Other Behaviors > Praise)

ECCA Preliminary Findings Continued Concordance between Session and Bi-monthly

version supporting by significant and large correlations between the average Session ECCA with the Bi-Monthly version on the EBP Target composites (range= .62 to .71; M=.68).

Concordance between therapist report and direct observation in process Early data based on a small subset of therapist reports

with corresponding observer ratings indicate greater concordance for strategies that are part of the practice type being delivered.

Next Steps

1) Finish validation of therapist-report ECCA with observational coding system 1) Projected sample of 1080 sessions with therapist report

and observer ratings

2) Examine inner context factors associated with ECCA composite scores.

3) Refine ECCA instrumentation based on concordance analysis and end user feedback on utility.

Potential Inner Context Determinants of Sustainment

• Changes in state regulatory requirements

• Changes in implementation strategies used

• Adoption, de-adoption of EBPs over time System Level

• Implementation support

• Organizational climate

• Early implementation condition

Organizational Level

• Therapist attitudes: perceptions of effectiveness and fit

• Therapist clinical adaptations to practices Therapist Level

Therapist and Practice Characteristics as Facilitators of Multiple EBP Implementation

Practices

Therapists

• The PEI Context and fiscal mandate for EBP reform presents a unique context • EBP vs. Practice as Usual • EBP1 vs. EBP2 vs. EBP3…

• Allows for parsing the variance in attitudes associated with • Therapists and therapist

characteristics • Practices and practice

characteristics

Demographics M (SD) or %

Age 37.00 (9.28)

Gender (female) 88%

Hispanic 43%

Race

White 50% Asian/ Pacific Islander 11% African American 7% American Indian/Alaska Native 1% Multiracial 8% Other 23%

Deliver Services in >1 Language

Spanish 47%

Other Language 10%

Survey: Therapist Characteristics (n=790)

Training Background M (SD) or %

Years Practiced as Therapist 7.21 (6.16)

Years Worked at Current Agency 5.12 (4.56)

Current # Clients 14.65 (10.48)

MH Discipline

MFT 55%

Social Work 30%

Clinical Psychology 11%

Counseling 2%

School Psychology <1%

Other 2%

Measuring Therapist Attitudes Practice-Specific Attitudes General Attitudes towards EBP

Perceived Characteristics of Intervention Scale, Cook et al., 2014

• Relative Advantage • Complexity • Compatibility • Potential for Reinvention • Total

Evidence-Based Practice Attitudes Scale, Aarons, 2004

• Openness • Divergence

Attitudes Differ by Practice

F b

Intercept (grand mean) 9667.23*** 3.29***

TF-CBT 183.10 *** .45***

CPP 29.73*** .28***

Triple P 11.77** .16**

MAP 5.28* .08*

SS 11.47** -.12**

CBITS 117.16*** -.84***

*p<.05; **p<.01; ***p<.001

Practice Characteristics

Practice PCIS

Alphas

Narrow Age

Range

Consultation

Required

Prescribed

Session

Content/ Order

CBITS (N = 65) .96

CPP (N = 140) .94

MAP (N = 527) .92

SS (N = 491) .93

TF-CBT (N = 582) .93

Triple P (N = 184) .94

Therapist Attitudes Differ by Practice Characteristics

1

2

3

4

5

Prescribed Session Content/Order

Consultation Required Narrow Age Range

PC

IS IT

EM M

EAN

PRACTICE CHARACTERISTIC

Yes

No

* * *

* = p < .001

Very Great Extent

Not at All

• Use sequential QUANqual design to understand:

• The types of adaptations that therapists make in community mental health settings.

• Which therapist characteristics predict types of adaptations (fidelity consistent vs. inconsistent).

• The reasons therapists make adaptations.

Therapist Adaptations

Fidelity Consistent

Modify presentation

Integrate supplemental

content

Lengthen/extend Pacing

Fidelity Inconsistent

Remove/skip components

Shorten/condense pacing

Adjust order of sessions/components

Characterizing Adaptations

1

2

3

4

5

Mea

n E

xten

sive

nes

s

Fidelity Consistent

CBITS CPP SS TFCBT TP

1

2

3

4

5

Fidelity Inconsistent

Types of Adaptations

• Therapists reported more fidelity consistent adaptations (EMM= 9.30; SE = 1.79) than fidelity inconsistent (EMM= 5.88; SE = 1.79) adaptations • F(1, 1,075.59) = 1,332.00, p <

.001

• Therapist-reported attitudes towards a practice: • Did not predict fidelity

consistent adaptations B = -.02, t = -1.47, p = .14

• Did predict fidelity inconsistent adaptations

• B = -.07, t = -5.67, p < .001

0

1

2

3

4

5

6

7

8

9

10

Fidelity Consistent Fidelity Inconsistent

Mea

n C

om

po

site

CBITS CPP SS TFCBT TP

Reasons for Adaptations

Culture

Crises

Adjust Order

Extend Pacing

Modify Presentation

Omit

Developmental Level I think the length of treatment is part of that too,

especially with the Hispanic families. Like you can’t just

jump right in; you need to spend a little more time

building rapport.

Some of the examples that they have there I think, you

know, might not necessarily fit like this particular culture

so I’ll find something that they could relate to a little bit

better, a situation they can relate to a little bit better.

I did work with one client that was developmentally

delayed. So for some of the cognitive behavioral piece it

was just really difficult to understand…I did my best to

teach it, but I didn’t feel like I focused a lot on it because

dude wasn’t really understanding it.

Oh, we’ll still go back to some of the relaxation

components, or we’ll kind of focus on what we can do to

manage our anxiety about it and whatnot.

You know, with people that are developmentally delayed or whether there needs to be more repetition, I’ll repeat it more.

I think it’s just different working with a child that young.

So we do a lot of art. So instead of like a trauma

narrative, we’ve been doing a lot of art on the

chalkboard.

You do have those clients that maybe might become homeless, or maybe parents are having a really hard time, maybe with other kids, older kids. So you might have a lot of crises that might surface throughout treatment, and that might postpone certain parts of it

Summary of Preliminary Findings

The scope and size of the PEI transformation provides a critical opportunity to examine multiple EBP implementation in a large and diverse natural laboratory

Sustainment varies by practice and over time

Inner context factors are critical to examine within EBP implementation efforts

• Therapist attitudes vary across practices.

• Therapists report more extensive fidelity consistent adaptations than fidelity inconsistent adaptations

Funding Source: NIMH R01MH100134