@aypf_tweets  

“Examining Strategies to Improve the Use of Research Evidence in Child Welfare”

Monday, October 5, 2015

                                   

Innova'on  and  the  Use  of  Research  Evidence  in  Public  Youth-­‐Serving  

Systems  

Lawrence  A.  Palinkas,  PhD  Albert  G.  and  Frances  Lomas  Feldman  Professor  of  Social  Policy  and  

Health  (funded  by  William  T.  Grant  FoundaBon,  Grant  No.  10648)  

Vulnerable  youth  need  evidence-­‐based  services  

•  One  in  five  youth  in  the  United  States  are  in  need  of  mental  health  services  (New  Freedom  

Commission,  2003).  •  There  are  currently  415,000  youth  in  child  welfare  in  the  United  States  today  (Admin  Child  Fam.,  2015).  

•  50%  of  these  youth  need  mental  health  services  (Burns  et  al.,  2004).  

 

These  youth  are  not  geSng  the  services  they  need    

•  Only  20%  of  youth  who  need  mental  health  services  get  the  services  they  need  (Kataoka  et  al.,  2002).  

•  90%  of  publicly-­‐funded  child  welfare,  mental  health  and  juvenile  jus'ce  systems  do  not  use  evidence-­‐based  prac'ces  (Hoagwood  &  Olin,  2002).  

•  Only  half  of  all  children  in  child  welfare  receive  care  consistent  with  any  one  na'onal  standard  and  less  than  10%  receive  care  consistent  with  all  10  na'onal  standards  (Raghavan  et  al,  2010).  

Use  of  research  evidence  is  cri'cal  to  adop'on  and  implementa'on  of  evidence-­‐based  prac'ces  

•  Implementa'on  of  evidence-­‐based  prac'ces  and  policies  (EBPPs)  involves  some  degree  of  use  of  research  evidence  (URE).    –  To  iden'fy  appropriate  EBPPs  –  To  support  decision  to  adopt  EBPPs  –  To  support  implementa'on  of  EBPPs  

•  However,  the  extent  to  which  URE  contributes  to  EBPP  implementa'on  is  unknown.  

Specific  Aims  of  My  Research  

1.  Understand  and  measure  the  use  of  research  evidence  by  decision  makers  of  public  youth-­‐serving  agencies.  

2.  Prospec'vely  determine  whether  use  of  research  evidence  predicts  stage  of  EBPP  implementa'on.  

The  CAL-­‐OH  Study  

•  Objec've:  Determine  whether  Community  Development  Teams  (CDTs)  are  more  effec've  than  services  as  usual  in  “scaling  up”  implementa'on  of  Mul'dimensional  Treatment  Foster  Care.  

•  Design:    Adap've  or  rolling  RCT  in  which  40  California  and  11  Ohio  coun'es  are  randomized  into  two  condi'ons  (CDT  vs  SU).  

Methods  –  Semi-­‐structured  interviews  and  focus  groups  to  assess  how  systems  

leaders  determine  a  prac'ce  is  evidence-­‐based  and  how  they  acquire  informa'on,  evaluate  it,  and  apply  it  in  making  decisions  about  adop'ng  and  implemen'ng  new  programs  and  prac'ces.  

 –  Development  of  two  new  measures  

•  Structured  Interview  for  Evidence  Use  (SIEU)  •  Cultural  Exchange  Inventory  (CEI)    

–  Web-­‐based  survey  of  202  leaders  of  state  and  county  child  welfare,  juvenile  jus'ce  and  mental  health  systems.    

LESSONS  LEARNED  

1.  Networks  within  and  across  agencies  affect  the  flow  of  informa'on  and  what  research  is  

ul'mately  applied    •  Leaders  of  child  serving  systems  

rou'nely  create  and  u'lize  networks  of  other  systems  leaders  for  informa'on  and  advice  and  to  pool  resources  when  implemen'ng  EBPs.  

•  The  larger  the  network  with  higher  levels  of  in-­‐person  centrality,  the  more  likely  these  systems  will  implement  an  EBP.  

2.  Networks  can  be  developed  and  strengthened  through  implementa'on  strategies  like  

Community  Development  Teams    

CDT Control

3.  Use  of  research  evidence  does  inform  policy  and  prac'ce  

SIEU  measures   SIC  Stage   Dura5on   Propor5on  of  ac5vi5es  Phase  1   Phase  2   Phase  3  

r   r   r   r   r  Input          .15*    .03   -­‐.10      .29*        .37*  Process     .19    .28    .07   .28   .14  Output        .29*   -­‐.11    .15   .25   .16  Total  SIEU  score      .29*      .13    .02          .38**        .31*  

* p < 0.05; ** p < 0.01

•  Engagement  in  evidence  use  was  significantly  associated  with  stage  of  implementa'on  and  comple'on  of  ac'vi'es  during  the  implementa'on  and  sustainment  phases  of  the  project.  

•  This,  in  turn,  was  associated  with  significantly  more  clients  being  served  and  a  significantly  greater  likelihood  of  achieving  competency  in  use  of  MTFC.    

4.  Priority  is  given  with  respect  to  how  evidence  is  accessed,  evaluated  and  applied  

 •  Above  all,  they  were  most  

likely  to  use  the  evidence  to  support  exis5ng  decisions  than  to  make  new  decisions.  

•  Child  welfare  leaders  were  more  likely  than  leaders  of  other  systems  to  ignore  the  evidence.  

 

Factor   Mean  SIEU  score  

Acquisi'on  of  evidence  (Input)   2.84  

Acquisi'on  from  experts   3.13  

Acquisi'on  from  documents  and  published  materials  

2.98  

Acquisi'on  from  network  members   2.53  

Evalua'on  of  Evidence  (Process)   3.74  

Self  assessment  of  relevance   4.08  

Self  assessment  of  validity  and  reliability   3.70  

Assessment  by  others   3.40  

Applica'on  of  Evidence  (Output)   3.22  

Use  the  evidence  to  make  or  support  decisions   3.65  

Not  ignore  the  evidence   3.18  

Total  Use  of  Research  Evidence   3.38  

5.  Systems  leaders  use  three  other  types  of  evidence  when  considering  whether  to  seek  and  apply  research  evidence  in  making  decisions    

 •  Evidence  of  resources  necessary  and  available  for  making  use  of  

research  evidence  (supply),    •  Evidence  of  the  need  for  research  evidence,  usually  obtained  

from  local  condi'ons  of  client  and  service  needs  (demand),  and    •  Evidence  gained  from  personal  experience  (i.e.,  is  the  research  

evidence  consistent  with  prac5ce  experience  or  personal  observa5on).    

6.  Certain  ins'tu'onal  incen'ves  support  the  use  of  research  evidence  

•  Accountability  –  Systems  are  increasingly  being  required  to  u'lize  EBPs  if  they  wish  to  be  paid  for  the  services  they  provide.  

–  Choice  of  EBPs  requires  use  of  research  evidence.  •  Con'nuous  quality  improvement  

–  Ac've  engagement  in  URE  can  help  to  insure  that  current  prac'ces  meet  highest  standards  and  if  not,  what  prac'ces  can  achieve  these  standards  and  how  these  standards  can  be  implemented  and  sustained.  

7.  Two  specific  obstacles  to  use  of  research  evidence.  #1:  The  disconnect  between  “local”  and  “global”  

evidence      Global  Evidence  

•  External  –  originates  outside  of  agency  or  jurisdic'on  

•  Based  on  standards  for  scien'fic  rigor  (e.g.,  RCTs)  

•  Emphasis  on  generalizability  or  transferability  or  findings  from  one  state  or  county  to  another    

Local  Evidence  •  Internal  –  originates  within  agency  

or  jurisdic'on,  may  include  administra've  data  

•  Based  on  personal  experience  (either  involvement  in  data  collec'on  and  analysis  or  familiarity  with  popula'on  studied)  

•  Emphasis  on  uniqueness  of  popula'on  and  its  needs  (specificity)  

7.  Two  specific  obstacles  to  use  of  research  evidence.  #2:  The  disconnect  between  the  producers  and  

consumers  of  research  evidence    •  Evidence  constructed  on  the  basis  of  Randomized  Controlled  

Trials  has  poor  external  validity  because  these  trials  omen  cannot  mirror  real  world  condi'ons.  

•  Much  evidence  is  produced  in  the  absence  of  engagement  with  consumers  beforehand,  resul'ng  in  limited  u'lity  and  relevance  to  local  condi'ons.  –  This  is  why  EBPs  are  omen  used  without  fidelity,  when  they  are  used  

at  all.  

7.  Two  specific  obstacles  to  use  of  research  evidence.  #2:  The  disconnect  between  the  producers  and  

consumers  of  research  evidence    

•  Consequently,  the  methods  we  use  to  generate  evidence  are  omen  not  properly  aligned  with  the  nature  of  current  policy  problems.  – When  you  need  answers  now,  wai'ng  for  a  researcher  to  submit  a  proposal,  get  funding  to  conduct  a  study,  publish  the  evidence,  seek  confirma'on  of  the  findings,  etc.,  is  just  a  frustra'ng  experience.  

 

8.  Research  evidence  can  bener  inform  policy  in  'mely  and  useful  ways    

 •  Through  research-­‐policy  partnerships.  •  Through  the  use  of  local  as  well  as  global  evidence.  •  Through  the  development  of  bener  methods  for  producing  and  dissemina'ng  research.  

 

9.  Effec've  partnerships  in  child  services  possess  a  set  of  common  elements      

•  An  effec've  and  sustainable  research-­‐prac'ce  partnership  builds  upon  the  exis'ng  organiza'onal  cultures  of  research  and  policy/prac'ce.    

•  However,  it  is  not  merely  an  aggrega'on  of  these  cultures  but  rather  the  product  of  their  transforma'on  resul'ng  from  the  exchange  of  understandings,  values,  aStudes,  and  rules  for  engagement  that  occur  between  researchers,  prac''oners  and  policymakers.  

Intrapersonal  (trust,  commitment,  openness,  honesty,  willingness  to  learn,  sensi'vity,  flexibilty)  

Interpersonal  (trust,  communica'on)  

Organiza'onal  (roles,  distribu'on  of  funds,  

goals,    transparency)  

Environmental  (funding,  resources,  

demand)  

Cultural  (guidelines,  brokers,  

exchanges)  

10.    Building  such  partnerships  and  promo'ng  use  of  research  evidence  requires  your  help  

•  Funding  •  Engagement  in  using  evidence  •  Par'cipa'on  in  partnerships  

THANK  YOU    

QUESTIONS?  

Co-Creating the Infrastructure for

Implementing Evidence-Based Practice

American Youth Policy Forum

Washington, DC

October 5, 2015

Allison Metz, Ph.D., Director National Implementation Research Network

Senior Scientist, FPG Child Development Institute University of North Carolina, Chapel Hill

Socially Significant Outcomes

Effective Interventions

Effective Implementation

Methods

Enabling Contexts

Formula for Success

Formula for Success

•  Effective collaboration of cross-sector child welfare stakeholders

•  Aligned policy and practice •  Organizational and system structures

to support and sustain improvement efforts

•  Intensive and structured feedback loops

•  A commitment to learning and intentional improvement

Enabling Contexts

Challenges and Opportunities

Research evidence can be continually improved, boosting sustainment in practice, and can enable ongoing learning among developers, interventionists, researchers and consumers (Chambers et al., 2014)

Improving Evidence

Improve outcomes: prevent placement, reduce repeat maltreatment, improve family functioning and child wellbeing. Increasing emphasis in child welfare on data-driven decision-making. 11 evidence-based models serving 3,000 families at any point in time.

Case Example NYC Public Child Welfare System

Making It Happen Stakeholder Interactions

Research demonstrates that successful uptake of evidence requires genuine interaction among stakeholders. However, we know that in many instances, collaborations among stakeholders, including researchers and community members, are strained by a lack of mutual understanding of each other’s goals and expectations.

Community and Consumers

Intermediary Organizations

Program Developers and Researchers

Service Providers

Funders and Policy-Makers

Co-Creation Environment

Infrastructure Outcomes

© 2014, Metz & Bartley

Making It Happen Co-Creation Arenas

Interactions take the shape of “mutual consultations” that mediate the use of research evidence in complex child welfare service systems and political contexts. • Model Developers and ACS (public agency)

• ACS and Service Providers

• Model Developers, ACS and Service Providers

Making It Happen Mutual Consultation Major

Two-way communication Minor

One-way communication Absent

Lack of communication

joint activities common understanding; communication, negotiation and mutual understanding; development and use of products and processes

one-way communication; lack of common understanding; convincing one another of perspective

disagreement and mutual misapprehension

Ø  Intensity Ø Structure Ø Product development

Making It Happen Mutual Consultation

ACS and Model Developers

Address

misalignment between

policies and standards and

practice expectations

ACS and Service

Providers

Build capacity to support the

use of evidence

ACS, Model Developers and

Service Providers

Translate outside evidence into child welfare

context and use data to drive

decision-making and optimization

Co-Creation

Products and Processes

•  translate evidence into child welfare context Logic Models

•  revised to address misalignment between policy and practice Preventive Standards

•  web-based decision-making tool for referrals that uses predictive analytics Referral Systems

•  common constructs across models Fidelity Desk Guide

•  implementation science to build capacity of agencies to use evidence Learning Modules

•  risk assessment, case management Tailor Models

Co-­‐Crea5ng  Capacity  

ACS  

Providers   Model  Developers  

Families  &  Communi5es  

Product    

Co-Creation Recommendations

Intensive interactions among public child welfare agencies, researchers, and service providers…

with

Highly structured communication and feedback protocols

and a

Focus on product development

Goals for Today’s Session Thank You

Casey Family Programs

William T. Grant Foundation

Goals for Today’s Session For More Information

Frank Porter Graham Child Development Institute University of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/ www.scalingup.org

www.implementationconference.org

Allison Metz

allison.metz@unc.edu

From  Research  to  Learning,  Policy,  and  Breakthroughs  

Alex  Morales,  CEO  Children’s  Bureau  of  Southern  California  

 American  Youth  Policy  Forum:  Research  Use  In  Policymaking  

October  5,  2015  

Key  Points  to  Deepen  Learning  into  Prac'ce,  Policy  and  Breakthroughs  

   •  Launched  from  a  prac'ce-­‐based  research  project  to  organiza'onal  learning  culture  

•  Research  that  immediately  benefits  prac''oners  •  The  Disrup've  Ques'on:    How  can  society  move  to  scale  with  limited  financial  resources?  

•  Power  of  integra'ng  prac'ce,  research,  learning,  policy  and  advocacy  

•  Cri'cal  funding  for  research/evalua'on/learning  

30  Years  Ago  Launched  First  Prac'ce-­‐Based  Research  Project  

•  Tes'ng  of  an  interven'on:  in-­‐home  counseling  to  prevent  foster  care  placement.  

•  Development  of  the  Family  Assessment  somware  to  assist  counselors  in  family  assessment,  planning  interven'ons,  and  measuring  impact.  

•  Successfully  advocated  for  legisla'on  in  California  to  fund  in-­‐home  family  preserva'on  counseling.  

Strengthening  Prac'ce  with  Family  Assessment  and  Planning  Somware  

Building  a  Learning  Organiza'on  

•  ChildStrength:    Children’s  Bureau’s  in-­‐house  research,  evalua'on,  and  con'nuous  learning  program.  

•  Engagement  of  all  stakeholders  in  learning:  – Clients  – Staff  – Management  – Board  of  Directors  – Outside  researchers  

The  Breakthrough:    Crea'ng  a  Learning  Network  of  70+  

Organiza'ons  and  Engaging  Parents/Neighbors  

•  A  voluntary  network  made  up  of  government,  non-­‐profits,  faith  community,  universi'es,  and  businesses  to  work  in  a  new  way.  

•  Engaging  volunteer  Belong  Neighborhood  Ambassadors  to  create  a  movement  where  everyone  works  to  build  a  neighborhood  where  we  all  would  want  to  Belong.  

An  Example  of  Learning  Our  Way  Forward:  

A  Scalable  Model  to  Promote  Reading  

•  Improving  daily  parent  reading  to  their  young  children  under  age  5  years  from  43%  to  60%.  

•  Harnessing  the  power  of  trained  volunteer  parents  to  teach  &  promote  reading  to  their  neighbors.  

 

This  is  What  it  Looks  Like