Practical and Creative Solutions to Cross-Cutting Challenges to the Evaluation of

Children’s Bureau Diligent Recruitment Projects

2011 National Child Welfare Evaluation Summit

Grand Hyatt Washington, D.C.

August 31, 2011

Panel Participants

• Elliott Graham, Ph.D., James Bell Associates (moderator)

• Shannon D. Rios, Ph.D., Oklahoma Department of Human Services, Office of Planning, Research and Statistics

• Michele Hanna, MSW, Ph.D., University of Denver Graduate School of Social Work

• Crystal Collins-Camargo MSW, Ph.D., University of Louisville Kent School of Social Work

• Additional contributions from Amy D’Andrade, MSW, Ph.D., San Jose State University School of Social Work

Cross-Cutting Evaluation Challenges for the2008 Diligent Recruitment Grantee Cluster

• Dealing with “survey fatigue” and low response rates, esp. with certain research subjects (e.g., control group children/families, resource families, and agency managers and executives).

• Evaluating a program that is constantly changing, i.e., major

program activities, services, etc. are modified during grant in response to personnel, organizational, budgetary, political, othercontextual issues.

• Addressing bureaucratic/institutional barriers to primary data

collection, e.g., gaining access to case records and other client-level data sources while abiding by laws and policies governing data privacy and confidentiality.

Survey Fatigue and Survey Response Rates: Addressing the Challenges

Shannon J. Rios, Ph.D.

Oklahoma Department of Human Services

Oklahoma’s Diligent Recruitment Project: Bridge to the Future

Year 1

Collect data

Use data to create implementation

plan

Identify or develop training to support

implementationIntegrate grant initiatives with other existing grants,

projects, and recruitment acitivies throughout the state

to insure system change and sustainability

Years 2-5

Increase core skills and capabilities of resource

families, community partners & child welfare staff

through training

Implement customer service interventions with staff

Implement systematic process improvements to

increase program recruitment, retention and the

completion of the approval process

Collaborate & continue to develop public/private

partnerships

Oklahoma’s Evaluation Plan

• KIDS data (SACWIS)

• Focus Groups – 11 total

• Surveys – 21 different instruments to date, 11 of which are ongoing

• CW Staff and Supervisors, CW Administrators and County Directors, Resource and Pre-Resource Families, Community Partners (including Tribes)

Measurement with Families

Surveys of Resource Parents

• 2009 Survey with Current Bridge Resource Families: Random Sample of 764 current parents selected (emailed 135, mailed 629), n=146, 19% response rate

• Strategies included: drawing for iPod shuffle, sending out both emailed and mailed instruments

• Realized that our emailed surveys got much higher response rates than our mailed surveys

Response Rates & Survey Fatigue

• 2010 Survey with Current Bridge Resource Families: Non-Random Sample of 2,516 parents, all via emailed link to survey, n=549,21% response rate. Strategies included: • Collected email addresses in advance via a

postcard mail out

• Only mild improvement in response rate but much larger sample size

• Demographic comparison of this sample v. generalpopulation of parents

Measurement with Administrators

Implementation Reports with administrators : administered via emailed survey instruments

• Year 1 - 11/11 Leadership Staff completed, 100% response rate

Took 6 weeks to collect the data.

Quality of responses was poor and lacked detail

• Year 2 - 6/11 Leadership Staff completed, 54% response rate

Took 3 weeks for data collection

Complaints about how time consuming it was to complete

Quality of the responses was poor and lacked detail

––

–––

Response Rates & Survey Fatigue

Strategies Implemented:

Year 3 – 15/15 staff participated, 100% response rate

• Implementation Data from administrators collected via Focus Group

• Provided lunch

• Rich detail and high-quality responses

• More people contributing feedback than we had gotten in the past

Measurement with Staff

Surveys with field staff: high response rates, 87-90%

Strategies:

• Email sent out in advance to County Directors by our Director of Field Operations Division

• Random sampling

• Email sent with link to survey including information about:

• What data will be used for

• Why data is being collected

• For whom data is being collected

• Ensuring confidentiality

• How much time it will take them to complete

Santa Cruz County : Roots and Wings Project

• General recruitment activities

• Targeted recruitment activities for high removal communities

• Child-specific recruitment activities and services

• Systems-change to promote concurrent planning/permanency

• Enhanced resource family support, training, and services

Santa Cruz: Initial Survey Strategy

• Worked with county to keep survey as brief as possible

• Determined that email would not work with foster caregiver population; mailed survey instead

• Used Dillman's survey process:

– Intro letter from county head, letter with survey a few days later, thank you card about a week after that, then, follow-up letter and survey to non-respondents

• Incentive Year 1 - coffee gift cards

40% response rate •

Santa Cruz: Year 2 Strategy

• Incentive of pretty stamps, plus:

– Added incentive of entering drawing for $100 giftcertificate to clothing store if response received

– Added contribution to non-profit aiding foster youth in name of Santa Cruz county caregivers if response rate hit 50%

• 50% response rate.

Santa Cruz: 2011 Changes this year:

• More clearly explain purpose of pretty stamps (some respondents added them to the pre-stamped envelope we'd provided rather than keeping them for their own use)

• Contribution to non-profit if response rate hits 60%

• Include a 2-page summary of prior surveys with initial letter

• Each year drop a question or two that wasn't useful

Evaluating a Moving Target: Denver’s Village

Michele D. Hanna, MSW, Ph.D. University of Denver

Graduate School of Social Work Butler Institute for Families

Denver’s Village Wrapping Families with Community Support

Margaret Booker – Project Director Denver Human Services

Linda Trantow – Project Administrator Denver Human Services

Anthony Clayton – Project Coordinator Fabiola Esposito – Grant Specialist

Denver Human Services

Michele Hanna – Evaluator Butler Institute for Families

University of Denver

Project Description

• Denver’s Village is a community-based, data-driven recruitment model focused on keeping children in their home neighborhoods and recruiting resources who reflect the race and ethnicity of the children in the care of Denver Department of Human Services. Major components include: – Recruitment & Retention – Agency Cultural Shift – Permanency & Concurrent Planning – Data Management

Grant Strategies

• Community Based Resource Teams (CBRTs)

– CCPC site coordinators

– Community Outreach Workers

– Other community members

– Denver DHS staff

• Engaging the ethnic communities

– African American, Latino/Hispanic, Native American

• Engaging the community of resource families

• Engaging youth

• Engaging DHS management & staff

Evaluation

• Process • Observations

• Interviews

• Progress Towards Goal Instrument (PTGI)

• Resource Family Surveys – Spanish Speaking Focus Group

• Outcome • Community Level

• County Level

The Moving Target of Public Child Welfare

• Re-organization

• Leadership change

• Fiscal challenges

• Staff turnover

• Shifting priorities

• Concurrent implementation of multiple initiatives

The Moving Target – A Mile High

Major Evaluation Challenges

• Competing goals Year 1

– Preexisting county goals

• Administrative data

– Third party access

– County administered system

• Initiatives, Initiatives, Initiatives

– Minimal evaluation staff

– Prioritization

Denver’s Village: Year 1

Planning Year

• Driving the car without an engine

• Pre-existing county goals

• Pre-existing infrastructure

Denver’s Village: Year 2

• Restructuring

– Fiscal Agent

– COWs

• Grant Objectives andGoals

– Logic Model

– Realistic

Denver’s Village: Year 3

• Initiatives

– Kinship Dream Team

– Customer Service

– Targeted African American Recruitment

27

DU - Evaluation

DATA TRAILS

Work Management System

Denver Child Placement Database

AdoptUSkids Website

Permanency &

Concurrent Planning

Kinship Support – Dream Team

AdoptUSkids

Website Partners for Permanency (P2) Project

Adoption Exchange

Expedited Adoption Program

Customer Service Initiative

Work Culture/ Climate

Learning Circles

Development of Department Child Welfare Values

Child Placing Agency (CPA) Collaborative

Community Recruitment Events

Community Outreach Workers

SupportGroups

Community Collaborative Partnership Centers

Denver’s Village

Recruitment & Retention

Permanency Decision Making (PDM)

Agency Cultural

Shift Children’s Bureau T/TA

Community Based Resource Teams

Information Sessions

Evaluation Priorities

• Data

– Logic model

• Kinship Surveys

• Resource Family Surveys

– Spanish speaking focus groups

• Placement Exit Surveys (Year 4)

• Ongoing initiative process observations

Recommendations

• Evaluation must be guided by the logic model

• Ongoing communication with grant management

• Prioritize

– Know your limitations, i.e. staff, etc. …

– Relate to goals and objectives of grant, i.e. logic model

• Be flexible and adapt

Bureaucratic and Organizational Barriers to Data Collection in Child

Welfare

Crystal Collins-Camargo, MSW Ph.D.

University of Louisville Kent School of Social Work

Kentucky’s Diligent Recruitment Project: Project MATCH

• Implementing an array of inter-related interventions in 4 service regions using a quasi-experimental design – Targeted and child-specific recruitment through DR specialists,

market segmentation and family-finding – Customer service in recruitment, selection and retention – Collaborative use of data for localized practice improvement

among public and private agency staff and resource parents

• Evaluation includes analysis of aggregate data quarterly; pre-, interim and post-intervention surveys, focus-groups, and interviews – Statewide and in the 4 intervention regions – Public and private sectors – Staff and resource parents

Child Welfare Systems are Under a lot of Pressure

• Diversion from agreed upon protocols for procedures, instrumentation, and timing of data collection

• Pressure to rollout promising practices statewide despite comparative research design

• Agency tendency to select intervention regions through cherry picking or to rescue areas that are struggling

Child Welfare Data Systems

• Reliability and worker data-entry

• System overhauls midstream

• Maturity and nature of relevant data indicators and how they are represented in the system

• De-identification of data

• Connecting indictors from multiple databases (i.e. data related to children; resource parents; public and private agencies)

Access to Subjects

• Public agency staff – Workload and priorities – Morale

• Public agency resource parents – Contact information inaccurate/changing

• Private agency staff and resource parents – Lack of contact lists – Issues with inter-organizational trust and collaboration – Private agency sanction/authorization

• Timing – Differing schedules of staff vs. families are complicated by

agency overtime policies and workload/fiscal realities – Agency initiatives and crises

Institutional Review Board Issues

• Conflicting reviews from university and agency IRBs

• Disagreement on informed consent language

Strategies for Addressing Institutional Barriers

• The public agency must be actively engaged in evaluation design and implementation – But control as much actual data collection as possible – Have a high-ranking agency representative on the

project team

• Recognize that researchers, practitioners and administrators are coming at this from different perspectives – Purpose – Timeframes – Measurement

Strategies Continued

• Leverage active support of agency leadership often

• It is all about relationships at multiple levels in the organization

• Share data with stakeholders regularly and use it as the impetus for problem-solving regarding data collection and interpreting findings—use healthy competition to your advantage

Strategies Continued

• Keep the reason for the evaluation at the forefront of all discussions—make it about helping the agencies achieve their outcomes – Put evaluation on every meeting agenda/report

– When possible someone from the evaluation team should be a part of every project discussion—even if it is largely focused on the intervention design or logistics

• Tie the intervention and its evaluation to key agency priorities such as the CFSR PIP.

• Build in multiple methodologies and data points so that when inevitable problems occur you are not caught with no data

• Employ a participatory, utilization-focused approach

Contact Information

Michele D. Hanna, MSW, Ph.D. Associate Professor University of Denver

Graduate School of Social Work 2148 S High Street

Denver, Colorado 80208 303.871.3444

mhanna3@du.edu

Crystal Collins-Camargo MSW, Ph.D. Assistant Professor

University of Louisville Kent School of Social Work

307 Patterson Hall Louisville, KY 40292

502.852.3174 crystal.collinscamargo@louisville.edu

Shannon J. Rios, Ph.D. Research Manager

Office of Planning, Research and StatisticsOklahoma Department of Human Services

Oklahoma City, OK 73125 405.522.3167

shannon.rios@okdhs.org

Amy D'Andrade, MSW, Ph.D. Associate Professor

San Jose State University School of Social Work

One Washington Square, WSQ 215H San Jose, CA 95192-0124

408.924.5830 amy.dandrade@sjsu.edu