Narratives

Executive Summary

OVERVIEW: United Way for Southeastern Michigan (UWSEM), an existing grant making

institution, is the intermediary of the Bib to Backpack (BtoB) SIF initiative. BtoB is geographically

based in the Metro Detroit region of Michigan (Wayne, Oakland and Macomb counties) and is

focused on Youth Development (Technology, Early Childhood Education) This tri-county area faces

many economic, social and educational challenges, with particularly acute need in Detroit and

Pontiac. Subrecipients will include non-profit agencies that have deep roots in tri-county communities

of need and understand the cultural, environmental and social context our families are facing. Wayne

State University, Social Solutions, and Help Me Grow-Michigan will each play a key role in

implementation of strategy. UWSEM requests $1.5 million for each of 5 years. Sources of

commitment of match include Community Impact funds directed to BtoB and UWSEM Campaign

funds.

PROJECT SUMMARY OVERVIEW: The proposed SIF, Bib to Backpack, impacts society on three

levels. From a systems-building perspective, we will work with subrecipient organizations to expand

programming, strengthen connections between programs, and increase outcomes. From a

community and family perspective, we will support skill-building for parents and other adults who

care for young children to empower those adults to expand their social and resource networks and

increase their self-sufficiency and use of resources. All this work converges on the third level, that of

young children, improving their developmental readiness and setting them up for lifelong success.

THEORY OF CHANGE: We will build a parenting continuum that recruits, engages, and

empowers parents to improve their family's self-sufficiency and parenting skills and, in turn, their

children's developmental readiness with the following framework: (1) Recruit eligible families through

a network of new and existing partners and systems into The Incredible Years, REDI-P adapted Play

& Learns and Baby Box; (2) Co-create a plan with the family to address concerns identified by

engaging the families with the Family Check-up model, the Arizona Self-Sufficiency Matrix and the

Ages and Stages Questionnaire; (3) Use our existing 211 community-resource call center and 211 Care

Advocates to create a 'light-touch' care coordination system to help families stay on track and

navigate the early childhood ecosystem; (4) Continue to monitor participant families annually to

discuss progress, new priorities, and potential supports; (5) Evaluate the impact of the integrated early

childhood (EC) system on families' self-sufficiency and developmental readiness scores.

SUBRECIPIENT STRATEGY: A clear and effective Request for Qualifications (RFQ) process will

include inviting agencies in Macomb, Oakland, and Wayne counties to bid on work based on our

For Official Use Only

Page 2

Narratives

Theory of Change. Selection will depend on a scoring rubric and an external review panel. Clear goals

and targets that support the end-goals and the evaluation of same will be included in contracts and

reviewed. Continuing funding will be based on performance. UWSEM has a proven track record of

effective subrecipient growth by taking an asset-driven perspective and applying technical assistance

and capacity-building supports in order to expand upon an agency's strengths.

UWSEM'S ASSETS: UWSEM has in place a highly-effective, robust early childhood department,

including a network of Early Learning Communities that achieve early childhood outcomes with early

learning resources, technical assistance, training, evaluation support and funding. UWSEM's

experience, accomplishments and outcomes operating and overseeing programs in Technology

include leading 211, Michigan's largest community resource Call Center. We also have two Social

Solutions Efforts to Outcomes databases supporting our EC work and have several staff capable of

adapting the software to new data and reporting needs. UWSEM's internal Data and Evaluation Team

manages central data collection and evaluation activities for our partners and for 211. Staff and

consultants are richly experienced and well-educated in a wide range of disciplines.

Program Design

(b) PROGRAM DESIGN

(i) RATIONALE AND APPROACH

Intro: The United Way for Southeastern Michigan (UWSEM) believes that every family has the

right to the supportive networks and resources that middle- and upper-income families take for

granted. If we can increase access to supportive networks and resources for families in at-risk

communities, then the families and children in those communities will be more resilient. Parents and

other adult family members will be self-sufficient. Children will be healthy, valued and thriving.

We take a multigenerational approach to the work of early childhood (EC) well-being and

developmental readiness. By ensuring that the adults who care for children have access to the

supports they need, and that they are empowered to make the best use of those supports and to be the

voice for their young children, those adults can take better care of their children and themselves.

The proposed SIF builds on our existing early child work, "Bib to Backpack", and impacts society

on three levels. From a systems-building perspective, we will work with subrecipient organizations to

expand programming, strengthen connections between programs, and increase outcomes. From a

community and family perspective, we will support skill-building for parents and other adults who

care for young children to empower those adults to expand their networks and increase their self-

sufficiency and use of resources. This work all converges on the third level, that of young children,

For Official Use Only

Page 3

Narratives

improving their developmental readiness and setting them on a trajectory for lifelong success.

This body of work is conceived of as a continuum -- a series of discrete elements that lead to a

transformational change (Overton, 2013) for families and children. We will build out a continuum

that starts with pregnant mothers and ends with a connection to school entry. Continuum approaches

are ways to track, analyze, and build connectivity among families and organizations to influence

systems for successful results (Bailey & Terry, 2006). The welcoming, intentional pathways of BtoB

will empower our target populations, pregnant mothers and the adults who care for young children,

to easily access a series of evidence-based resources that will lead to positive outcomes for the children,

the adults, and our communities.

STRATEGY B: The proposed BtoB SIF will be a geographically based program focused on Youth

Development throughout Michigan's Wayne, Oakland, and Macomb counties. Within this region, it is

our mission to ensure that children 0-5 are healthy, valued, and thriving and we believe this happens

as a result of knowledgeable, responsive, and integrated parenting. Our goal, specific to that mission,

is to build a parenting continuum that will recruit, engage, and empower parents to improve their

family's self-sufficiency and, in turn, their children's developmental readiness. By building this system

on a foundation of evidence based parenting strategies and technologically driven care-coordination

supports we believe we can both educate and empower parents to improve the developmental

readiness and well-being of their children.

Theory of change: Build a parenting continuum that will recruit, engage, and empower parents to

improve their family's self-sufficiency and parenting skills and, in turn, their children's developmental

readiness. This will be accomplished by using the following framework: (1) Recruit target-population

participants through a network of new and existing partners and systems into The Incredible Years,

the Research Based Developmentally Informed Parenting Program (REDI-P) adapted Play & Learns

(REDI-PPL) and Baby Box (BB); (2) Once they are enrolled in programming, engage interested,

eligible families with the Family Check-up model (FCM) and perform full family self-sufficiency and

developmental screenings using the Arizona Self-Sufficiency Matrix (ASSM) and Ages and Stages

Questionnaire (ASQ) and co-create a plan with the family to address immediate concerns; (3) Use a

'light-touch' care coordination system to help BtoB SIF families stay on track and navigate the early

childhood ecosystem. We will use our existing 211 system and 211 Care Advocates to help monitor

participant progress and make additional referrals based on the families' plans; (4) Continue to

monitor participant families annually to discuss progress, new priorities, and potential supports; (5)

Evaluate the impact of the integrated EC system on families' self-sufficiency and developmental

For Official Use Only

Page 4

Narratives

readiness scores.

CRITICAL PROBLEM (RATIONALE): The tri-county area in Southeastern Michigan faces many

economic, social and educational challenges: almost half (48%) of the estimated 236,511 children 0-4

years of age live below 200% of the federal poverty rate. Detroit's unemployment rate at 11% is twice

the national average. County unemployment rates are more in line, as the Bureau for Labor Statistics

reports them in March 2016 at 6.4, 4.3 and 5.7 percent in Wayne, Oakland and Macomb, respectively.

While economic challenges are prevalent throughout the region, the need is particularly acute in

some communities, including the cities of Detroit and Pontiac. Detroit has a poverty rate of 40%, with

two-thirds of children under 18 living at less than 125% of poverty (U.S. Census Bureau, Table S1703,

2010-2014). Of adults over the age of 25, 22% do not have a high school diploma (compared to 13.6%

nationally; U.S. Census Bureau, Table S1501, 2010-2014). In Pontiac, conditions are similar with a

38% poverty rate and 61% of children under 18 living at less than 125% of the federal poverty rate

(U.S. Census Bureau, Table S1703, 2010-2014). Of adults in Pontiac, 23% do not have a high school

diploma (U.S. Census Bureau, Table S1501, 2010-2014).

Language will also be an important consideration for implementing the proposed SIF initiative. In

Macomb County, 13.5% of the population speaks a language other than English, with 42.8% speaking

English "less than very well". In Oakland and Wayne Counties, 14% and 37.2% of the population

speaks a language other than English respectively, with 32.6% and 37.2% speaking English "less than

very well". (U.S. Census Bureau, Table S1601, 2010-2014).

The social and economic challenges faced by our region have many implications for the health and

developmental readiness of our young children. Infant mortality remains a great community concern

as Detroit is over twice the national average with 13.2 infant deaths per 1,000 reported, compared to

5.9 nationally. There are 10.3 infant deaths per 1,000 reported in Pontiac and the infant mortality

rates in the three counties, Wayne, Oakland, and Macomb are 9.6, 6.3 and 5.8 respectively (Michigan

Department of Health & Human Services, 2010-2014).

The developmental readiness of our region also remains an area of deep concern. UWSEM heavily

invests in and promotes the use of the Ages and Stages Questionnaire (ASQ), a screening tool that

monitors young children's (ages 0-5) development and focuses on children's general and social-

emotional development. Investment in these screenings is important because children living in high-

risk communities suffer the most frequent and the most severe developmental delays and take longer

to recover from delays than children living in other communities. High-risk communities are

generally characterized by concomitantly high rates of poverty, unemployment and violence.

For Official Use Only

Page 5

Narratives

71,749 Ages and Stages Questionnaire screens have been completed on children 0-5 as of

September 2015 in the three counties and confirm this finding. In each domain, over a third of

children screened were at risk. The following is a breakdown of percentage of children at strong risk or

potential risk, combined, of each developmental domain: Communication- 36%, Gross Motor- 38%,

Fine Motor- 54%, Problem Solving- 38% and Personal/Social- 35%. According to James Heckman,

Nobel Laureate in economics, addressing these deficiencies early has vast returns both socially and

financially for both the children and society (www.heckmanequation.org).

CURRENT SOLUTIONS NOT WORKING: In these at-risk communities, there is no system that

readily connects people to all of the resources they need to support them as parents and to meet their

objectives as families. While many resources are available, they are sometimes underutilized and there

is a lack of continuity and connectedness to ensure that families can easily identify and access the

specific supports they need. Parent support programs exist in Greater Detroit, yet they are not

coordinated and connected in a way that can be monitored to better understand how to best serve our

community and achieve outcomes at scale, while efficiently stewarding our resources.

Although we have deep regional penetration around ASQ screenings, following-up and connecting

parents and caregivers with referrals and ongoing support remains a challenge and an area that

requires further investment. While each county has active pilots to better understand how to track and

support connections to referred services, few have the staff capacity to make sure that the referrals

lead to connections. Creating a connection to our 211 system under this pilot would fund our Care

Advocates to track our participants using Help Me Grow (HMG; an ASQ implementation

organization) services and perform the follow-ups necessary to ensure connection to services, while

allowing HMG staff to focus on building their network of referral agencies and analyzing screens for

referrals.

We see a similar issue in our own 211 call center. Our call center is a 24/7 referral service that

connects people with information and resources to build healthy, safe communities. The center serves

eight (8) counties in southeastern Michigan, including Wayne, Oakland and Macomb counties. Every

year the center receives approximately 300,000 calls from individuals needing assistance. 211

maintains a validated database of approximately 12,000 programs and 2,000 organizations and

agencies.

There were a total of 10,347 calls from individuals seeking individual, family, and community

assistance, not including. However, not all of these calls result in a referral for the individual for a

variety reasons. 7.7%, 18.8%, and 9.5% of calls seeking individual, family, and community assistance

For Official Use Only

Page 6

www.heckmanequation.org).�

Narratives

calls ended with no referral being made for Wayne, Macomb and Oakland respectively.

Additionally, not all individuals referred by 211 receive services from the agencies they were

referred to. On a 2015 211 follow-up survey comprised of 1,125 respondents, 69.4% reported not

receiving services from the referred agencies. The most common reasons respondents reported not

receiving services were 1) no funds available (17.8%), 2) agency determination pending (13.8%) and 3)

Full/At Capacity/ Waitlist (11.4%). By maintaining consistent contact with B3 families we hope to

keep them engaged with our system to work through these barriers in partnership.

Each of these programs are largely successful at what they were designed to do and we know that

with additional coordination supports we can bolster their effectiveness with our most at-risk families

through the approach outlined below.

APPROACH - PROPOSED EVIDENCE-BASED INTERVENTION COMPONENTS: These are

descriptions of the core, evidenced-based components of our BtoB continuum. Please see the

evaluation section for the justification regarding their evidence status; each core component is

currently either at a preliminary or moderate level of evidence. The section following this describes

how the components are integrated and how a family will navigate through the system under our

Theory of Change.

Family Check-Up Model (FCM): the FCM is a care management model that involves a lighter

version of care coordination than many existing models. It differs from traditional clinical models and

practice in three important ways: (1) it utilizes a health maintenance model, (2) it derives much of its

power from a comprehensive assessment, and (3) it emphasizes motivating change. In contrast to the

standard clinical model, the health maintenance approach of the FCM explicitly promotes periodic

contact with families (yearly at a minimum) over the course of key developmental transitions (Gill, A.

et al, 2008), potentially reducing coordination costs.

The FCM facilitates change through two main components: motivational interviewing and family

management practices. In working with families of young children, the FCM motivational

interviewing session is designed to elicit motivation for the parent(s) to change problematic behavior

in their child, which is often achieved by modifying parenting behavior (Forgatch et al., 2005) or

aspects of the caregiving context that compromise parenting quality. The motivational interviewing

focuses on sharing the assessment data results and working with the family to identify potential

problem areas they wish to address (Gille, A., et al, 2009). Our model includes use of the Ages and

Stages Questionnaire (ASQ), when appropriate, to empower parents with additional developmental

readiness information and referrals aimed at improving or maintaining their children's developmental

For Official Use Only

Page 7

Narratives

readiness. By using these techniques and focusing on priority areas for the family we hope to increase

and maintain engagement in the SIF parent continuum.

After understanding a family's motivation for change, the FCM provides options for intervention.

Options may include referrals for help with problems outside of parenting, but the focus of most

interventions address family management issues. Family management includes a collective set of

parenting skills, commonly referred to as Parent Management Training (PMT), based on social

learning principles of reinforcement and modeling (Forgatch et al., 2005; Patterson, 1982; Webster-

Stratton & Hammond, 1997). PMT has been consistently associated with improvement in parenting

and reductions in child conduct problems (Bullock & Forgatch, 2005; Patterson, Reid, & Dishion,

1992).

PMT focuses on four main skill sets for the parents of young children: limit setting, proactive

parenting, positive reinforcement, and relationship building. Using PMT typically involves providing

parents with a rationale to stimulate interest, careful explanation of new skills, and in-session practice

using role plays and practice with the child. In the FCM, PMT is applied to specific behavior problems

highlighted in the assessment. Our PMT interventions will consist of the Incredible Years Training

(IYT) and Play and Learns adapted from the REDI-P home visiting model (REDI-PPL), as well as

coordinated referrals into other UWSEM parenting classes and/or community supports to deal with

specific issues that may not be covered in the IYT or REDI-PPLs.

Incredible Years Training: This training is based on principles of video modeling, observational and

experiential learning, rehearsal and practice, individual goal setting, self-management, self-reflection

and cognitive self-control. It looks to reduce risk factors and increase protective factors through

increasing parent, teacher, and child competencies. Most of our recruited families will have self-

selected into the parenting classes and our focus will remain on the parenting components of the IYT.

Numerous randomized control group trials have been conducted over three decades by the developer

and independent investigators with parents in high risk populations (e.g., Head Start and Sure Start)

showing positive outcomes for parents.

REDI-P - REDI-P is a parent-child interaction intervention that is adapted from a preschool based

intervention. It involves a 16 "lesson" home visiting curricula which will be adapted for this proposal

to be used in play-and-learn groups (REDI-PPL). This intervention is considered to have preliminary

to moderate evidence as it has been evaluated with at least one randomized control trial comprised of

200 ethnically diverse children and families recruited from three counties in Pennsylvania. More

importantly, the components of the intervention are based upon an extensive review of the literature

For Official Use Only

Page 8

Narratives

on positive parent-child interactions and their link to positive child outcomes (Bierman et al, 2015).

Baby Box: Finland's universally-distributed Baby Box is literally a box outfitted with a mattress

that can be used safely as a bassinet. It is credited with helping to reduce that country's infant

mortality rate from 65 to 3 deaths per 1,000 children from 1938 to 2013. When distributed the Box is

filled with infant products and information linking new parents to critical resources.

Baby Box University: A vital component of the Baby Box program is Baby Box University. Expectant

and new parents have access to thousands of short, online videos, in many languages, from

professionals talking about safe sleep, breastfeeding, positive parenting and other important aspects of

caring for newborns. These educational opportunities are available on mobile devices 24/7 and

engagement can be tracked. Research has shown that one of the most universally impactful tools for

reducing child mortality is parent education.

THEORY OF CHANGE (TOC) and PROGRAM DESCRIPTION: Building a parenting continuum

that will recruit, engage, and empower parents to improve their family's self-sufficiency and parenting

skills will improve their children's developmental readiness. By adopting Strategy B and the above

evidenced-based interventions we can exercise greater oversight over the evaluation, increase the

chance of successful scaling by using well-known, evidenced-based national models, and family

resiliency.

Initially, families will be recruited into the SIF intervention by our partner agencies and

subrecipients and placed into one of two integrated pathways: expectant mothers and existing families

(families with children 0-5 already in the home). Specifically, we will recruit pregnant women,

parents, and families of children age 5 and under living at or below 200% or less of the FPL who reside

in Wayne, Oakland, or Macomb County. Initial recruitment will be based on a need recognized by our

partner organizations and the mother or family's interest in the Baby Box and/or our two parenting

supports: Incredible Years Training and REDI-P Play and Learns. All families participating in these

interventions will be prescreened for BtoB eligibility to ensure they are within our target population.

Subrecipients are ideally either parent-centric or multifaceted family services agencies with deep

roots in tri-county communities of need that understand the cultural, environmental, and social

context that our target families are facing. As an existing SIF intermediary, we have extensive

experience in identifying prospective partners whose work aligns with our TOC, and providing value-

added activities and supports to support their success. We will draw on this experience to ensure that

selected subrecipients are provided with capacity building technical assistance including but not

limited to: finance and accounting systems, procurement policies, National Service Criminal History

For Official Use Only

Page 9

Narratives

Check procedures, program goal setting and implementation, data collection and reporting, and rapid

cycle evaluation support.

Existing families meeting the BtoB SIF eligibility requirements will be recruited to fully participate

in the BtoB SIF program. Each BtoB SIF family will be engaged with the Family Check-up Model

(FCM), including a full family assessment with portions of the Arizona Self-sufficiency Matrix

(ASSM), a nationally vetted family self-sufficiency assessment, looking at relative poverty and a

supplemental child developmental screening using the ASQ. These assessments and screens will

inform the Family Plan that will be created in conjunction with a trained FCM Specialist (FCMS)

using motivational interviewing to help parents identify their family's priority areas as they discuss the

assessment results together.

At the end of the co-planning session, the FCMS discusses a menu of family-based interventions

with the caregivers. These options may include (a) regular follow-up support, either in-person or by

phone; (b) assistance with specific child behavior problems or parent issues; (c) Parent Management

Training; (d) preschool/day care consultations; and (e) community referrals with light care

coordination provided by specially trained 211 Care Advocates. The FCMS encourages the parents to

choose the level and type of services that best meet the family's needs.

Having completed the initial stages of the FCM, a family will have identified focus areas that they

wish to work on during the next 6-12 months and UWSEM will support their continued connection to

both evidenced-based SIF PMT models as well as other community referrals. The families FCM/ASSM

and ASQ results and family plan will be recorded and their activities will be tracked using the 211

system REFER and 211 care advocates. 211 care advocates, in partnership with the FCMS, will

monitor a family's progress through a centralized system put in place to send reminders, track PMT

activities, community referrals and connections, and other messages to keep families engaged with the

system. A variety of strategies will be tested to understand and improve the effectiveness of this

approach.

Text messaging was used successfully by nurse discharge planners to remain in contact with new

mothers who were discharged early. (Boe et al., 2015). The study suggested nurses were able to offer

support and information to mothers and their families in a way that enhanced the families' sense of

security and self-efficacy. The intervention included text chats with nurses as needed and automated

messages about their babies' first 4 days, delivered every 12 hours. The nurses regarded the automated

messages as a tool to stimulate both the families' curiosity and capacity to take control of their

situation. The nurses believed that the interactive links in the automated messages encouraged parents

For Official Use Only

Page 10

Narratives

to read additional information material in the knowledgebase or question the nurses on the postnatal

ward directly through their phones. They also felt it helped manage the vast amounts of information

provided at discharge (Boe et al., 2015). Both our 211 and ETO databases can be configured for text-

based messaging and information delivery and we will work with our IT department to design content

and messages around our BtoB focus areas, e.g. have you read to your child today?, follow this link

for a free interactive story.

Additionally, research performed by Dr. Vishal Mehta relying on randomized, controlled trials and

systematic reviews of the literature concludes that "there is now convincing data that text messaging

can influence health behavior and decisions." UWSEM's Healthy Kids staff have developed a similar

text-messaging system to help people find locations and set reminders for meal times at their food sites

around MI.

Engaging Expectant Mothers: Expectant mothers will largely be recruited using the UWSEM Baby

Box that will be promoted through partner organizations, including hospitals, family service

organizations, WIC offices, Nurse Family Partnerships, Maternal Infant Health programs, Early Head

Start programs and others who will inform expectant or new mothers of the box and provide them

with a link to three mandatory Baby Box University classes (approx. 5 minutes each). We will select

three pilot cities within our tri-county regions for universal deployment, meaning there will be no

restrictions on who may receive a Baby Box. Universal deployment is important because it removes

any implied stigma around receiving the box if every mother in the three initial test cities receives one.

Upon completion of these classes, families will receive an electronic certificate of completion which

they will present to a Baby Box distribution partner (all BtoB partners will be Baby Box distribution

partners) and receive their box. BtoB sites embedded in our most at-risk neighborhoods will then use

this contact point to recruit the mothers into BtoB SIF. Those interested mothers screened as eligible

will be enrolled into the BtoB SIF program and the FCM for additional assessment. It is expected that

UWSEM will distribute approximately 12,000 Baby Boxes each year, casting a wide net for

recruitment.

In addition to the initial entry tracks of expectant mothers and existing families, we have identified

three focus pathways that will be priorities as part of our pilot continuum. These are connections to

health care services, including insurance and primary care; connections to nutrition programming;

and connections to Help Me Grow's referral network. By addressing each of these early we can allow

the family to begin to focus on other social determinants and our parenting supports.

Connections to Primary Care: UWSEM recognizes that connection to consistent and quality

For Official Use Only

Page 11

Narratives

primary care is important for both maternal and child health and all BtoB SIF participants will be

screened on insurance and primary care status to determine if a referral needs to be made. We are

actively engaged in a pilot test using 211 Care Advocates to screen 211 callers for potential connection

to a primary care physician, health insurance, and other health services through our Children's

Healthcare Access Programs (CHAP) partners. CHAPs provide families with resources, programs,

services and specific health supports based on the callers' family needs and interests. Under this model,

a 211 Care Advocate addresses the caller's immediate reason for calling by making the needed referral;

however, the 211 CA will also ask probing questions to ascertain if a CHAP connection would be

worthwhile. The 211 Care Advocate then gives the caller general information about CHAP. Consent is

provided by the caller to get connected to a Virtual CHAP Specialist or local CHAP program, based

upon the county they reside in. The CHAP partner then helps connect the family to needed health

services. Under the BtoB model the connections and activities supported by CHAP will be recorded and

monitored to ensure that referral connections are being made and any barriers to services addressed.

Connections to Help Me Grow: We have a longstanding partnership with Help Me Grow -- a

national affiliate organization supporting the use of the ASQ -- to provide screening and referral

services for families using the ASQ. They currently have a network of approximately 400

organizations to help provide ASQ screenings and developmental services for families. Aside from a

few pilots to track referrals to pre-school and Head Start, their ability to monitor connections to

referred services is limited. Under this BtoB model if a family chooses to connect with HMG as a result

of a ASQ screen, our 211 Care Advocate would be able to record that referral and follow up to see if the

connection was made and work to remove any barriers if it was not, sharing back this information

with HMG regularly to help maintain an active network of agencies.

Connections to Nutrition Programming: Lastly, we intend to build on the work of our very

successful Meet Up and Eat Up summer meals program. Research indicates that proper parental

modeling during the toddler years plays a significant role in establishing longer-term eating behaviors

(Birch et al, 2007). We intend to use Cooking Matters to help our families learn how prepare healthy

meals on a budget. The first-ever long term study of Cooking Matters showed the six-week cooking,

shopping and nutrition course has a powerful, sustained impact that is significantly greater than

changes that would have occurred without an intervention. After the course, families felt more

confident that they will be able to afford enough food, were eating healthier, putting them at lower

risk for diet-related diseases like obesity, diabetes, heart disease and cancer, and cooking meals more

often, and making meals healthier and more budget-friendly. Additionally, there was a 10% increase

For Official Use Only

Page 12

Narratives

in families that are more confident in their cooking abilities and an 11% decrease in barriers seen by

families to making healthy, affordable meals. Cooking Matters is the nutrition education program of

Share Our Strength's (a current SIF Intermediary) No Kid Hungry campaign, an initiative to end

childhood hunger in America.

ALIGNMENT TO CURRENT SIF: We are not applying for renewed funding of our current 2011

SIF project, but UWSEM is a current recipient of other federal funding, and this proposal is an

opportunity to supplement and integrate those existing bodies of work into our continuum.

Specifically: (1) As a SIF intermediary (Youth Development), we fund five subrecipients to test

innovative approaches to kindergarten readiness. Approaches include family/child-centric

interventions, interventions focused on immigrant communities, and a healthy eating intervention.

This 2016 SIF proposal is an opportunity to create a true continuum that has the potential to connect

with the 2011 interventions. (2) As a SIF subrecipient of the intermediary Share Our Strength

(Healthy Futures), UWSEM is establishing regional collaborative tables to engage key local leaders in

each of the project's six geographic program target areas. They are working to provide comprehensive

outreach for their "Meet Up and Eat Up" summer meals program, contracting with breakfast coaches

to encourage alternative breakfast models, and recruiting year-round sites to serve both summer and

after school meals. Eventual integration with this network is a mid-term goal for our scaling efforts.

(3) As the state of Michigan's implementation partner for the Race to the Top - Early Learning

Challenge in Macomb, Oakland, and Wayne Counties, we work closely with licensed and unlicensed

childcare providers to ensure that children receive quality care. This SIF proposal further supplements

that work by ensuring that parents are provided with the supports they need to parent effectively and

understand the need for quality childcare.

Creating a coordinated system that can eventually incorporate all of our UWSEM programming

will help support and expand each of these programs, by helping ensure families that need the

supports offered by these programs connect with these programs.

In creating a continuum of adult-focused supports for children's earliest years, BtoB SIF will

ultimately incorporate and/or connect to all of the UWSEM's interventions that make up our existing

body of work in early childhood, Bib to Backpack. With appropriate data collection, care coordination,

and referrals, we will ensure that families empowered by BtoB SIF are invited to other interventions.

Each intervention will become part of a fully-integrated system supporting young children and their

families. Parents and other adult family members will be self-sufficient. Children will be healthy,

valued and thriving.

For Official Use Only

Page 13

Narratives

ii. Proposal for Subrecipient Selection

SUBRECIPIENT PROFILE: Subrecipients best suited to implement our approach would share a

profile that demonstrates the ability to implement parent-focused interventions with fidelity.

Investments will be made in tri-county nonprofit agencies that have the capacity to recruit and retain

participants, to undertake data collection, to help users navigate through the social services system,

and to integrate referrals across the continuum for the best possible user experience. Consistent with

our theory of change agencies will meet parents at whichever door they enter, and engage and

empower them to best prepare their children to succeed.

PLAN FOR SUBRECIPIENT SELECTION: Our plan to carry out a competitive selection process is

clear and comprehensive and is formulated based on extensive experience in doing so successfully.

Subrecipient agencies are ideally either parent-focused or multifaceted family services agencies with

deep roots in tri-county communities of need. We expect to fund four to six agencies with grants

ranging from $150,000 to $300,000. This will allow for a total programmatic investment of

$300,000 to $600,000 per subrecipient. Subrecipients will meet the criteria outlined in Appendix 1.

Based on our eight years' experience implementing Early Learning Communities in greater Detroit,

we are confident that budgets in this range will allow for appropriate programming, data collection,

and referral support.

As an existing SIF intermediary, we have extensive experience in identifying prospective

subrecipients whose work aligns with our Theory of Change, and providing value-added activities and

supports to support their success.

Our plan for carrying out a clear and effective bidding process will mirror our recent experience in

re-bidding our Early Learning Communities (ELCs) in 2015, using a process similar to STRATEGY B.

We provided a Theory of Change and a complete Request for Qualifications (RFQ) document to

agencies in Macomb, Oakland, and Wayne counties, inviting them to bid to undertake work as an

evolved ELC. We modeled the entire process on SIF, including a scoring rubric and an external

review panel.

A total of 13 agencies submitted bids. The review process was thorough and included scoring each

bid against a rubric, writing a narrative summary of the bid and the agency's relevant body of work,

and the convening of a volunteer Review Panel that reviewed all these materials and provided

recommendations. A UWSEM staff member with the appropriate background and expertise to enable

him/her to evaluate the bid content scored each rubric section. The scoring was consistent in that the

For Official Use Only

Page 14

Narratives

same person scored the same section(s) of each bid. The Review Panel included a member of

UWSEM's board, a program officer from a major funder of our early childhood work, an external

expert on our region's early childhood system, and a UWSEM Team member with expertise in

financial stability.

The process from release of the RFQ to final determination of agency partners took just over three

months. Based on this experience, we are confident in proposing the following timeline, so that the

selection process for this 2016 SIF will be complete six months following the start of our SIF

intermediary award.

Timeline: September 2016: Intermediary award start date. Draft Request for Qualifications

(RFQ), including Theory of Change, detail on specific interventions to be implemented, target zip

codes, expected capabilities of subrecipient agencies, and scoring rubric. October 2016: Solicit

volunteers for review panel, include review of potential conflicts of interest; panelists are likely to

include representatives from area universities, hospitals, foundations and parents. Release RFQ to

agencies operating in greater Detroit, including not just current and past UWSEM-funded agencies

but all those that may be eligible to apply. November 2016: Letter of Interest deadline with 25-30

LOIs expected. Technical assistance (TA) workshops (in-person) conducted. December 2016:

Subrecipient bids due; 15-20 bids expected. Initial intake and review of bids to ensure completeness

and eligibility. January 2017: Site visits by staff and review panel. Convene review panel for final

funding recommendations. February 2017: Staff review of funding recommendations, including

consideration of potential conflicts of interest. Notify bidders and begin contracting process.

IMPLEMENT PROCESS: In addition to our current SIF initiative, we have demonstrated a

selection process with a high likelihood of identifying high performing subrecipients that meet the

criteria outlined in Appendix 1 through the process used to select grantees for the Evolved ELC project.

The process included details on number of awards, funding levels and associated criteria, range of

implementation, and evaluation stipulations. Bidders, in turn, provided data-informed proposals

demonstrating a capacity to implement the RFQ with fidelity. TA, written and disseminated, on-line

and in-person regarding the process was provided.

UWSEM has extensive experience selecting and awarding competitive grants to non-profits. In the

last year, we awarded $12.2 million Community Impact investment dollars in the tri-county area.

HIGH PERFORMING RECIPIENTS UNDER STRATEGY B: Under Strategy B, we will select

subrecipients who are well suited to implement the predefined intervention. Successful applicants will

demonstrate that the proposed interventions will be an integral component of their agency offerings,

For Official Use Only

Page 15

Narratives

including demonstrating how the goals of the BtoB SIF are but one part of a larger system in which

we all operate. Agencies will also be selected based on their expected ability to implement the evidence-

based interventions with fidelity.

Commitment to families with young children will be demonstrated by an agency's near perfect

alignment with UWSEM's three Early Childhood Outcomes. These outcomes are children are

developmentally ready to learn; parents and caregivers nurture children's development and

communities promote children's well-being. Agencies will demonstrate that the Early Childhood

Outcomes are integral to their body of work as a whole and that they raise awareness and inspire

action on interventions to ensure children age 0-5 are developmentally ready to learn.

To demonstrate a commitment to BtoB SIF, an agency will demonstrate a superior understanding

and alignment of BtoB with the agency's own mission. It will demonstrate an extended and successful

history, knowledge, and experience serving pregnant women and parents of young children. It will

show how this initiative directly supports or is related to the organization's mission or vision. And it

will give clear example(s) of foreseen obstacle(s) and solution(s) and how unforeseen obstacles have

been dealt with in the past.

A successful candidate will demonstrate a community presence by showing a strong track record of

success in recruiting and retaining program participants. Moreover, it will demonstrate a superior

understanding and alignment of recruitment and retention strategy with other agency efforts.

The agency must be known for, and demonstrate, making newcomers feel welcome and

encouraging parents and caregivers to return for additional enrichment. It is identifiable and easily

recognizable as a resource to the community. The agency knows the target population and provides

resources and content to meet their needs and creatively works to ensure services are accessible.

Agency's facilities are known for being easily accessible and it has measures in place to assess

participant safety and comfort while on site.

A subrecipient demonstrates the use of partnerships to deliver deeper results or to reach more

participants. Providing a list of key partners will be required. The agency helps to foster a cultural

change and movement around the importance of early childhood interventions and the importance of

parent engagement and empowerment in the community.

Implementation fidelity and data competence are critical attributes of the successful bidder. The

agency demonstrates a successful history of delivering program components in ways known to

enhance effectiveness. It demonstrates a strong capability for implementation fidelity for the model

component(s) the agency proposes to undertake. There is a process in place for monitoring model

For Official Use Only

Page 16

Narratives

implementation and continuously checking program quality (e.g. staff training on the model,

constant monitoring of staff and programs, etc.). The agency demonstrates strong experience

monitoring program performance. Its strategy includes measures for quality and impact and captures

participant feedback. Agency has accountability measures in place for data quality and timeliness and

has strong experience with different methods of collecting data. The subrecipient demonstrates

significant capacity for data collection and performance management including necessary staff to

manage data collection, technology to manage and store data, and policies/procedures to collect data.

Finally, the agency successfully disseminates knowledge and data to other organizations and in a

manner relevant to the EC Network.

READINESS ASSESSMENT: Specific criteria for assessing applicant readiness will include

considerations relevant to staffing, use of data, community integration and more. As an existing SIF

intermediary, we have extensive experience in providing value-added activities and supports to

subrecipient agencies. We would draw on this experience to ensure that selected subrecipients are

provided with capacity building technical assistance including but not limited to: Finance and

accounting systems, procurement policies, National Service Criminal History Check procedures,

program implementation, data collection, and reporting.

iii. Proposal for Evaluation

CAPACITY TO ENSURE EVALUATION: Our capacity to successfully evaluate our portfolio has

been demonstrated repeatedly. Specifically, we have extensive experience in managing and supporting

evaluations of past funded program models, including our current SIF program. This SIF experience

also demonstrates how UWSEM has supported recipients to improve program performance, to apply

evaluation results to decision making and to successfully complete evaluations of their respective

program models.

We are currently in year five of a current SIF grant in which subrecipients are conducting rigorous

evaluations of programs designed to increase skills and attributes related to school readiness among

children aged 0-5; increase the capacity for parents of at-risk children aged 0-5 to support school

readiness in the home; and increase community level awareness of the need to support early

childhood learning as well as increase formal and informal processes through which communities

support families and children in achieving improved school readiness.

The initial portfolio was comprised of 11 different sub-grantee partners implementing a wide array

of intervention strategies. The sub-grantee selection process was designed to ensure that interventions

For Official Use Only

Page 17

Narratives

ranged from community based primary prevention efforts, to secondary prevention interventions

focused on early childhood initiatives implemented through preschool, head-start, and parenting

groups, to secondary and tertiary prevention efforts focusing on intervening directly with families in

the home.

UWSEM developed a technical support system and review process, which led to the successful

development of evaluation protocols that were able to meet at least moderate levels of evidence by

year 4 of the SIF programming for all subrecipients. These evaluations involve rigorous evaluation

designs including Randomized Comparison Groups at both individual and group levels, Waitlist

Randomized Controls, and Propensity Score Matched Control Designs. In addition, all but two of the

evaluations incorporate multiple post-test repeated measures designs in conjunction with

experimental or quasi-experimental control groups.

UWSEM currently has a portfolio of 5 SIF sub-grantees which have successfully implemented

sophisticated and rigorous evaluations across a 4 year period. In addition, an overall portfolio

evaluation has successfully integrated the individual SIF evaluation data into a comprehensive

overview of SIF impact across the entire portfolio.

UWSEM has implemented a number of strategies for managing and supporting evaluation efforts

at both the portfolio and sub-grantee level. These include: Monthly Evaluator Network meetings,

bimonthly Learning Community meetings integrating SIF program discussions with evaluation

discussions, an open system of continuously available technical support for all aspects of evaluation,

annual SIF summits through which, best practices, lessons learned, and resources are shared across

SIF programs, the development of a unique universal identifier for all SIF participants to facilitate

data management at the local and portfolio level of evaluation and the creation of a data

management system and protocol for all SIF programs at the portfolio level.

We have used the model developed in the SIF program to manage and support evaluation efforts in

other initiatives. The basis for the RFQ in our Evolved ELC initiative rested in part on an evaluation

conducted by JFM Consulting. Those evaluation results, coupled with what we had learned in the

execution of the SIF model, led to a tightening of the logic model/theory of change, a reallocation of

resources, a reduction in number of partner agencies and a reinvigorated collaborative decision-

making structure. The on-going evaluation of the Help Me Grow Initiative, by Oakland University,

has scaled efforts toward proven successful strategies.

In addition to incentivizing agency partners through the RFP/RFA/RFQ process, UWSEM has

continually moved forward in building our capacity and policy of providing technical support and

For Official Use Only

Page 18

Narratives

resources to facilitate program partners to increase their own capacity for and organizational culture

toward rigorous evaluation. For example, UWSEM has invested in an improved data system for its

Early Childhood work. Internal and partner staff have been trained in Social Solutions' Efforts to

Outcomes database to collect and analyze data in order to make efficient resource allocation decisions,

improving program performance.

Ten years ago UWSEM began an organizational cultural and practice shift toward evidence based

decision making and data informed organizational strategy. This process has culminated in a recent

reframing of organizational strategy throughout the entire organization highlighting the importance

of applying evidence and evaluation results to decision making.

UWSEM also has developed an entire unit focused on evaluation and research relating to UWSEM

programs. The organization has committed to continuing to build capacity to use performance and

results data to assess strategies, measure results and engage in ongoing improvement efforts for

UWSEM initiatives and partnerships.

UWSEM has worked extensively with contractors to facilitate capacity building both internal to the

organization and with all of our program partners for the successful implementation and utilization of

rigorous evaluation designs and findings. Contractors include Child Trends, Data Driven Detroit,

Here2There, Ty Partridge Ph.D, and JFM Consulting.

Under our Strategy B plan, we will lead the evaluation effort of the BtoB SIF model and evaluation

design detailed below. A critical part of this plan will be assessing sub-recipient applicants for readiness

and capacity to fully engage and support this effort. Our scoring rubric for evaluating potential

program partners will include their ability to implement evidence-based interventions with fidelity and

to collect, store and use data at a level consistent with an evaluation plan that can achieve moderate

or strong evidence over a three-to-four year time period.

Since we will be leading the single evaluation plan, we envision the technical assistance needed will

be focused on ensuring that partners understand the plan and are able to support the data collection

needed. Our selection process is rigorous enough to make this determination going in. We will draw

on the SIF model of regular Learning Community meetings and the ELC model of providing data

collection TA. Our RFQ will include descriptions of the role of staff. Our experience in the Evolved

ELC RFQ supports this approach.

ASSESSING EVALUATION CAPACITY and TECHNICAL ASSISTANCE: We intend to use the

first six months of the grant period to conduct a comprehensive evaluability assessment to identify,

develop and appraise evaluation documentation and assessment procedures, implementation logistics,

For Official Use Only

Page 19

Narratives

data management protocols, and organizational capacity in terms of staffing, training, technological

infrastructure, and other evaluation related resources.

This is an essential part of the evaluation process as it identifies on the front end, prior to

participant enrollment and data collection, that the evaluation process can be effectively implemented

and has the capacity to yield high quality information that captures program implementation as well

as program outcomes and impact. The evaluability assessment will be comprised of at least 4 core

elements: (1) Assessment of partner agencies, (2) development of a data model, (3) development of a

data collection and management procedure, and (4) specification of specific evaluation questions at all

phases of evaluation (i.e., implementation, outcome, and impact) and analysis strategies which are

isomorphic to each evaluation question.

UWSEM will develop a scoring rubric through which partner agencies will be assessed for their

readiness to participate in a rigorous evaluation process. The rubric will include, but not be limited to:

(1) Agency experience implementing and utilizing high quality evaluations in their work; (2) a

demonstrable organizational culture which embraces evaluation and uses a data driven strategy; (3)

the technical/technological capacity for the collection and management of evaluation data; (4) the

agency's staffing and budgetary capacity to be able to participate in the BtoB SIF evaluation and meet

the expected standards of rigor.

UWSEM will base partnering and sub-grantee decisions in part on the agency's evaluation

readiness score using this rubric. UWSEM will also provide technical assistance to agencies that play a

critical role in program implementation and have adequate evaluation capacity, but would benefit

from increasing their evaluation capabilities.

UWSEM will develop and document a comprehensive data map which will identify each source of

information required to accurately assess program implementation, potential barriers to

implementation, program outcomes, and program impacts. Each information source will be

operationally defined and a quantitative indicator(s) will be identified. All data elements will be

explicitly linked to the logic model and theory of change. Further each data element will include a

description of the data source, responsibility for collection, and the procedures for obtaining,

managing, and accessing the data. In the case of evaluation outcomes and impacts which employ a

psychometric assessment tool, a full description of the measure, reliability and validity information,

and a verification that the measure meets or exceeds acceptable scientific criteria for use (e.g.,

Chronbach's alpha >.7 and demonstrated construct validity). Drawing on the data model,

implementation, outcome and impact evaluation questions will be operationally specified.

For Official Use Only

Page 20

Narratives

As a final step in this evaluability assessment, UWSEM will develop a training protocol to be

delivered to all evaluation project staff at UWSEM and partner agencies. This will be a comprehensive

training focusing on evaluation design, data collection and data management protocols. UWSEM will

also implement "booster" trainings on a regular basis (at least quarterly). The purpose of these booster

trainings will be to obtain feedback regarding effective evaluation related activities, barriers and/or

challenges to the evaluation process, and to ensure that any drift in the fidelity of evaluation

procedures is minimized.

Following the six-month evaluability assessment phase of the project, UWSEM will initiate an

evaluation implementation pilot phase. Evaluation procedures will be implemented on a randomly

selected group of participants with a target sample size of n=40. This will allow for a thorough

assessment of the evaluation process including, data collection, data management, and

methodological protocols prior to implementing the evaluation on a large scale basis. The piloting of

the evaluation process is intended to identify any barriers or challenges to the integrity of the

evaluation and address those issues with minimal impact on the quality of the evaluation findings.

The assessment of the pilot evaluation will involve a complete review of all aspects of the evaluation

and the use of key informant interviews (frontline staff, program leads, program participants, data

collection and management staff, and all other staff involved in the evaluation process). The

procedures developed to conduct this review will be maintained on a smaller scale throughout the

course of the entire evaluation to ensure timely feedback between all key individuals involved in the

evaluation process for continuous quality control.

In addition, special focus will be given to assessing the efficacy of program recruitment and

retention procedures and the adequacy of any propensity and/or case-control matching procedures.

Moreover, an initial data entry auditing process will be implemented in which no less than 30% of the

pilot cases will be double entered and checked for consistency. Any data entry discrepancies will be

compared to the original raw data source and corrected. If there appear to be systemic factors which

made data entry errors more likely the UWSEM evaluation team will adjust the data collection

protocol in order to resolve these issues. If the data entry error rate exceeds 10% a regular auditing of

20% of all evaluation cases will be implemented throughout the evaluation process in order to

continue to ensure the highest quality data accuracy possible without incurring the cost of complete

double entry methods.

Beginning with year 2 of the BtoB SIF project UWSEM will begin to implement the full scale

evaluation. The detailed evaluation plan will be further developed and refined during the first year

For Official Use Only

Page 21

Narratives

evaluability and piloting phase of the project. However, we have outlined a core set of evaluation

requirements which represent the minimum standards for the evaluation plan.

ACHIEVING MODERATE EVIDENCE with STRATEGY B: Because of the nature of the proposed

intervention program the evaluation design will inherently involve a longitudinal design. A

longitudinal design has a number of key strengths for elucidating evaluation outcomes. Specifically,

the use of longitudinal designs will allow for the assessment of intra-individual variation which is a

necessity when the research aim is to uncover the impact of intervention effects on developmental

trajectories. The proposed intervention model is heavily informed by contemporary developmental

science and the key unit of analysis is the process of change within the individual over time. Indeed,

what we are most focused on in this evaluation is the prediction and ability to change the course of

development within individuals and families as opposed to mean scores on a single measure at a given

point in time.

In addition, within the context of the longitudinal design we will employ a baseline assessment

with multiple post-intervention assessments. This design feature controls for a number of threats to

internal validity when comparing baseline and post-intervention differences. It also increases the

statistical power of the evaluation.

The key evaluation research question concerns the degree to which the intervention alters the

developmental trajectory of "at-risk" children and families in comparison to the absence of the

intervention. The longitudinal/repeated measures design will allow for the assessment of trajectories

of change, however, in order to fully address this key research question a comparison group which is

also assessed longitudinally is necessary. Complete random assignment is the gold standard design and

has the greatest degree of efficiency for drawing causal inferences. However, because we are using a

"no wrong door" approach to program enrollment a true random assignment design is impractical

from a logistical standpoint. Moreover, the comprehensive ecological nature of the proposed

intervention further restricts the feasibility of a employing a randomized control group. As such we

propose to use a propensity scoring method from which we can draw a comparable control group

from a sample of children and families served by UWSEM program partners who are not directly

involved in BtoB SIF.

The use of propensity scoring methods has been expanding in program evaluation as it provides an

alternative to randomized control trials, which are often untenable to implement (Guo & Fraser,

2013). Propensity scores are conditional probability estimates of receiving treatment, given a set of

measured covariates. This can be expressed as a binary logistic regression, where P is the probability of

For Official Use Only

Page 22

Narratives

person i receiving treatment T, given that person i has a specific set of values x on the vector of

conditioning or covariate variables X. These covariate variables X are potential confounds related to

self-selecting into treatment, which might be an alternative explanation for mean differences in the

outcome for those receiving treatment and those in a control group.

Using this approach, an individual propensity score can be calculated in the form of a logit, or log

odds, estimating the likelihood of receiving treatment--given the values of the covariates for that

person. Once these logits are calculated, individuals in the treatment group and the control group are

matched based on their propensity scores, or their individual logit values. Since these values are

usually not exactly the same, a method is needed to determine how close a control propensity score

needs to be to a treatment propensity score in order to be considered a match.

There are a number of methods for estimating matches, two of the most common being nearest

neighbor-matching and calculating Mahalanobis d (i.e. distance). Mahalanobis d is commonly used in

multivariate statistics to estimate the average difference in scores on a set of variables across

participants. The method has been adapted to propensity score-matching as well.

We will use the Mahalanobis d method because of the potentially large number of variables across

which we will want to match intervention and control participants. Propensity scores will be

calculated based on the following covariates: family income, age of parents, age of participating child,

number of other support programs in which the family participates, parent education, primary

language used in the home, ethnicity, and other variables identified during the evaluability and pilot

phases.

In addition we will take advantage of two features of the proposed intervention strategy which will

likely create natural control groups. The intervention model targets families during the prenatal and

neonatal periods and also incorporates a routine check-up intervention in which families are regularly

contacted and can be assessed. We anticipate that not all potential program participants will utilize the

program during the initial contact or even follow-up contacts. However, we intend to assess those

families to the extent possible thus providing a quasi-waitlist control group.

Baseline equivalence: As a result of the no wrong door policy for program entry a randomized

control group cannot be feasibly implemented. Further, there will be multiple "baseline" conditions

depending upon the manner and timing in which a family enters the program. As such, we will

rigorously assess families in the intervention and in the selected control groups to determine

equivalence at baseline entry points. All key measures will be compared between the two groups using

an independent sample t-test using an a priori alpha significance criterion of alpha=.10 in order to

For Official Use Only

Page 23

http:alpha=.10

Narratives

obtain a conservative estimate in the context of a large sample size. Any variables on which the

intervention and control groups are found to differ at the p=.10 level will be included as baseline

controls in all subsequent evaluation analyses comparing the two groups.

External validity: One of the central aims of the proposed BtoB SIF initiative is to bring the

intervention to scale to at least a regional scale. It is important therefore to include an explicit

assessment of the external validity of the findings. UWSEM will develop a regional demographic

model using a variety of data sets obtained through 2010 census data, Data Driven Detroit, and Kurt

Metzger and Associates. Using this demographic model an assessment tool will be developed to use

with all program participants. This measurement tool will allow for the direct comparison of the

intervention sample to the larger regional population.

Organizational Capability

Sample Size, Statistical Power, and Effect Size Considerations: Through the current SIF project

UWSEM has served, through our SIF sub-grantees, 19,000 children and families. Additionally,

through the Help Me Grow project, UWSEM has gathered data on over 40,000 children aged 0-3 in

the Metro-area and approximately 95,000 early childhood assessments using the Ages and Stages

Questionnaire. Since inception, UWSEM has provided services to approximately 29,000 adults,

through our ELC networks. As such, UWSEM has developed the capacity to recruit and assess

relatively large samples drawn from the same "at-risk" population which we are aiming to serve in the

current proposal.

Based on our working history with the currently operating components of the proposed

intervention we anticipate enrolling approximately 1,200 youth and families drawn from

approximately 12,000 screened families as well as recruiting a control group of a minimum of 800

families across the four year intervention period. In addition we intend to get a minimum of three

assessments of all outcome measures for each family. Assuming an 80% retention rate across the

three assessment periods we will have a base observation sample of 2,880 observations. This will yield

more than adequate statistical power for all proposed statistical models. However, for each proposed

analysis a detailed power analysis will be conducted. For traditional general linear model analyses

(e.g., ANOVA, MANOVA, regression analyses) statistical power will be assessed using G*power v. 3.1.

For more advanced statistical modeling (e.g., Latent Growth Models, Structural Equation Models,

Latent Class Models, etc.) statistical power will be estimated using the Monte Carlo feature of Mplus v.

7.1. This latter method allows for the specification of the statistical model to be tested with minimally

acceptable parameter estimates. These fully specified models are then simulated using Monte Carlo

For Official Use Only

Page 24

Narratives

methods across 1,000 simulated samples. The proportion of samples with which the path coefficients

are determined to be statistically significant at the p=.05 level are generated and if the proportion is

greater than 80% with a specified sample size, then it is determined the model has adequate statistical

power.

In addition, for all structural equation based models we will calculate power at the model level

using the root mean square error of approximation (RMSEA) fit index as the effect estimate

epsilonfor; epsilon=0 (exact fit), epsilon less than or equal to .05 (close fit), and epsilon > .05 (poor

fit). This approach estimates the likelihood of the value of epsilon; under the null and alternative

hypotheses for a given sample size N, significance level alpha; and model degrees of freedom (ratio of

observed variables to estimated parameters).

All statistical power estimates will assume alpha=.05, power =.80, and a moderate effect size of d =

.35. Generally a Cohen's d of d = .2 is considered a small effect size and d = .5 a medium effect size.

Effect is of particular importance when testing statistical models in large samples because significance

levels are exponentially proportional to sample size such that as sample sizes increase, smaller effects

are determined to be statistically significant. By explicitly reporting the effect size, the results can be

interpreted in terms of meaningfulness as opposed to statistical significance. This is particularly

relevant for applied evaluation research which has the aim of informing policy and practice. In this

context reporting very small effects to be statistically significant resulting from a large sample size will

likely result in the inefficient use of money and resources to produce a very small impact. As such, we

will not interpret findings with an effect size smaller than d=.35 which is considered to be in between a

small and medium effect.

Missing Data: All evaluation analyses will use an intent-to-treat analysis approach and incorporate

all participants in both intervention and control groups irrespective of program adherence and /or

missing assessments. Intent-to-treat analyses correct for the potential overestimation of treatment or

intervention effects which result from removing non-adhering participants from the sample.

In addition, prior to analysis all data will be assessed for patterns of missingness in order to

determine the likelihood that missing data are; (1) Missing Completely At Random (MCAR), (2)

Missing At Random (MAR); or (3) Missing Not At Random (MNAR). If at least MAR a combination

of multiple imputation techniques and the use of Full-Information Maximum Likelihood estimates

will be employed depending on the type of analysis being conducted. Under conditions of MNAR,

patterns of missingness will be included in all statistical models as a control.

Analysis strategies - Implementation and Program Fidelity Analyses: The complete set of

For Official Use Only

Page 25

http:alpha=.05

Narratives

evaluation research questions will be specified and refined during the evaluability and pilot phases of

the project. However, there is a core set of general evaluation questions which will be required as part

of the final evaluation.

All key program components will be tracked and evaluated to assess the degree to which the fidelity

of the intervention model was upheld and the degree to which the intervention was fully implemented

across participants. Fidelity and implementation questions will include: adequacy of training for

intervention staff, degree of adherence to intervention protocols, number of children and families

served, types of services provided, and number of referrals to partner agencies.

In addition, program participation variables will be assessed and incorporated as moderators in all

outcome and impact analyses. Because UWSEM is adopting a "no wrong door" program engagement

policy it will be important to identify participants characterized as early adopters (i.e., prenatal,

neonatal, early infancy), middle adopters (late infancy thru toddlerhood), and late adopters (preschool

-- K). Evaluation analyses will utilize discriminant function analysis to identify clusters of baseline and

demographic variables which predict adopter status (i.e., early, middle, late). Adopter status will also

be included as a moderating variable for analyses of program outcomes and impacts.

Similarly, Cox regression hazard models will be conducted to predict the likelihood of intervention

duration or the length of time a participant stays engaged with the intervention. Baseline and

demographic variables will also be included as predictors of intervention duration. Hazard models are

required for these types of analyses because incorporating a unit of time as an outcome variable

violates several core assumptions of inferential statistics (David 1972, Dickman, Sloggett et al. 2004).

Time is a monotonically increasing variable and thus cannot be treated as a random variable with a

normal distribution and independence of observation. Additionally, there will be individuals for whom

intervention participation does not end before the end of evaluation data collection. This creates an

inestimable probability distribution referred to as right data censoring which severely biases traditional

statistical tests. Survival or hazard models are explicitly designed to address these two features of

"time-to-event" data such as intervention duration.

The level of intervention engagement will also be assessed. Intervention engagement will be

operationally defined as: (1) the ratio of number of intervention services for which a participant is

eligible to the number of intervention services received; (2) the number of intervention components a

participant utilizes; and (3) a self-report index of the degree to which intervention participation is felt

to be an important part of the participant's life. A composite engagement variable will be created from

these three indice (Whether this is a latent construct or a weighted linear sum will be determined

For Official Use Only

Page 26

Narratives

during the pilot phase of the evaluation) and used in all outcome and impact analyses as either a

predictor or as a moderator.

Full Intervention Model v. Intervention Components: The proposed intervention model (see

LM/TOC) is a multifaceted intervention with multiple intervention components. It is anticipated that

some participants will engaged in some intervention components but not others. Indeed, the adoption

of a "no wrong door" intervention access policy ensures differential intervention utilization patterns to

some degree. To the extent possible, the evaluation strategy will attempt to capitalize on this aspect of

the intervention and evaluate intervention outcomes for both the entire intervention model as a whole

as well as individual components of the intervention as quasi-standalone interventions. This will allow

for the comparative efficacy of the integrated intervention model relative to individual intervention

components.

Outcome and Impact Analyses: As discussed above, the specification of specific evaluation research

questions, especially with respect to impact and outcome analyses is premature at this point and will

be fully developed during the evaluability and pilot phases of the evaluation. However, as with the

implementation and fidelity evaluation there are several analytic strategies which will be necessary

requirements for the final evaluation plan.

Multilevel Models: The data collected as a part of this evaluation plan will inherently be nested or

non-independent on a number of levels of analysis. Because of the longitudinal design the repeated

observations over time within each participant will by definition be non-independent. Additionally,

because components of the intervention will be delivered in a group context it is assumed that

participants within a given group will have non-independent data. There are also potential non-

independence effects due to factors such as neighborhood, classroom, hospital setting, etc. In cases

where data could potentially have a nested or non-independent structure intra-class correlations

(ICC) will be estimated for all salient variables. The ICC is an index of non-independence of

observation ranging from 0 (independent) to 1 (complete dependence). In cases where the ICC is

statistically significant, indicating non-independence all analyses will be conducted using a multilevel

modeling framework. Where level 1 represents the relationship between the independent and

dependent variable and level 2 disaggregates the independent relationship of the independent and

dependent variable and the variance due to non-independence of observation such as time nested

within person or person nested within group. These models yield unbiased estimates of relationships

between independent and dependent variables, as well as allowing for the explicit estimation of group

level predictors. This multilevel framework has also been extended beyond the regression based

For Official Use Only

Page 27

Narratives

framework to latent variable model frameworks as well.

Latent variable structural equation modeling (LVSEM): One of the key evaluation questions is to

test an ecological model linking social determinants of family risk to child outcomes through a

number of mediating child and family process variables. Social determinants will include; relative

poverty as measured by the Arizona Self-Sufficiency Matrix (ASSM), barriers to health care access,

and cumulative family risks. Child outcomes will include assessments of the five core areas of child

health and well-being: (1) language and communication; (2) gross and fine motor skills; (3) general

cognitive skills; (4) socioemotional well-being; and (5) physical health and nutrition. A

comprehensive set of mediating variables which are directly linked to the intervention targets in the

proposed intervention model will be assessed as well. The selection of these variables will be drawn

from an extensive review of the infant and early-childhood risk and resilience literature and will

include factors such as: Caregiver health and emotional well-being, caregiver social support,

knowledge and use of positive parenting practices, parent-child relationship quality, access to and

utilization of high quality childcare, home supports for child development. Further, important

moderating factors including; caregiver race/ethnicity, community risk factors, and the economic,

health, and family policy contexts will also be considered.

In order to fully evaluate these process models latent variable structural equation models will be

employed (MacCallum and Austin 2000, Tomarken and Waller 2005, Weston and Gore 2006, Little,

Card et al. 2007). These models have a number of advantages over regression based models. First,

they allow for more precise measurement models of key constructs by estimating latent variables

across multiple observed variables which are considered to measure the same underlying construct.

These latent variables can be represented in matrix form as a series of weighted linear sums generally

referred to as a measurement model. The second key advantage of LVSEM is that it allows for the

specification of complex relationships between latent constructs through simultaneous estimation of

structural (i.e., regression) equations. This structural part of the model can also be represented in

matrix form. Further, the path coefficients in the model can be decomposed into direct and indirect

effects to provide formal tests of mediation processes.

In order to test categorical moderating factors at the statistical model level, multi-group LVSEM

will be used in which a particular SEM will be specified for each level of the moderating variable (e.g.,

ethnicity) and all path coefficients will be constrained to be equal across moderator levels. Overall

model fit in this constrained model will be assessed and a poor fitting model will suggest moderation

effects. Subsequently, the path coefficient constraints will be removed across levels of the moderator

For Official Use Only

Page 28

Narratives

allowing each level to produce different path coefficient estimates which can then be compared to

better interpret the distinct statistical processes for each level of the moderator.

Mixture model latent growth modeling: Another key evaluation question will be to determine the

process of change in a number of key index variables including relative poverty and child outcomes.

Of particular interest is the assessment of trajectories of change in these variables over time as

opposed to a single point-in-time assessment. By using latent growth modeling (LGM), which is a

special case of LVSEM, intra-individual variance or growth trajectories can be estimated for each

participant and predictors of variability (e.g., treatment v. control) can be incorporated (Muthén and

Muthén 2000, Li, Duncan et al. 2001, Duncan and Duncan 2004). These models are an integration of

the multilevel models and LVSEM approaches described above in which the latent variables are

comprised of two weighted linear sums of individual scores on a given measure over time referred to

as the latent intercept and the latent slope. The means of the intercept and slope are the sample

average for a measure at a given point in time(Intercept) and the average rate of change in that

measure over time (slope). The variance of these two latent variables provides an index of inter-

individual variance in intra-individual variability over time (Nesselroade & Ram, 2004; Willett,

2004). As such, change in a variable over time is explicitly incorporated and can be both predicted by

and predictive of other salient variables in the model.

Mixture models or latent class models are a special case of latent growth models in which latent

classes C are estimated from the variance of the latent intercept and slope and represent distinct

statistical populations based on the overall level and rate of change in the outcome variables over time

(Muthén and Muthén 2000). So for example, a class may be comprised of individuals that have

relatively high initial levels of the outcome variable and show rapid reductions over time whereas

another class may be comprised of relatively high initial levels but show little to no change in outcome

over time. These latent classes can be used to identify participants that could be categorized as "high

responders", "moderate responders" and "low responders" to the intervention. These latent classes of

"responder" can then be predicted by baseline, demographic or intervention engagement/utilization

variables thus informing optimal target populations or the need to adjust the intervention for some

groups of participants based on the response likelihoods.

STRATEGY B -- CURRENT AND POTENTIAL LEVELS OF EVIDENCE:

Family Check-Up Model: The family checkup model is a motivational interviewing base