Parent Support Circles Los Angeles County

Preliminary Evaluation Report

Research & Evaluation Center

Children’s Institute, Inc.

April 30, 2016

Table of Contents

I. Program Summary 2

II. Program and Evaluation Background 3

III. Evaluation Goals and Purpose 4

IV. Evaluation Methods 5

V. Evaluation Questions and Preliminary Results 6

Chart A: Protective Factors Survey Results 8

Chart B: Satisfaction Survey Results 10

Chart C: Program Data Collection and Demographics Report 13

VI. Next Steps: Final Evaluation Reports 15

2

I. Program Summary

Parent Support Circles of Los Angeles County (PSCLAC) is an innovative program to create clusters of parents of young children around the county who come together to share and learn. Funded by First 5 LA as part of their strategic planning efforts to help communities enhance protective factors, the program’s goals are to increase social connections, knowledge of parenting, and knowledge of resources for participants. These voluntary, parent-led groups are based on the principle that people feel most supported in the company of others who share their experiences. Children’s Institute, Inc. (CII) developed the program and oversees the work of the five community-based organizations (CBOs) who provide the support groups across LA County.

Since the support groups are voluntary and designed for sharing, there are no specific curricula; instead, the groups are organized around topics of interest to communities. Based on interviews with parents around LA County, CII developed topic guides for the most requested subject matter. For example, self-care, nurturing infants and toddlers, children with special needs, and behavior challenges were themes that arose throughout the county. Each of the nine topics has guides available in English and Spanish that include information on various aspects of the topic, resources for further information, red flags, and tip sheets for distribution to participants.

At the center of the model are the facilitators who are recruited by the CBOs. The CBOs look for “natural helpers” with lived experience as parents in communities of need. Facilitators attend an intensive 3-day training in the approach. Importantly, they also participate in regular reflective supervision sessions in which they can air their concerns and share their learnings.

CBOs work with community partners to develop support groups and choose topics. The groups run for 8 to 12 weeks and include a minimum of five participants. The group facilitators and participants determine the frequency of the meetings and the length of the sessions, although most meet weekly for an hour to an hour and a half. Child care and light refreshments are provided.

3

II. Program and Evaluation Background

In November 2010, the First 5 LA Commission approved funding for the Parent Support Circles Project, for a Lead Agency to work with five community-based organizations to establish 100 peer support groups for parents of children from the prenatal stage to age 5 in Los Angeles County communities where support, social cohesion, and connection to neighborhood peers are weak or lacking. The project aims to increase participants’ social connectedness; increase participants’ knowledge, confidence, and efficacy in parenting skills; and raise participants’ awareness and use of resources available to support them. In September 2011, the First 5 LA Commission approved additional funding for a Curriculum Developer/Training and Technical Assistance (TA) Provider, responsible for enhancing and modifying the implementation of the project by including the development of core competencies, training, and technical assistance. In 2013, Children’s Institute, Inc. was selected as both the Lead Agency and Curriculum Developer/Training and TA Provider to implement the Parent Support Circles Project. Program evaluation responsibilities were added to CII’s contract in October 2014; evaluation protocols were implemented in January 2015 across all peer support groups and have been supported by CII’s Research and Evaluation Center (REC). Guided by evaluation consultant Dr. Bruce L. Baker, REC also manages data processing, analysis, and reporting.

4

III. Evaluation Goals and Purpose

As described in the Evaluation Plan, the evaluation data are intended to guide program improvement and assess program impact on participants and facilitators; results are therefore shared by REC on a regular basis with CII management, CBO coordinators, group facilitators, First 5 LA collaborators, and other stakeholders.

In order to support a data-informed program, participant satisfaction data collected at the midpoint session of groups have informed program improvement and led to programmatic changes for subsequent group sessions. Midpoint satisfaction data reports for individual groups are generated and sent to CBO coordinators within 7 days of the satisfaction surveys being submitted to REC; CBO coordinators can then review these results with each group’s facilitators and make program improvements as needed. Examples of adjustments have included extending the duration of a group beyond the original 8 weeks, extending the length of group sessions, and improving facilitator communication skills and knowledge of resources.

Additionally, process data are tracked and reported on a monthly basis in order to support the CBOs with participant recruitment and data collection goals. At monthly CBO meetings since January 2015, REC shares program-level reports with number of participants, number of completed and active groups, number of referrals, data collection totals and compliance, and participant demographics. The evaluation discussion at the monthly meetings has allowed staff to jointly address programming and evaluation challenges; for example, discussion of the ethnic breakdown among participants (predominantly Latino or Hispanic) reinforced CBO commitment to cultural sensitivity and inclusiveness in their recruitment efforts. CBO and REC staff also collaborated on finalizing the original satisfaction survey’s wording (English and Spanish) in order to ensure participant comprehension.

In order to assess the cumulative effects of the program and inform program improvement, pre- and post-data from the protective factors survey are analyzed and shared with CBOs and other stakeholders. Protective factors results in the first Preliminary Evaluation Report (October 2015) showed increases in Social Support and Knowledge of Parenting factors but a decrease in Concrete Support factors; these results helped to inform program improvement and reinforce the need to increase participant knowledge of community resources. Qualitative data collected via facilitator interviews will supplement participant outcome data in demonstrating program impact. Content analysis of facilitator interviews is in progress and will be included in the final PSCLAC Evaluation Report.

5

IV. Evaluation Methods

Data collection protocols with the following outcome measures were implemented across all support groups, beginning in January 2015:

i. Participant Protective Factors Survey1: The 15-item, self-report Protective Factors Survey (PFS) consists of four subscales: Social Support, Concrete Support, Nurturing and Attachment, and Knowledge of Parenting. At intake and the last session of the group, the PFS is administered to participants and collected by the facilitator or another CBO appointee, who may assist the participants with reading comprehension as necessary. Completed surveys are submitted to the CBO coordinators, who send them electronically to REC. REC staff enter participant responses into an internal data sheet; numerical identifiers are assigned to ensure participant confidentiality and track individual outcomes across all participant measures.

ii. Participant Satisfaction Survey2: The 15-item, self-report Satisfaction Survey asks participants to rate group implementation, participation from group members, and group facilitators. The survey is administered at the midpoint session and the last session of each group. A CBO appointee (not the facilitator) may assist participants as necessary. The CBO appointee collects the completed surveys and sends them electronically to REC. REC staff enter participant responses in an internal data sheet; numerical identifiers are assigned to ensure participant confidentiality and track individual outcomes across all participant measures.

iii. Facilitator Interviews: Based on a review of the literature, REC developed an interview protocol focusing on the major areas of impact on the facilitators from running groups; domains include personal enjoyment, professional development, and relationships with group members. As a sample, 12 facilitators were selected across several criteria (e.g., language, CBO affiliation, experience with facilitating at least two groups) and were interviewed in either English or Spanish based on their preference. Interviews have been translated, transcribed, and coded (via Dedoose coding software) in preparation for content analysis.

CBO and REC staff enter process data (e.g., service delivery totals, data collection totals, and participant demographic information) into an online group tracking sheet, an online platform for participant registration, and REC’s internal data sheets.

1 Developed by the FRIENDS National Resource Center for Community-Based Child Abuse Prevention, in partnership with the Center for the Study of Social Policy. 2 Developed by CII’s Research and Evaluation Center and CBOs.

6

V. Evaluation Questions and Preliminary Results

Participant Outcomes

Results from the Protective Factors Survey (PFS) address the following evaluation questions:

• Does participants’ social connectedness increase? Is there statistically significantimprovement between pre- and post-ratings for the Social Support items of the PFS?

• Does participants’ knowledge of parenting increase? Is there statistically significantimprovement between pre- and post-ratings for the Nurturing and Attachment orKnowledge of Parenting items of the PFS?

• Does participants’ knowledge of community resources increase? Is there statisticallysignificant improvement between pre- and post-ratings for the Concrete Supportitems of the PFS?

Based on data collected between January 2015 and April 2016, preliminary results from the PFS (see Chart A, p. 8) are very encouraging.

The PFS is organized into three composite scores: Social Support (three items), Concrete Support (three items), and Nurturing and Attachment (four items). In addition, there are five Knowledge of Parenting items that are considered separately. Items present a statement (e.g., “When I am lonely there are several people I can talk to”) and ask for a response on the following scale: 1 (Strongly disagree), 2 (Mostly disagree), 3 (Slightly disagree), 4 (Neutral), 5 (Slightly agree), 6 (Mostly agree), or 7 (Strongly agree). Each subscale score ranges from 1 to 7and is calculated from the mean of the subscale items.

At the pre assessment, many participants scored high on the composite scales, which would be expected, as PSCLAC is not a clinical program and participants were not referred in for existing problems. Despite this, there were meaningful and statistically significant pre- to post-group gains in Social Support, Nurturing and Attachment, and Knowledge of Parenting factors.

Social Support is the domain most relevant to the PSCLAC group experience, and the gains here are gratifying. For all participants, the overall gains in the means of Social Support items (from 5.2 to 5.8) were highly statistically significant; t(180) = 4.20, p < .001. (A p-value below .001 indicates that something could happen by chance less than one time in a thousand.) The Social Support scores were examined another way; the change in means was considered for just those participants who initially expressed lower available support and for whom the groups could be of particular benefit. At pre, 36% of participants scored below the cutoff score of 5 (on average, below “Slightly agree”); at post, only 17% scored below 5. For these participants who entered low in expressed social support, the gain following participation in the group (from 3.4 to 5.3) was highly significant; t(65) = 7.78, p < .001.

7

In the Nurturing and Attachment domain, the findings are also encouraging. For all participants there was a statistically significant gain from 6.1 to 6.4, despite relatively high scores to begin with and little room for change; t(172) = 2.66, p = .009. For those participants who initially scored below 5 (Slightly agree), there was a sizeable gain from 4.0 to 5.2; t(24) = 4.38, p < .001.

The five items related to Knowledge of Parenting are not to be combined into a composite score but should be considered separately. On the two positive items (“I know how to help my child learn” and “I praise my child when he/she behaves well”), there were statistically significant increases when considering all participants (p < .001 and p = .042, respectively). The percentage of participants indicating “Mostly agree” or “Strongly agree” for these two items increased from 49% to 67% and from 66% to 75%, respectively. On the three negative items (“There are times when I don’t know what to do as a parent,” “My child misbehaves just to upset me,” and “When I discipline my child I lose control”), participants as a whole did not change significantly. However, for the subgroup of participants who at least somewhat agreed with these negative items at the pre assessment, there were major decreases on all three items from the agree range to the disagree range (all with p < .001). Thus the support groups considerably affected Knowledge of Parenting factors for those participants who were most in need.

In the Concrete Support domain, mean scores for all participants decreased slightly from 4.3 to 4.2 but not at a significant level; statistically speaking there was no change between pre and post assessments. The analysis for first Preliminary Evaluation Report showed a larger decrease between pre and post (from 4.6 to 4.1) for the Concrete Support items, indicating a potential need for the support groups to make programming adjustments and focus more on sharing community resources with participants. A final analysis of Concrete Support data will be conducted in June 2016 after all groups have finished and all PFS data has been collected.

Chart A. Protective Factors Survey Results

5.2

4.3

6.1

4.0

5.1 5.3

5.9 5.6

5.8

4.2

6.4

4.3

5.7

5.2

6.2

5.7

Social Support Concrete Support Nurturing &

Attachment

Knowledge ofParenting -

Know what to doas parent

Knowledge ofParenting -

Help child learn

Knowledge ofParenting -

Child misbehavesto upset me

Knowledge ofParenting -

Praise child forgood behavior

Knowledge ofParenting -

Lose control whendisciplining child

PRE POST Matched Pairs = 181

8

9

Results from the Participant Satisfaction Survey address the following evaluation questions:

• What is the participants’ level of satisfaction with the groups and the facilitators?• Does the participants’ level of satisfaction improve between midpoint and post-

ratings on the satisfaction survey?

The satisfaction survey asks participants to express their agreement or disagreement with 15 statements across three categories: “About the Group” (e.g., number of meetings, length of group sessions), “About Your Participation” (e.g., information was valuable, would recommend group to others), and “About the Facilitators” (e.g., understood participants’ issues, knew information well). Response options for the first category are: 1 (Too small/short or not enough), 2 (Just right), and 3 (Too big/long/many/much). Response options for the second and third categories are: 1 (Definitely disagree), 2 (Disagree), 3 (Agree), and 4 (Definitely agree).

As shown below (see Charts B1 and B2, p. 10-11), participants’ ratings at both midpoint and post assessment have been almost universally positive. Impressively, even with relatively high satisfaction levels to start with, there was statistically significant improvement in participation satisfaction between the midpoint and end of groups (from 44.3 to 45.1 out of a possible 50); t(172) = 2.03, p = .044.

Participants completing the satisfaction survey at the last group session overwhelmingly endorsed the overall experience: 94% of participants agreed that they had changed their parenting as a result of the group, 98% of participants agreed that the facilitators cared about their issues and made it possible for them to talk about difficult topics, and 98% of participants agreed that they would recommend their group to others.

On items about the group implementation (ratings 1-3), most participants responded with “Just right” to every statement. Exceptions were generally due to wanting more of the group experience, with 28% of participants indicating that there were not enough sessions and 19% indicating that sessions were too short.

On items about the participation experience and the facilitators (ratings 1-4), over 94% of participants agreed or strongly agreed with every statement, indicating high overall satisfaction across various support groups, group topics, and facilitators.

3.49 3.53 3.58

3.39

3.66 3.6 3.58

3.66

3.47

3.74

Respected by othermembers

Had a chance to speak Information wasvaluable

Changed my parenting Would recommendthis group

1.93 1.89 1.86

1.99

1.88 1.91 1.83

1.72

2.03

1.86

Group size Length of meetings Number of meetings Information presented Discussion time

About the Group MID POSTHighest Rating = 2

3.55

3.62

3.55 3.58

3.56

3.65 3.69

3.62

3.68 3.66

Cared about our issues Made memberscomfortable

Possible to talk aboutdifficult topics

Made sure everyonehad chance to speak

Knew how to findresources

About the Facilitators Highest Rating = 4

Chart B1. Satisfaction Survey Results (by Means)

10

Matched Pairs = 173

About Participation Highest Rating = 4

CBO All Groups 57 290

Collection Post Dates 1/1/15

Participants

to 4/1/16 Earliest Latest

12

17

29

3

16

87

83

71

91

82

2

6

2

Group size

Length of meetings

Number of meetings

Information presented

Discussion time

About the Group

Too Small/Short or Not Enough Just Right Too Big/Long/Many/Much

0

1

1

1

1

5

28

31

29

42

21

71

68

71

52

78

Respected by other members

Had a chance to speak

Information was valuable

Changed my parenting

Would recommend this group

About Participation

Definitely Disagree Disagree Agree Definitely Agree

2

1

2

1

2

29

26

32

26

30

69

73

66

73

69

Cared about our issues

Made members comfortable

Possible to talk about difficulttopics

Everyone had a chance to speak

Knew how to find resources

About the Facilitators

Definitely Disagree Disagree Agree Definitely Agree

Chart B2. Satisfaction Survey Results at Post (by Percentage)

11

12

Facilitator Outcomes

Results from the facilitator interviews will address the following evaluation question:

• What are the facilitators’ perceived benefits and effects of facilitating groups (e.g., increasing personally-relevant knowledge, gaining social approval and social support, developing interpersonal competence, achieving professional growth, and acting as a community resource for others)?

Content from 12 facilitator interviews has been transcribed, translated as needed, and coded across various domains including personal growth, social support, and professional growth. Feedback from facilitators appears to be generally positive with some suggestions for program improvement; further content analysis will be included in final evaluation report.

Sample excerpts from interview transcripts:

• “I’ve loved being able to participate in it. I’ve learned a lot.” • “You start gaining confidence and that confidence makes you get more

motivated, and it makes you get more involved with the community.” • “I didn’t think I had the confidence to be up there and to speak and to guide and

to listen, which is one of the key skills I think I learned through the facilitating.”

Process Data

Based on the April 2016 Program Data Collection Report and Participant Demographics Report (updated monthly since January 2015):

i. Service delivery (see Charts C1 and C2, p. 13-14)

• Number of groups implemented: 71 completed, 13 active • Number of group attendees (based on CBO estimates): 961 • Number of referrals: 332 • Number of participants with measures: 793 • Number of matched pairs for measures: 205 for protective factors survey, 196

for satisfaction survey

ii. Demographic information (see Chart C2, p. 14)

• Gender: 88% Female, 12% Male • Age: Highest proportion (39%) of participants are in 30-39 range • Ethnicity: Highest proportion (82%) of participants are Latino or Hispanic

Individual CBO reports have been generated periodically to assist with boosting data collection compliance and share CBO-specific demographics and outcomes.

793

645

324 342

TOTAL CLIENTSW/MEASURES

PRE MID POST

Data Collection Totals

196

205

Satisfaction SurveyMID & POST

Protective Factors Survey PRE & POST

Matched Pairs

Program Data Collection Progress Report

January 2015 - April 2016

COMPLETED GROUPS (2015-2016)

71 ACTIVE GROUPS (2015-2016)

13 CBO-ESTIMATED ATTENDEES (2015-2016)

961 COMPLIANCE RATE FOR MEAS. COMPLETION

82%

Chart C1. Program Data Collection Report

13 13

13

Participant Demographics

7, 0.7%

155, 16.4%

40, 4.2%

149, 15.7%

379, 40.0%

16, 1.7%

Widowed

Single

Separated

Partnered

Married

Divorced

Marital Status

113 12%

835 88%

Male Female

January 2015 - April 2016

26 2.7%

217 23.0%

372 39.2%

172 18.1%

39 4.1% 20

2.1%

15-19 40-49 50-59 60-69

Age (Years)

21, 2.2%

45, 4.7%

4, 0.4%

3, 0.3%

774, 81.6%

3, 0.3%

57, 6.0%

23, 2.4%

Other

Non-Hispanic White/Euro-American

Native American/Alaskan Native

Middle Eastern/Arab American

Latino/Hispanic American

Hawaiian/Pacific Islander

East Asian/Asian American

Black/Afro-Caribbean/African American

Ethnicity

REGISTRANTS ENTERED IN WEBSITE

948

INVOLVED WITH BEST START

56 INVOLVED WITH NEIGHBORHOOD ACTION COUNCIL

19 REFERRED BY HOSPITAL/CLINIC (WELCOME BABY)

58 REFERRED BY FRC/ COMMUNITY AGENCY

274

Chart C2. Participant Demographics Report

Gender

20-29 30-39

13 14

14

15

VI. Next Steps: Final Evaluation Reports

In the last few months of PSCLAC’s programming, REC will continue to track and report outcome and process data. The final Data Collection Reports and Participant Demographics Reports (program-level and CBO-level) will be shared in May 2016 at the last CBO meeting. Midpoint satisfaction reports will continue to be shared with individual groups and CBOs as needed.

For the Final Evaluation Report due July 2016, REC will conduct a final analysis of all outcome data and matched pairs collected. With the largest possible sample of participant data, REC will examine additional variables (such as attendance) that could be associated with gains from the PSCLAC support groups. Attendance data is currently being processed and aggregated by evaluation consultant Bill Monro; preliminary results have already been shared at recent CBO meetings. As previously mentioned, REC will also conduct further content analysis of facilitator interviews across multiple domains in order to address evaluation questions about benefits to facilitating PSCLAC support groups. Final evaluation results will be shared with group facilitators, CBO coordinators, CII management, First 5 LA collaborators, and other various stakeholders.