2014 americorps state and national symposium research and evaluation: grantee panel
TRANSCRIPT
2014 AmeriCorpsState and National Symposium
Research and Evaluation: Grantee Panel
Objectives
• Share examples of how programs are building evidence at various stages of the evidence continuum
• Highlight diverse grantees and subgrantees– Small and large– Focus area representation– Multi-site/multi-issue programs– Rural
-Gather evidence supporting the intervention-Design/Adopt a strong program-Develop a Logic Model-Create Implementation Materials-Pilot implementation
-Document program process(es)-Ensure fidelity in implementation-Evaluate program’s quality and efficiency-Establish continuous process improvement protocols [Performance Measures - Outputs]
-Develop indicators for measuring outcomes-Conduct pre-/post- intervention evaluation to measure outcomes-Conduct process evaluation[Performance Measures - Outcomes]
-Examine linkage between program activities and outcomes -Perform multiple pre- and post-evaluations (time series design)-Conduct independent (unbiased) outcome evaluation(s) -Conduct meta-analysis of various studies
-Establish causal linkage between program activities and intended outcomes/impact (e.g. Conduct quasi-experimental evaluation using a comparison group , evaluation with random assignment (RCT), regression analysis, or other appropriate study design) -Conduct Multiple independent evaluations using strong study designs-Measure cost effectiveness compared to other interventions addressing same need
Identify a strong program design
Attain strong evidence of positive program
outcomes
Assess program’s outcomes
Ensure effective implementation
Obtain evidence of positive program
outcomes
Building Evidence of Effectiveness
Evidence Informed
Evidence Based
Panelists
• Fa’iz Rab, AmeriCorps Program Manager, Lutheran Family Services
• Julie McClure, Director, Napa County Office of Education Community Programs
• Amy Foss, Vice President, Conservation Legacy
• Karen Domerski, General Counsel, Jumpstart
Nebraska Economic Opportunity Corps(AmeriCorps State and National)
Support | Educate | Empower
The Nebraska Economic Opportunity Corps is an AmeriCorps State program of Lutheran Family Services of Nebraska, Inc. Our mission is to improve the economic opportunities of at-risk youth, new Americans, and veterans through education and employment programs that empower individuals towards self-sufficiency.
NEOC Performance MeasuresSelf-Sufficiency Matrix
Outcomes are measured through the use of an adapted version of the Snohomish County Matrix developed by the Snohomish County Self-Sufficiency Taskforce in 2004*.
– Assessed domains are tied to factors impacting improved employment and education prospects.– Beneficiaries scored on a five level scale from “in- crisis” to “thriving.”
*The taskforce was collaborative partnership of the county’s human services community including the Snohomish County Community Action Division of the Human Services Department, United Way of Snohomish County, and community partners.
NEOC Self-Sufficiency Matrix
Benefits of the Matrix
• Flexibility – Customize for individual host sites intervention method.
• Collaboration – Design encourages a collaborative effort among providers in defining improvement.
• Case Management – Can be used for both long term and short term interventions.
• Self-Assessment – Keeps the member focused on the goal of self sufficiency.
• Management – Assists in determining what is or is not working.
• Communication – Helps communicate provision of service.
• Measurement – Measures beneficiary progress.
Challenges
• Lack of understanding of how to complete the Matrix
• Technology failures
• Collecting reports
• Basic math issues
• Counting assessments
Overcoming Challenges
• More training and communication of requirements
• Researched and experimented with different reporting methods, which led to using Google for Nonprofits
Google for Nonprofits
• Free service: Nonprofits qualify for business level service at no cost.
• Improved communication: Multiple tools including email, chat, and Google Hangouts.
• Less paper: Assessment forms are included on a separate tab for easy access.
• Security: Encryption of mobile devices using our account.
• Reduces reporting obstacles: Members may complete assessments through their phone.
Example Google Sheet
Going Forward
• Improve methods to capture outcomes.– Work out any issues that arise from this year.
• Create process and assessment of beneficiaries’ perspective on services provided.
– Potentially using online survey methods or randomized follow-up calls from program staff.
CalSERVES: AmeriCorps Programs ofNapa County Office of Education
CalSERVES AmeriCorps Programs
• After School: Local Direct Service, 34 FT/60 HT
• Collective Impacts: Local Direct Service, 20 FT/36 HT Members
• PREP: State-wide Capacity Building, 40 FT Members
• VIP: State-wide Capacity Building, 100 FT Members
What We Did
• Develop Evaluation Goals and Questions• Employ an Evaluation Firm that Understands
AMC• Determine the Method of Evaluation• Determine Strategy for Comparison Group• Conduct Analysis• Report Findings• Discuss Program Changes and Next Steps for
Research
Evaluation questions
• Does participation in the CalSERVES VIP program increase
organizational capacity to utilize volunteers compared to
organizations that did not take part in the program?• Does participation in the CalSERVES VIP program improve
organizational capacity concerning volunteer recruitment,
training, and retention compared to organizations that did
not take part in the program?• Does participation in the CalSERVES VIP program improve
organizational capacity to create and sustain successful
volunteer programs compared to organizations that did not
take part in the program?
VCA Instrument
• To measure the effect of program participation on organizational and volunteer capacity over time, data were collected using the Volunteer Capacity Assessment (VCA), a survey that includes three sections with items focused on: organizational capacity, volunteer recruitment, and elements of a successful volunteer program.
• The VCA is designed to assess how well a non-profit or educational organization is prepared for recruiting, training, and utilizing volunteers to achieve its mission and goals. The instrument asks for organizations to report how well their practices align with the best practices listed on the survey.
Propensity Score Matching
List of Characteristics Used in Propensity Score Matching
What the Results Showed
Average Pre/Mid VCA Score and Average Post VCA Scores, Matched Comparison
Conclusion
“Overall, the partner sites that participated in the CalSERVES VIP program reported strong, positive changes over time on the items included in the VCA instrument. Organizations that did not participate in the program reported much smaller (and sometimes negative) changes on the VCA items. Because the matching procedure diminishes the likelihood that factors other than participation in the program influenced the change in reported scores over time, the evidence strongly suggests that the CalSERVES VIP program positively impacts organizations’ capacity concerning volunteer programs.”
Challenges
• Finding the right evaluation firm• Willingness of comparison sites to participate• Resources to fund evaluation
Successes
• Kept costs reasonable using comparison group design.
• Able to develop and implement a comparison group design that allowed for the development of some real understanding of impact.
• Able to use data to inform the program and determine next steps.
Next on Our Research Agenda
• Examine the type of organizations (both in size and scope of work) that benefit the most from the CalSERVES VIP program.
• Explore the specific types of services that are most beneficial to these organizations.
Conservation Legacy:
A Progressive and Phased Approach
Amy Foss
Overview – Our Projects and Past Evaluations
What problems do our land management agencies face?
• Multiple uses that impact the land, how to maintain natural resources that communities and human use, preservation, habitat and species protection
Our Projects/Intervention• Trail improvement, forest fuels reduction, habitat restoration,
invasive species removal, surveys and research
Our Evaluation History • Measurement of member outcomes via the Public Lands Service
Coalition Study
• Project specific measurement of outcomes via agency experts
• Challenges/What we are pursuing: Wanting to delve deeper into these questions and research additional ways to make these measurement more evidence based
Challenges We Face
– Outcomes are so varied and broad, not possible to measure everything that we do.
– We don’t have a singular activity with a singular output.
– Outcomes are not immediate, can take years to materialize.
– Don’t know where we are going to end up!
Phased Approach to New EvaluationThree Phases
• Phased Approach – Three phases over multiple years
• Independent Research Team working in partnership with us throughout the three phases. – Research team consists of professors and graduate
students at NC State and BYU. NC State and BYU – NC State is the lead.
– Researchers also working on similar studies including one focused on the community impacts of Rails to Trails.
Phase 1 Literature Review
2 Goals of Literature Review– First, it was designed to provide a summary of scientific
evidence about the impacts of conservation activities similar to those conducted by conservation corps.
– Second, it was designed to identify potential strategies and indicators to measure impacts in future research and evaluation with conservations corps.
Phase 2
• Find a few things that conservation corps do from the literature review and then use similar measurement tools (as were used in the literature review) and apply them to our activities to determine possible impacts.
• Will be a trial of different measurement approaches to determine best possible connections, impact, and outcomes that are reliable.
• Identify how to measure multi-year impacts or change in condition in partnership with land management agencies.
• Determine what is feasible and cost effective without over burdensome cost.
Step 3
• Expand upon Step 2 and incorporate additional conservation corps in different areas across the country.
• Final assessment of the results.• Figure out the final implementation costs and
determine reality.
Our Goal
• Multi-year approach to finding best possible measurement tools using experts in the field and researchers to help us.
• Never possible to have a fully experimental evaluation.
• We don’t know exactly where we are going to end up!
Jumpstart Learning Collaborative
• Jumpstart provides small grants ranging between US $7,500 and $10,000 for promising research that addresses significant questions on the processes and impact of Jumpstart.
• Provides researchers with the opportunity to investigate the influence of Jumpstart (1) on the lives of young children who live in low-income and high-stress communities, and (2) on adult volunteers who implement the Jumpstart program.
• Up to three grants will be awarded each year.
• Funded studies may be carried out using any research method or approach as long as the focus of the project is on examining the Jumpstart program. Priority will be given to applications that use rigorous research designs and methodologies.
Research Questions1. In what ways does Jumpstart contribute to children’s kindergarten readiness, specifically social emotional and executive functioning skills?
2. In what ways does Jumpstart support the development of children’s phonological awareness, oral language, and/or books and print knowledge?
3. In what ways does Jumpstart’s implementation fidelity (e.g. fidelity to curriculum, Corps member training) affect 1) Jumpstart children’s kindergarten readiness and 2) Corps members’ learning, knowledge, and overall Jumpstart experience?
4. In what ways does Jumpstart’s Corps member training affect children’s kindergarten readiness?
5. Does the Jumpstart Program impact children with different demographic backgrounds in different ways (e.g., Dual language learners or entering language levels)?
6. What impact does Jumpstart have on our Corps members?
Examples of Funded Studies
A Proposal to Study the Impact of Participating in Jumpstart on Corps Members
• Impact that participation in Jumpstart has on Corp members’ civic engagement, knowledge of child development, work readiness, and teamwork and collaboration skills, when compared to a similar group of students who are not involved in Jumpstart.
Proposed Evaluation Study of Full-Class Delivery of Jumpstart in Chicago Sites
• Quasi experimental evaluation - researchers will be able to compare changes over time in a large sample of preschoolers receiving the full-class delivery model, to demographically matched children attending similar preschool programs, but not receiving Jumpstart.
Assessment of Language, Literacy, and Social-Emotional Development Among Children Attending Preschools Served by Jumpstart
• This study will pilot a comprehensive assessment battery including direct measures of cognitive and social skills targeted by Jumpstart.
A Multi-system Approach to Examining Effects of Jumpstart on Children’s Stress Response
• The purpose of the current project is to explore the effects of Jumpstart involvement on children’s stress physiology. We hypothesized that enrollment in Jumpstart would decrease children’s levels of cortisol and alpha-amylase across the preschool day.
Jumpstart Research Consortium
Bi-annual Convening of Researchers
Goals of Research Consortium
• To leverage learning from external research studies for program improvement
• To expand our knowledge on establishing and maintaining university research partnerships
• To determine avenues for future research through the Jumpstart Learning Collaborative
What questions do you have?