success measures: transforming outcome evaluation presentation to grassroots grantmakers maggie...
TRANSCRIPT
Success Measures: Success Measures: Transforming Outcome EvaluationTransforming Outcome Evaluation
Presentation to Grassroots Grantmakers Maggie Grieve, Director, Success MeasuresDebby Visser, Director, Success Measures Investments and PartnershipsNancy Kopf, Senior Manager, Success Measures
Evaluation ServicesRobert Krompak, Economic Development Specialist,
Neighborhood Housing Services of Toledo
September 13, 2011
Success Measures OverviewSuccess Measures Overview Providing the community development field and
philanthropy with practical, credible and accessible ways to collect and use outcome data for continuous evaluative learning, to demonstrate results and tell stories of change.
• Participatory evaluation method• Indicators/data collection tools that measure outcomes• Training and technical assistance to build organizational
capacity to conduct evaluation• Consulting to plan evaluation frameworks and support
the entire evaluation cycle• Web-based data system
Shared Outcome Indicators/Data ToolsShared Outcome Indicators/Data Tools• For a typical evaluation, users select 3 – 5 indicators/data
collection tools from over 240 choices in 8 categories:Affordable Housing Community Building and Organizing Community StabilizationEconomic Development Financial Capability Green Affordable Housing and CommunityRace, Class and CommunityValue of Community Development Intermediary Services
• Tools focus on valued levels of changeIndividualCommunity OrganizationalBroader community
Consulting and Training/TA SupportConsulting and Training/TA Support
• Assess evaluation interests, needs, capacity• Document program rationale/theory of change• Plan evaluation framework including definition of
evaluation questions and measures• Develop data collection plans• Train organizations in data collection methods and/or
coordinate data collection• Support implementation and data analysis/reporting• Facilitate organizational learning from data and
integration of evaluation process into ongoing work
Success Measures Data SystemSuccess Measures Data System
• Simple to use, sophisticated in design and scale
• Web-based service provides:– Process Guidance– Success Measures Indicators and Data Collection Tools– Ability to send surveys electronically – Data Tabulation and Aggregation– Data Storage – Data Export to Excel– Options to add own indicators and data tools
ExamplesExamples
• Wells Fargo Regional FoundationEngaging grantees to explore grant & portfolio level outcomes
• Habitat for Humanity International Building the capacity of staff and affiliate organizations to measure program and community change
• NeighborWorks AmericaUsing a common framework to document community building and organizing strategies
• Neighborhood Housing Services of ToledoDemonstrating neighborhood and resident change, leveraging investments
Wells Fargo Regional FoundationWells Fargo Regional FoundationEvaluation of Neighborhood Revitalization PortfolioEvaluation of Neighborhood Revitalization Portfolio
• Purpose: Document outcomes of neighborhood planning and revitalization grant programs; analyze across portfolio
• Participatory Engagement: 50+ grantees use common framework and data collection tools to conduct resident quality of life surveys and document neighborhood change
• Result: Foundation/grantee engaged in a learning process through use of common evaluation; longitudinal data on neighborhood change
• Benefits: Improved grantee evaluation capacity; comprehensive, evolving understanding of outcomes of neighborhood change investments
Habitat for HumanityHabitat for HumanityEvaluating Homeowner and Community Evaluating Homeowner and Community Change Change • Purpose: Develop an outcome framework for primary
Habitat programs; build staff and affiliate evaluation skills
• Participatory Engagement: Involved Habitat leadership and affiliates in defining outcomes and selecting common measures at homeowner and community levels
• Results: A pilot evaluation testing new evaluation framework; incorporation of this framework and evaluation method into new Neighborhood Revitalization Initiative
• Expected Benefits: 10 HFHI staff trained to provide evaluation TA to affiliates; 100+ affiliates providing common data to demonstrate results of new Habitat approaches
Habitat for Humanity Evaluation Habitat for Humanity Evaluation FrameworkFramework• Residents’ Perceptions of Habitat Homeownership and Living in
the Neighborhood– Homeowner Benefits Survey– Homeowner Customer Satisfaction Survey– Resident Perception of the Neighborhood Survey– Resident Perception of the Neighborhood Focus Group/Interviews• Understanding Effectiveness of Affiliates’ Collaborations– Interviews/Surveys: Organizations’ Perceptions of Collaborative
Effectiveness • Community Physical Conditions – Block Conditions Observations– Residential Property Observations– Vacant Property Observations• Local Economic Impact of New Construction/Renovation for
Homeownership– Forecasts the economic impact of homeownership projects
NeighborWorks America NeighborWorks America A Common Evaluation for Community Building/OrganizingA Common Evaluation for Community Building/Organizing
• Purpose: Develop a common evaluation for community building and organizing programs that measures change at individual, group/organization and community levels.
• Participatory Engagement: Framework developed by sample of organizations interested in measuring outcomes of community building efforts; built on existing Success Measures evaluation tools. Peer calls during planning and implementation.
• Expected Results: 12 NeighborWorks organizations collecting baseline data in 2011.
• Benefits: Baseline data on incremental development of individual, group and community change resulting from organizing and small project actions.
Community Building/Organizing Community Building/Organizing Evaluation Framework Evaluation Framework
1. INDIVIDUALS: Measures perceptions, capabilities, and behaviors of residents, tenants, or other participants in community building and organizing activities.
2. GROUPS/ORGANIZATIONS: Measures functioning and effect of ongoing resident-led entities, such as neighborhood associations, tenant councils, and crime watches.
3. NEIGHBORHOODS/COMMUNITIES: Measures impact of community building and organizing on the neighborhood or community overall.
Evaluation Tools: IndividualsEvaluation Tools: Individuals
• Individual Demographics Age, gender, own/rent, tenure in community, property resident or not
• Resident/Tenant Engagement & Empowerment Participate in activities, serve as leader, comfortable voicing opinions, influence decisions, believe can bring about change
• Leadership Development Follow up to resident leadership training: Leadership roles taken on, how apply skills learned, sought out as leader, changes attributed to training
Evaluation Tools: Groups/OrganizationsEvaluation Tools: Groups/Organizations
• Group Description Name, type of group, when established, purpose
• Organizational Capacity Representative, accountable to community, effective leadership
• Group Influence/Impact Successful in addressing issues, serve as catalyst, significant accomplishments
Evaluation Tools: NeighborhoodsEvaluation Tools: Neighborhoods
• Neighborhood/Community DescriptionName, boundaries, population size, # households
Plus, select one or more of the following
• Social Connectedness Social capital, neighborliness, sense of belonging
• Capacity for Collective Action Advocacy, collaboration, ability to act as a group, voice is heard
• Quality of Life Safety, physical environment, public space, responsiveness of public services, available amenities, health and wellness, satisfaction, recommend neighborhood
NeighborWorks America NeighborWorks America A Common Evaluation for Community A Common Evaluation for Community StabilizationStabilization• Purpose: Develop a common evaluation for community
stabilization efforts that measures market health, physical conditions, community perceptions.
• Participatory Engagement: Framework developed by nonprofit practitioners, researchers and funders. Staff and volunteers involved in data collection, analysis and use of results. Peer convening to maximize learning.
• Result: 18 NeighborWorks organizations completed baseline data collection; repeating in 2012 or 2013.
• Benefits: Baseline data on neighborhood conditions and vacancies directly used for strategy development. Community confidence data and neighborhood surveying also resident engagement tools.
Example: Example: Neighborhood Housing Services of Toledo Neighborhood Housing Services of Toledo
• Documented change from organizing, investments and other stabilization efforts in neighborhoods hard hit by foreclosures and other factors.
• Evaluation in one neighborhood, led to expansion to 4 other areas.
• Created ongoing partnership with large hospital to examine conditions in neighborhood surrounding medical center; led to new investments due to better understanding of safety and other issues.
WHAT ELEMENTS MAKE UP A SUCCESS MEASURES EVALUATION?
UTILIZING SUCCESS MEASURES AS A CATALYST FOR COMMUNITY CHANGE
Response to Housing Issues Identified by Legacy Area Residents
Response to Public Safety Issues Identified by Legacy Area Residents
Building Community Connections
www.nw.org/network/ps/successmeasures/
Maggie Grieve, Director [email protected] (202) 220-2337
Debby Visser, Director, Investments and [email protected] (202) 220-2348
Nancy Kopf, Senior Manager, Evaluation [email protected] (202) 220-2427
Bob Krompak, Economic Development Specialist, NHS of Toledo, Inc.
[email protected] (419)691-2900