performance measures americorps project director training saratoga, ny october 7 th – 9 th, 2013
TRANSCRIPT
Performance Measures
AmeriCorps Project Director TrainingSaratoga, NY
October 7th – 9th, 2013
Theory of Change
Activity
Performance Measure Basics Overview
Activity/Questions
AGENDA
Theory of Change
INTRODUCTIONTheory of Change Perspective
Looks at cause and effect relationships
Identifies specific interventions to achieve the desired result
Shifts thinking from “what are we doing” to focus on “what do we want to achieve”
Overview of Prohibited ActivitiesTheory of Change Elements
STATISTICS documenting the need
STATISTICS documenting the need
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Prohibited ActivitiesEveryday Life Example
I have strep throat (problem) I will take antibiotics (intervention) I will get better (outcome)
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Prohibited ActivitiesEveryday Life Example cont.
EVIDENCE Penicillin Amoxicillin×Tetracycline
EVIDENCE Penicillin Amoxicillin×Tetracycline
There are many antibiotics out thereWhich antibiotic do I take for strep?Look at evidence to make the choice
Overview of Prohibited ActivitiesTheory of Change Elements
STATISTICS documenting the need
STATISTICS documenting the need
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community?
Overview of Prohibited ActivitiesTheory of Change Elements - Community Need
Data documenting problem/need should answer these questions:
SCOPE: Who and how many are directly affected? How severe is this?
SIGNIFICANCE: What makes this a compelling need? Is it likely to become worse? What will happen if we do nothing?
CAUSE(S): Why does the need exist? How is it perpetuated?
Overview of Prohibited ActivitiesTheory of Change Elements - Statistics
STATISTICS documenting the need
STATISTICS documenting the need
The best data comes from reputable primary sources like government agencies, institutions, and universities that do their own research that is: Locally relevant Up-to-date
Note: New reports are not primary sources but could be used as back-up to primary evidence.
Data documenting problems/needs should be statistic that document the extent and severity of the problem.
Overview of Prohibited ActivitiesTheory of Change Elements - Outcome
STATISTICS documenting the need
STATISTICS documenting the need
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
What change are you hoping to make related to the identified need?
Overview of Prohibited ActivitiesTheory of Change Elements - Outcome
Healthy Futures Program Example
Possible outcomes to measure: Increased knowledge of what is “healthy food” More frequent choice of healthy foods to eat More frequent involvement in physical activity or exercise Improved physical condition
Identifying the Intended Outcome.
Overview of Prohibited ActivitiesTheory of Change Elements - Intervention
STATISTICS documenting the need
STATISTICS documenting the need
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome?
Overview of Prohibited ActivitiesTheory of Change Elements - Intervention
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Design – who does what with whom? Dosage
Frequency – how many sessions a week? Intensity – length of each session. Duration – how many total weeks of sessions?
Describe the design and dosage of your intervention (service activity):
Overview of Prohibited ActivitiesTheory of Change Elements - Intervention
Design: national service participants implement the Shape Up curriculum with economically disadvantaged urban girls ages 14-16 to increase physical activity (30 minutes/session) and educate them on healthy eating
Frequency: twice a week afterschool Intensity: 60 minutes per session Duration: 12 weeks
Healthy Futures Program Example
Overview of Prohibited ActivitiesTheory of Change Elements - Evidence
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Evidence demonstrates that the proposed intervention (design, dosage) is likely to lead to the outcome.
Service activities that are supported by research to demonstrate a high likelihood that these activities will address the need, and result in intended changes.
Overview of Prohibited ActivitiesTheory of Change Elements - Evidence
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Evidence provides a reality check for theory of change
New programs: What is the recommended design (specific program activities) and dosage (frequency, intensity, and duration) to achieve an intended outcome? For existing programs: Is there sufficient evidence for the intervention to continue its use? Based on the evidence, are there modifications to the intervention that should be considered to have greater impact? Do you need to choose a new intervention?
EVIDENCE: Information that supports your choice. It is NOT enough to just say “we believe” our intervention is “likely” to be successful.
Overview of Prohibited ActivitiesTheory of Change Elements - Evidence
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
EVIDENCE •Guides choice of intervention
•Supports cause-effect relationship
Your past performance measurement outcome data
Results from an impact evaluation of your program
Research that documents the outcomes of similar programs
Impact evaluations that document outcomes of similar programs
Possible sources of evidence:
Overview of Prohibited ActivitiesTesting Your Theory of Change
PLAUSIBLE: Does the logic of the model seem correct: “if we do these things, will we get the results we expect?”
FEASIBLE: Are resources sufficient to implement the chosen intervention?
MEANINGFUL: Are intended outcomes important? Is the magnitude of expected change worth the effort?
Is your Theory of Change:
Overview of Prohibited ActivitiesTheory of Change Summary
A theory of change identifies cause/effect
The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence
Data documenting community need should show scope, significance, and causes
Summary of Key Points:
Case Study
Performance Measure Overview
Overview of Prohibited ActivitiesPerformance Measures
Accountability to funders and stakeholders Tell your story, justify funding
Recognizing progress; measuring change Reliable information collected in a systematic way
Program improvement Spot and correct problems Strengthen the intervention Determine where to allocate limited resources
Why Measure Performance?
Overview of Prohibited ActivitiesPerformance Measures
Using Performance Measurement in Program Management
Overview of Prohibited ActivitiesPerformance Measures
Using Performance Measurement in Program Management continued
Overview of Prohibited ActivitiesPerformance Measures – Outputs and Outcomes
OUTPUTSAmount of services being
providedo People servedo Products createdo Programs developed
OUTCOMESReflects the changes or
benefits that occur
Can reflect changes in individuals, organizations, communities, or environment
Address changes in attitudes/beliefs, knowledge/skill, behavior, or conditions
Overview of Prohibited ActivitiesPerformance Measures – Outcomes
Outcome Example: EDUCATION
Overview of Prohibited ActivitiesPerformance Measures - Alignment
Aligning Theory of Change with Performance Measures
Overview of Prohibited ActivitiesPerformance Measures - Alignment
Alignment: Logical flow and connection between the different elements.
Alignment helps you:
Strengthen your theory of change
Articulate you performance measures
Measure more accurately
Cleary report change
Overview of Prohibited ActivitiesPerformance Measures - Alignment
Example: Aligned Output-Outcome with Performance Measure
Overview of Prohibited ActivitiesPerformance Measures – Impact Evaluation
Differences Between Impact Evaluations and Performance Measures
Key AreasPerformance Measurement
Impact Evaluation
CausalityAssumes causality; does not “prove” theory of change
Seeks to show causality and “prove” theory of change
ImplementationTracks outputs and outcomes on a regular outgoing basis
May occur from time to time, but not on a regular ongoing basis
Time FocusShorter term focus; what can be measured within one year
Longer term focus
DesignBalance rigor with practicality
Uses most rigorous evaluation design
Overview of Prohibited ActivitiesPerformance Measures- Summary
Performance measurement is a systematic process of measuring progress (outputs and outcomes)
Performance measurement does not seek to “prove” a theory of change, but can provide snapshots.
Impact evaluation can determine if results occurred because of the intervention.
Strong performance measures align with the theory of change (need, intervention and outcome)
Key Points
Overview of Prohibited ActivitiesPerformance Measures- Summary
CNCS Priorities and Performance Measures:
http://www.nationalservice.gov/resources/npm/home
Program-Specific Notices of Funding Opportunities and Application Instructions:
http://www.nationalservice.gov
Additional Resources
Project Director TrainingSaratoga, NY
October 7th – 9th, 2013
Questions?