Helping Our Clients Identify and Measure
Outcomes
Edward M. Haugh Jr.ESC Consultant
ESC Brown Bag # 2, April 17 2013
Introduction
By the end of this session:• Define outcomes• Differentiate outcomes from outputs• Define the logic model• Describe several uses of a logic model• Develop a logic model• Define “indicators”• Develop a logical framework
Impact
• You will love or at least tolerate outcome-based planning
• You will be able to discuss outcomes based planning, monitoring and evaluation with your ESC clients
What is an outcome
Outcomes are benefits or changes for individuals or populations during or after participating in a program:
Outcomes
• Often expressed as verbs in past tense–Improved (nutrition, morale)–Increased (in employment) –Reduced (HIV transmission)–Enhanced (knowledge)–Maintained (vaccination coverage)
Outcomes
Reduced smoking among teenagers
Changed attitudes towards smoking (by non-smoking teenagers
School readiness among children attending preschool program
Other examples
Improved board performance after an ESC engagement
Improved strategic focus after an ESC engagement
Employment after attending a dress-for- success program
Improving the Delivery of Prenatal Care and Outcomes of Pregnancy: A Randomized Trial of Nurse Home Visitation
Outcomes of Nurse Visit Program
• Became aware of more community services• Attended birth classes more frequently• Made more extensive use of the nutrition
supplementation program for women, infants and children
• Made greater dietary improvements,
Outcomes of Nurse Visit Program
• Reported that their babies’ fathers became more interested in their pregnancies
• Were accompanied to the hospital by a support person during labor more frequently
• Reported talking more frequently to family members, friends and service providers about their pregnancies and personal problems
• Had fewer kidney problems
Outcome Indicators
• Outcomes sometimes confused with outcome indicators– Outcome - girls participating in a training program
obtain formal employment– Outcome indicators – the # and % of participants
employed within 3 months of completing the program
Importance of Focusing on Outcomes
• Outcomes are what matter to stakeholders• Focusing on outcomes promotes
accountability• Government, and funders demanding real
tangible results on the ground• Measuring outcomes helps us know if we are
we really getting value for our money
Outcomes help us
• Strengthen existing services• Target effective services for expansion• Identify staff and training needs• Develop and justify budgets• Prepare long-range plans• Focus board on programmatic issues
Funder Asked ESC the Following:
• What are your measurable outcomes, i.e. number of communities impacted, number of children served?
• Mark each category that applies to your grant measures.Increased visibility/awareness of issue
Effected policy changeEngaged constituents & beneficiaries and participationIncreased resources for issueEducated constituentsImproved condition/ quality of life in communityInfluenced stakeholders
Outcomes vs. Outputs
• Outcomes are the results
• Outputs are the tangible products produced by an intervention–Usually expressed as nouns–Can be counted
Outputs
• Classes taught• Brochures distributed• Clients served• Contracts completed• Board retreats completed• Strategic plans prepared
Many Nonprofits Report Outputs Rather than Outcomes
• Main Street Homeless Shelter provided 2,903 nights of safe shelter for 173 men.
• 23,661 hot, nutritious meals were served at the Maple Avenue Kitchen.
• 100 residents received free health care and 168 residents received free mental health care.
What Wrong with Using Outputs• Mosquito bed nets might be used for
wedding dresses or fishing nets • Loans repaid might not mean income
generated• Job training might not lead to jobs
So Why Don’t We Focus on Outcomes
• Outputs are tangible• Outcomes are intangible• Its easier to count tangibles than to describe
intangibles• Outcomes take longer to produce• Measuring outcomes is challenging • Measuring outcomes can be expensive and
time consuming
But Tracking Outcomes will Not……
• Explain why an outcome is or is not being achieved– Still need to track inputs, activities, outputs
• Prove that the outcome is the result of your program
Outcome Model
Logic Model
Inputs Processes Outputs Outcomes Impact
Logic Model Definitions
• Inputs are the resources required to undertake activities
• Activities (or processes) are what the agency, program or strategy does to fulfill its mission
• Outputs – tangible products• Outcomes – changes, benefits• Impact - a longer-term outcome
Inputs ProcessesOutputs Short-term
Outcomes
Medium-term
Outcome
• Money• Staff time• Facilities• Equipment• Laws• Regulations• Funders
requirements
• Educate consumers
• Educate providers
• Distribute vaccines
• Establish data base
• Establish reminder systems
• Monitor Coverage
• Brochures• Vaccine supplies• Doses of
vaccines delivered
• Data bases established
• Providers monitoring
• Parents aware of vaccine benefits
• Provider awareness
• Changed attitudes
• Children immunized
• Reduced incidence of HPV
Reduced number of girls who are infected with HPM
Long-term
• Reduced incidence of cervical cancer
Reduced mortality for cervical cancer
A logic model for a program to distribute HPV Vaccine
Impact
Reduced premature death
Inputs Activities OutcomesOutputs Impact
It is important to note the broad environmental context of the intervention:
Political environmentMacroeconomic picture
Other “players” in field Public attitudes Environment
Policy context
Source: Imas and Rist (2009)
How can we use the logic model
• ESC Project planning• Strategic Planning• Program Monitoring• Program Evaluation
Inputs Processes OutputsShort-term Outcomes
Medium-term
Outcome
• Staff time• CRM Time• Lead time• Consultant
time• Materials• Car and
gasoline• Knowledge• Experience• Place to meet
• Contracting• Analysis• Study• Meetings w
client• Meetings with
stakeholders• Consultant
meetings
• Progress reports
• Draft report• Final report• Board
presentation
? ?
Impact
• ?
Logic model for an ESC Engagement
Deliverables
Inputs Process Outputs
Short-term Outcomes
Medium-term Outcome
Strategic Plan Approved
Impact
Example of a logic model for an ESC Strategic Planning Engagement
Strategic Plan in regular Use
Longer-term Outcome
Strategic Plan improves agency
performance
Community benefits from
agency’s improved
performance
Inputs ProgramsOutputs
Short-term Outcomes
• Money• Staff time• Facilities• Equipment• Laws• Regulations• Funders
requirements
Program A
Impact
Non-profit Logic Model
Program B
Program C
Program D
Output A
Output C
Output B
Output D
Outcome 1
Outcome 2
Outcome 3
Outcome 1:Improved Financial Position
Outcome 5:New Training
Program Developed
Outcome 2:New Facility Developed
Outcome 3:Strengthened
Volunteer force
Outcome 4:Improved
Public Recognition
Improved Ability to Fulfill Mission
Outcome 1:Improved Financial Position
Improved Ability to Fulfill Mission
Strategy 1: Capital Campaign
Completed
Strategy 2: Increased donor
base
Strategy 3: Client Fees Increased
Indicators
• Measure the quantity or quality of the outcome
Indicators
Outcome: Participants in a job training program obtain a job in a timely manner
• # (%) of participants employed within 6 weeks • # (%) of participants still employed 3 months
after completing the program
Indicators
Outcome: Homeless people obtain secure housing
• # (%) of homeless moved into temporary quarters and stayed for 6 weeks
• # (%) of homeless who obtained a job and were still employed after 3 months
Indicators Targets
• # (%) of participants employed within 6 weeks
• # (%) of participants still employed 3 months after completing the program
• 60% of participants will gain employment within 6 weeks
• 95% of participants who obtained employment will still be employed in 3 months
Logical FrameworkComponent Indicator Baseline Target
Outcome
Output
Program/Activity/Process
Input
1. Choose the Outcomes to Measure
2. Specify the Indicators for the Outcomes
3. Prepare to collect data on the Indicators
4. Try out the Measurement System
5. Analyze and Report Findings
6. Improve the System
What Can Our Clients Do to Measure Outcomes
7. Use the Findings
Two Great Resources
• Measuring Outcomes, A Practical Approach, The United Way Foundation• Logic Model Development Guide,
WK Kellogg Foundation
Thank you for participating in ESC Brownbag No. 2.Look for more Brownbags to come