logic model: a program performance framework
DESCRIPTION
Logic Model: A Program Performance Framework. Madison Workshop December 5, 2001 Ellen Taylor-Powell, Ph.D. Evaluation Specialist UW- Extension-Cooperative Extension. A New Era What gets measured gets done If you don’t measure results, you can’t tell success from failure - PowerPoint PPT PresentationTRANSCRIPT
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic Model: A Program Performance Framework
Madison Workshop
December 5, 2001
Ellen Taylor-Powell, Ph.D.
Evaluation Specialist
UW- Extension-Cooperative Extension
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
A New Era• What gets measured gets done• If you don’t measure results, you can’t
tell success from failure• If you can’t see success, you can’t
reward it• If you can’t reward success, you’re
probably rewarding failure
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
• If you can’t see success, you can’t learn from it
• If you can’t recognize failure, you can’t correct it
• If you can demonstrate results, you can win public support
Osborne and Gaebler, 1992 in MQ Patton, 1997:14
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic Model is…
• Picture of a program • Graphic representation of the program
“theory” or “action” – what it invests, what it does and what results
• Logical chain of if-then relationships; if x occurs, then y will occur
• Core of program planning and evaluation
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
LOGIC Reasonable
To be expected
MODEL Represents reality, isn’t reality
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Why Logic Models--Why the Hype?
• Shows difference between what we do and impact we are having
• Provides a common vocabulary
• Focus on quality and continuous improvement
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic Model: Origins• Private Sector: Total quality management• Public Sector: GPRA, performance
budgeting• Non-Profit Sector: Outcomes measurement,
eg. United Way• International Arena: Results Framework of
USAID, etc. • Evaluators: Evaluability assessment,
bennett hierarchy
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic Model
SITUATION
OUTCOMESINPUTS OUTPUTS
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
HEADACHE
Feel betterGet pills Take pills
Everyday Logic Model
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
An Extension Example: Business Counseling
Extension invests time and resources
A variety of educational activities are provided to business owners who participate
These owners gain knowledge and change practices resulting in
Improved business performance
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
LOGIC MODEL: Program Performance Framework
INPUTS OUTPUTSActivities Participation
What we invest
StaffVolunteersTimeMoneyMaterialsEquipmentTechnologyPartners
What we do
WorkshopsMeetingsCounselingFacilitationAssessmentsProduct dev.Media workRecruitmentTraining
Who we reach
ParticipantsCustomersCitizens
Reactions
ASSUMPTIONS1)2)3)4)
ENVIRONMENTInfluential factors
SITUATION
What the short term results are
Learning
AwarenessKnowledgeAttitudesSkillsOpinionsAspirationsMotivations
Action
BehaviorPracticeDecisionsPoliciesSocial action
What the ultimate impact(s) is
Conditions
Social Economic Civic Environmental
OUTCOMES-IMPACTShort Medium Long-term
What the medium term results are
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logical Linkages: Series of If-Then Relationships
IF THEN IF THEN IF THEN IF THEN
INPUTS OUTPUT OUTCOMES
Programinvests time
&money
Resourceinventorycan be
developed
Familieswill knowwhat is
available
Familieswill access
services
Familieswill have
needs met
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Activity (p 2-12)
Completing the causal linkages
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
LOGIC MODEL: Program Performance
INPUTS OUTPUTS OUTCOMES
Program investments
Activities Participation Short Medium
What we invest
What we do
Who we reach
What results
SO WHAT??
Long-term
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Outcomes vs. Activities
BE OUTCOME DRIVEN,
NOT ACTIVITY DRIVEN
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Activity (p 2-13)
Program Performance Levels
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Situation• Situational analysis
– Need– Asset
• Problem analysis
• Priority setting
• Engaging others
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
INPUTS
FacultyStaff
MoneyTime
VolunteersPartners
EquipmentTechnology
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
OUTPUTS What we do Who we reach
ACTIVITIES Workshops Counseling On-farm research Facilitation Product development Curriculum design Trainings Field days Conferences Media
PARTICIPATION Participants Clients Customers Users
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
OUTCOMESWhat results for individuals, businesses, communities..…
SHORTLearning
Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation
MEDIUMAction
Behavior Practice Decisions Policies Social action
LONG-TERMConditions
Human Economic Civic Environment
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Staff
Money
Partners
Design parent ed curriculum
Provide 6 training session
Targeted
parents
attend
Parents increase knowledge of child dev
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS OUTPUTS OUTCOMES
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Activity (p 2-24)
Which are the Outcomes-Impacts?
Chain of OutcomesSHORT MEDIUM LONG-TERM
Seniors increaseknowledge of food contamination risks
Practice safe cooling of food; food preparation guidelines
Lowered incidence of food borne illness
Participants increaseknowledge and skills infinancial management
Establish financial goals,use spending plan
Reduced debt andincreased savings
Community increasesunderstanding ofchildcare needs
Residents and employersdiscuss options andimplement a plan
Child care needs are met
Empty inner city parkinglot converted tocommunity garden
Youth and adults learngardening skills, nutrition,food preparation and mgt.
Money saved, nutritionimproved, residents enjoygreater sense ofcommunity
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop Focus of Outcomes• Individual
– Child, parent, client, resident
• Group– family, team, community
group
• Agency, organization
• System
• Community
Child is prepared to enter school; teen uses
savings/spending plan
Community group has inclusive membership policy; family increases its savings
Communications are more open; agency adopted smoke-free policy
Family serving agencies share resources
Shared community responsibility has increased; youth are valued as contributing members
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
How far out the outcome chain do we go?
• What is logical?
• What is realistic?
• What is meaningful?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Staff
Money
Partners
Develop parent ed curriculum
Deliver 6 interactivesessions
Targeted
parents
attend
Parents increase knowledge of child dev
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS OUTPUTS OUTCOMES
WHICH OUTCOMES???
Research
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Outcome of Interest?
• Inherently valued outcome (Mohr, 1995)
– higher level outcome is immaterial– we are willing to assume that a higher outcome
will also be attained if we achieve the outcome of interest
• Participant valued outcome: if participants experience a change or benefit that makes a a real difference to them (United Way, 1999)
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Assumptions• Beliefs about the program
– the participants– the way the program will operate– how resources, staff will be engaged– the theory of action
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Assumptions, cont.
• Faulty assumptions are often the reason for poor results
• Check and test assumptions– Identify potential barriers for each
‘if-then’ sequence
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Environment–Influential Factors
• Extension program does not exist in a vacuum
• Context of the program – politics, family circumstances, cultural milieu,
demographics, economics, values, biophysical environment, policies, services
• What affects the program over which you have little control?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
What does a logic model look like?
• Graphic display of boxes and arrows
• Any shape possible
• Circular, dynamic
• Relationships, linkages
• Level of detail
• simple
• complex
• Multiple models
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Compare examples
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic Model: Limitations• Represents reality, not reality
Programs are not linear
• Focuses on expected outcomes
• Challenge of causal attribution
Many factors influence outcomes
• Doesn’t address: Are we doing the right
thing?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Benefits• Brings detail to broad goals
• Shows the ‘chain of events that link inputs to results
• Builds understanding and consensus
• Identifies gaps in logic and uncertain assumptions
• Signals what to evaluate and when
• Summarizes complex program to communicate with externals
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Building a Logic Model• New program
• Existing program
• Team; organization
• Involvement of others
• Keep it dynamic
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic Model: WORKSHEET
ASSUMPTIONS
•
•
•
INPUTS OUTPUTS Activities Participation
OUTCOMES - IMPACT Short Medium Long-term
Program TitleSituation/Problem
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Check Your Logic Model• Are the outcomes really outcomes?• Is the longest-term outcome
– meaningful?– logical?– realistic?
• Are the connections between inputs, outputs, and outcomes clear and reasonable?
• Does it represent research and best practice?• Does it represent the program’s purpose;
response to the situation?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
OUTCOMESINPUTS OUTPUTS
Activities Participation Short Medium Long
term
Programmatic investments
i
EVALUATION
PLANNING
Where does evaluation fit?
Decrease in rates - agency records
Actual use - follow-up phone interview
Increase in knowledge/skill- post session survey
# parents attending/sessionwhich parents% of parents
Quality of curriculum# of sessions delivered
Staff
Money
Partners
Design parent ed curriculum
Provide 6 training sessions
Targeted
parents
attend
Parents increase knowledge of child dev
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS OUTPUTS OUTCOMES
EVALUATION: What do you want to know? How will you know it?
Providing Leadership forProgram Evaluation
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Evaluation PlanSchedule -when/where
Method to collect info
Source of information
Indicators - how will you know it
What do you want to know?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
QUESTION TYPE OF EVALUATION
Need?
Needs assessment
Process or implementation? Process evaluation
Outcomes or impact?
Outcomes evaluation Impact evaluation
Costs and efficiencies? Cost-benefit/cost effectiveness evaluation
Evaluation Questions – What do you want to know?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Indicators - How will you know it?• The evidence or measures that indicates what
you wish to know or see:
• often multiple indicators are necessary
• may be quantitative or qualitative
• culturally appropriate
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Logic model with indicators for Outputs
and Outcomes
Program implemented
Targeted farmers
Farmers learnFarmers practice new techniques
Farm profitability increases
Number of workshops held
Number and percent of farmers attending
Number and percent who learned content
Number and percent who practice the recommendations
Number and percent show farm profits increase; amount of increase
OutputsOutcomes
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
INDICATORS: Examples
#, % with savings goal set#, % with debt reduction goal set#, % using spending plan#, % maintaining emergency fund
Improved family financial management
#, % acres managed according to BMP guidelines Quality of conservation plan implementation
Reduction in N and P application rates
#, % of Boards with youth participation
Increased youth-adult partnerships INDICATORS OUTCOME
Evaluating Your Program
Decrease in rates of abuse & neglect among these parents
#, % parents using improved skills (specify skills)
#, % parents who increased knowledge #, % parents who learned new ways
# and %parents who attended each session
Quality of curriculum
# sessions fully delivered
Staff
Money
Partners
Design parent ed curriculum
Provide 6 training sessions
Targeted
parents
attend
Parents increase knowledge of child dev
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS OUTPUTS OUTCOMES
Indicators
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Source and Method of Data Collection
Source of information• Participants• Parents• Teachers• Key informants
Method of collecting the information
• Survey• Interview• Observation• End-of-program
questionnaire• Focus group• Records
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Evaluation Plan cont. How will the data beanalyzed andinterpreted?
How will results beshared? To whom,how, when?
Who will do what, when,what resources?
University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation
Logic Model Workshop
Enfin…
• View evaluation as learning - integrate into the way we work
• Build evaluation in upfront• Ask ‘tough questions’• Make measurement meaningful• Be accountable for highest professional
standards