frameworks for program planning and evaluation: problem analysis and logic models geraldine oliva...
TRANSCRIPT
Frameworks for program planning and evaluation:
Problem Analysisand
Logic ModelsGeraldine Oliva M.D. , MPHRoberto Ariel Vargas, MPHJames Rouse-Iñiguez, MA
Paula Fleisher, MA
June 2013
Today’s Agenda
Why Evaluation?! Using a socio-ecological problem
analysis framework to identify “causal pathways”
Developing strategies to address a problem based on upstream causes and risks
Developing a logic model for program planning and evaluation
Why Evaluation?!
Root word: valueWhose values? Funders, clients, organization, etc. It helps us tell our story It helps us understand what we are
doing well and what we can improve…
“Traditional” Program Planning Cycle“Traditional” Program Planning CycleConvene
Public Health/ Community Coalition
Assess Community /MCAHResources &
Strengths/Capacity
Analyze Problem & Select
Interventions
Evaluate/ Measure
Performance
Plan & Implement Programs
DevelopObjective
s
Assess & Prioritize Health
Status/Problems
New Paradigms for Understanding Persistent Disparities
Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparities
Multi-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparities
Re-emergence of “life course model”, importance of prevention and intervention along age spectrum
Socio-Ecological Models
Lifecourse Model
UCSF Family Health Outcomes Project
Mother overweight/diabetic
Working Mom-Infant Fed
Formula
Family Culture - high fat, high
sugar diet- child overweight
Adolescent with increased # fat cells/abnormal GT
School with no PE, unsafe parks
latch key child watches TV /gets more overweight
Lifecourse Model as a Cycle
Poverty/fast food/food habits in young woman
Infant born LGA/Abnormal
GT
How do we Incorporate these New Paradigms into a Community/Public Health
Planning Framework ?
African-American, Latino Children Drink More Sugary Fruit Juice Than Their White PeersShare this story:Share PrintBy Juliana Bunim on May 30, 2013Email
While there has been a steep decline in kids’ consumption of sugar-sweetened beverages in California, African-American and Latino children may be replacing soda with 100 percent fruit juice while their white peers are not, according to a new study from UC San Francisco.The study was the first to compare trends of sugar-sweetened beverages and 100 percent juice consumption in California. “The decrease in the consumption of sugar-sweetened beverages among kids is a promising public health trend,” said Amy Beck, MD, MPH, lead author and pediatrician at UCSF Benioff Children’s Hospital and San Francisco General
Hospital and Trauma Center. “But it is concerning that minority children are increasing their consumption of 100 percent fruit juice, which often has just as much sugar as soda.”…Fruit juice is available in schools through the National School Lunch Program, which provided low-cost or free lunches to more than 31 million children nationally each school day in 2011. Juice also is distributed to parents through the Women, Infants and Children (WIC) program, a federal grant program for low-income, nutritionally at-risk women and children up to age 5 and through the Child Care Food Program.
Seeing solutions with “Ecological Lenses”
• At the individual level: Nutrition information (for parent or child)
• At the family level: Nutrition education for parent and child
• At the community level: a media campaign targeting families
• At the organizational level: commit to soda AND juice free zones in community centers, provide access to fruit and clean water
• At the policy level: change WIC policy to provide whole fruit and water filters, eliminate Juice vouchers for WIC
OREliminate fruit juice from subsidized school lunches; increase fruit and provide filling stations and water bottles in schools
Rationale for Doing a Formal Problem Analysis?
In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one another
Using a multilevel socio-ecological framework ensures that upstream factors are included
Relating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation
Steps in a Problem Analysis
1. Examine epidemiologic data
2. Examine literature and consult experts (community and science)
3. Determine extent to which these factors are active in the community
Steps in a Problem Analysis
4. Determine relative contribution of each identified factor
5. Identify the interrelationships among factors – causal pathways
6. Determine the most effective points in the causal pathways for intervention
SES
Social/Economic/Policy Level
Environment
Safety
A Generic Framework for Health Problem Analysis
Education
Economy Health Care Policies
Family/Community/Institutions Level
Family/Household
Community
School/Workplace
Health Care/Providers
Individual Level
Genetic/Biological Psychological Factors Health Status/Medical Conditions
Cognitive Factors Health Behaviors
Identified Problem
Culture
PROBLEM:
Family/community
SES/Policy
Individual
Unavailability of healthy food choices in local stores , schools and restaurants
Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits)
Families limited income
Unhealthy diet
Inadequate subsidized healthInsurance for the poor
Poverty City planning Policies that limit green space in some cities and neighborhoods
Childhood Obesity
Health serviceslack of knowledge of nutritionHigh costs for treatment
No opportunities for exercise in school or recreation areas
Insufficiency of physical activity
Unsafe neighborhood/streetsMarketing by fast food industry
Family lack of health Insurance
Lack of funding for education
Too much screen time (TV computer)
Linkages
Definition:
The association betweenprecursors and problem
Review the literature Consult experts
Community Science
Consult stakeholders Analyze your data
PROBLEM:
Family/community
SES/Policy
Individual
Unavailability of healthy food choices in local stores , schools and restaurants
Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits)
Families limited income
Unhealthy diet
Inadequate subsidized healthInsurance for the poor
Poverty City planning Policies that limit green space in some cities and neighborhoods
Childhood Obesity
Health serviceslack of knowledge of nutritionHigh costs for treatment
No opportunities for exercise in school or recreation areas
Insufficient physical activity
Unsafe neighborhood/streetsMarketing by fast food industry
Family lack of health Insurance
Lack of funding for education
Too much screen time (TV computer)
Diagramming Causal Pathways to Identify Strategic Interventions
Unsafe neighborhood
streets
No opportunities for exercise in school or recreation areas
Too much screen time (TV computer)
Inadequate physical activity
Overweight
Intervention
PROBLEM:
Family/Community/Local Institutions
Social Determinants
Individuals
Schools push kids out/racism in staff/poor quality/lack of cultural affinity
Barriers to accessing services:
Language/Cultural/ $Transportation/Legal
Mental illness/PTSD/ self esteem/ resiliency/ trust/ sense of belonging
Racism/Destruction of indigenous culture
Economic policy Poverty / no jobs or bias in hiring
crime/gun policies/ prevention $
Gang Membership/Incarceration
Community-lack of social capital/ cohesion/pride/trust
Alcohol/drug useUnsafe sex/ bad food/no exercise
$ for schools/low standards/legacy of colonialism
Problem Analysis – Homey Clients
Family disintegrationParents don’t care/use drug/don’t supervise/not educated/poor/lack cultural pride
FAS/ADD/ADHD/other drug exposure Learning difficulties/brain
trauma/school failure/no college prep
Jobs/lack of training opportunities
Police racial profiling
HOMEY- Kapuli Causal Pathway: Program Intervention Points
Society:RacismLack of respect for indigenous cultures
Communities:Racism in schoolsRacism by policeLack of cultural
identity in family/community
family disintegration
Kids: Mental health problemsSchool failureLow self esteem lack resiliencyNo sense of true cultural identity No sense of trust or belonging
Gangs/Jail
School failure
Self- Destructive Behaviors
Kapuli Interventions
Learning about history of social justice movements will give them tools for change
Learning about impact of racism/colonialism will give kids an undestanding of their own history
Episodes of tragedy due to racism in their community
Media/movie about examples of racism/harassment by police
ENVIRONMENTAL FACTORS
Street outreach
Educate about history of social justice movements for people of color/indigenous
10 strategies for change
Teach about racism/colonialism
Positive response during recap activity
Active participation in group
Successful school/CBO outreach
Schools
CBO’s
Kids from streets
Come back
Join political campaigns
Attend ceremonies
Express value of social action
Increase self esteem/resiliency/capacity for action
Kids return to school
Kids get jobs
Kids stay out of gangs
Kids stay out of jail
StaffCameronSpaceCore MembersFunds
INPUTS OUTPUTS OUTCOMES – IMPACTActivities Participatio
nShort Medium Long-Term
ASSUMPTIONS:
Adapted from
MOMEY-Kapuli Logic Model
Another Example: Obesity prevention/ intervention in the Bayview Hunters Point of SF
Designed in 2011, in partnership between:• UCSF • San Francisco Department of Public health• BVHP residents• other stakeholders
See handouts for problem analysis and logic models
As of 2013, focus is on:• Reducing consumption of sugar sweetened beverages• Activating community park space
Benefits
Clearly documents the decision making process for the group and for others
Can use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public
Logic Model - Defined
Logic models are a common tool used by evaluators and program planners
Logic models are graphic depictions of the relationship between a program’s activities and its intended outcomes.
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.
Logic Model – Terms Relationship
• Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes.
Intended• Logic models depict “intended” outcomes of a program’s
activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed . Over time, evaluation, research, and experience will help us learn what works, and the model will evolve.
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.
Logic Model - Components
Inputs • Resources that go into the program and are required to
make it happen Activities
• events or actions done by the program and its staff Outputs
• Direct products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants).
Outcomes • The changes that result from the program’s activities
and outputs, often expressed as short-term, intermediate, and long-term outcomes.
Basic Program Logic Model
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.
Let’s build a logic model!
Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest)
What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program does
What resources do you need (inputs)?
Q and A
What was clarified for you; any “aha!” moments?
What still concerns or challenges you?
Contact Information
Community Engagement and Health Policy Program, CTSI/UCSF
415-206-4048
www.sfhip.org