everyday program evaluation
DESCRIPTION
Everyday Program Evaluation. Sheena Cretella MSPH SC DHEC Diabetes Division Program Evaluator . Today’s Objectives. Planning Programs Logic Models Evaluation Standards of Evaluation Steps to Evaluation Indicators Moderators Success Stories Data Types Collection Management. - PowerPoint PPT PresentationTRANSCRIPT
1
Everyday Program Evaluation
Sheena Cretella MSPHSC DHEC Diabetes Division
Program Evaluator
2
Today’s Objectives• Planning Programs
– Logic Models• Evaluation
– Standards of Evaluation– Steps to Evaluation– Indicators– Moderators– Success Stories
• Data– Types– Collection– Management
3
PLAN Like Your Program Depends On It!
A Logic Model is like a road map Plan your route:
STARTwhere you are
now
FINISHwhere you intend
to be after the program
HOW DO WE GET FROM START TO FINISH?
4
Logic Model Pieces
Inputs andResources
• What do you have?
Activities
• What do you plan to do?
Outputs
• What targets do you have for your activities?
Outcomes
• What changes resulted from your activities?
5
Logic Model
Resources
Grant Funding
Volunteers
Church Facilities
Activities
Construct Walking
Trail
Conduct Sessions
Create Community
Garden
Outputs
Number of Miles
Walked
Number of Attendees
Number of Seed Hills
Short Term
Outcomes
Improved Attitude of
Physical Activity
Increase in knowledge of Healthy Behaviors
Obtain Horticulture
Skills
Intermediate Outcomes
Creation of an environment supporting
healthy behaviors
Increase in Physical Activity
Increase in Fruit and Vegetable
Consumption
Long Term Outcomes
Healthier Community
Reduce the Risk of Pre-
diabetes and Diabetes and
Prevent Complications
of Diabetes
6
Logic Model
Resources
Grant Funding
Volunteers
Church Facilities
Activities
Construct Walking
Trail
Conduct Sessions
Community Garden Created
Outputs
Number of Miles
Walked
Number of Attendees
Number of Seed Hills
Short Term
Outcomes
Improved Attitude of
Physical Activity
Increase in knowledge of Healthy Behaviors
Obtain Horticulture
Skills
Intermediate Outcomes
Creation of an environment supporting
healthy behaviors
Increase in Physical Activity
Increase in Fruit and Vegetable
Consumption
Long Term
Outcomes
Healthier community
Reduce the Risk of Pre-
diabetes and Diabetes and
Prevent Complications
of Diabetes
7
Program Planning
8
STOP: Check List for Program Planning
Do you and the stakeholders know what is going to be done, with whom, to whom/what?
Is the goal statement outcome oriented?
Are the expectations of stakeholders known to all?
Can the pieces of the logic model be measured?
Are there missing pieces or “gaps” in the logic model/plan?
Is the impact of the program important? Is there evidence?
10
Keep In Mind
• Key Questions Grantees Need to Answer About their Programs:– What role, if any, did my program play in the results?– What role, if any, did the moderators play?– Were there any unintended outcomes?– What will happen if I do not do something?
12
The Steps to Good Program Evaluation
1 •Engage Stakeholders
2 •Describe the Program
3 •Focus the Evaluation
4 •Gather Credible Evidence
5 •Justify Conclusions
6 •Use Lessons Learned
13
Moderators
Political Economical
Social Technological
14
Logic Model
Resources
Grant Funding
Volunteers
Church Facilities
Activities
Construct Walking
Trail
Conduct Sessions
Community Garden Created
Outputs
Number of Miles Walked
Number of
Attendees
Number of Seed
Hills
Short Term Outcomes
Improved Attitude of
Physical Activity
Increase in knowledge of Healthy Behaviors
Obtain Horticulture
Skills
Intermediate Outcomes
Creation of an
environment
supporting healthy behaviorsIncrease
in Physical ActivityIncrease in Fruit
and Vegetable Consump
tion
Long Term Outcomes
Healthier commun
ity
Reduce the Risk of Pre-
diabetes and
Diabetes and
Prevent Complicatio
ns of Diabetes
Moderators: Weather (drought, heat advisory, snow), safety (snakes/dogs), cost of gas (less volunteers/low attendance), neighborhood safety, inexperienced personnel, local habitat policies.
15
Measures
Each activity and outcome should have a measure.
A good measure is: Specific Observable Measurable
When choosing measures consider the following:• Quality of data• Quantity of data• Logistics • Sources of data collection• Primary vs. Secondary
16
Data Management
17
Data Sources
Primary• Collecting new data• Data is from the people you
work with• Common forms:
– Group discussions– Observation– Document review: logs,
journals, meeting minutes, sign-in sheets, etc.
– Surveys: telephone, e-mail, personal, etc.
Secondary• Existing data sources• Before using Secondary
data ensure that they will meet the evaluation needs
• Large ongoing surveillance systems
• Routinely collected• Not flexible
18
Data Sources
• Program participants and non-participants• Key members• Program staff
Who would you interview?
• Meetings• Special events / activities
What might you observe?
• Meeting minutes• Strategic plans• Registration form• Photos
Which Documents might you review?
19
Success Story•Catchy•Action verbTitle•Importance of the problem•EvidenceIssue•Describe program •Partner SupportProgram•Point out the most important measured outcomes/benefits of the programImpact
20
Success Story
• Content– Contact Information– Photos and Logos– Quotes
• Format– Brief– One Page– Bullets
• Possible Inclusions:– Testimonials– Promising practices– Lessons Learned– Partner Success– New Partners
Make it your own!
21
Issue
Contact
Impact
Title
Program
Photo
22
Types of data• Make sure you are choosing the right kind of
data for your specific evaluation questions.
• Qualitative• Quantitative– Continuous– Categorical• Ordinal• Nominal
23
Data Management
• Physical Records– De-identify records whenever possible– Store records in a safe locked place– Shred personal data upon completion of evaluation
• Electronic Records– Use unique identification numbers whenever possible– Use a password on your computer– Erase (not delete) personal data upon completion of evaluation.– Back up data.
24
Data Management
Id# Age County Variable 3 Variable 4
00100 23 Richland
00101 34 Fairfield
00102 45 Orangeburg
00103 34 Lexington
Id# Age County Variable 3 Variable 4
00100 23 Richland
00101 34 Fairfield
00102 45 Orangeburg
00103 34 Lexington
Id# Age County Variable 3 Variable 4
00100 23 Richland
00101 34 Fairfield
00102 45 Orangeburg
00103 34 Lexington
25
Everyday Program Evaluation
Thank You!
Sheena Cretella MSPHSC DHEC Diabetes Division
Program Evaluator [email protected]
803-545-4488