everyday program evaluation

1

Everyday Program Evaluation

Sheena Cretella MSPHSC DHEC Diabetes Division

Program Evaluator

2

Today’s Objectives• Planning Programs

– Logic Models• Evaluation

– Standards of Evaluation– Steps to Evaluation– Indicators– Moderators– Success Stories

• Data– Types– Collection– Management

3

PLAN Like Your Program Depends On It!

A Logic Model is like a road map Plan your route:

STARTwhere you are

now

FINISHwhere you intend

to be after the program

HOW DO WE GET FROM START TO FINISH?

4

Logic Model Pieces

Inputs andResources

• What do you have?

Activities

• What do you plan to do?

Outputs

• What targets do you have for your activities?

Outcomes

• What changes resulted from your activities?

5

Logic Model

Resources

Grant Funding

Volunteers

Church Facilities

Activities

Construct Walking

Trail

Conduct Sessions

Create Community

Garden

Outputs

Number of Miles

Walked

Number of Attendees

Number of Seed Hills

Short Term

Outcomes

Improved Attitude of

Physical Activity

Increase in knowledge of Healthy Behaviors

Obtain Horticulture

Skills

Intermediate Outcomes

Creation of an environment supporting

healthy behaviors

Increase in Physical Activity

Increase in Fruit and Vegetable

Consumption

Long Term Outcomes

Healthier Community

Reduce the Risk of Pre-

diabetes and Diabetes and

Prevent Complications

of Diabetes

6

Logic Model

Resources

Grant Funding

Volunteers

Church Facilities

Activities

Construct Walking

Trail

Conduct Sessions

Community Garden Created

Outputs

Number of Miles

Walked

Number of Attendees

Number of Seed Hills

Short Term

Outcomes

Improved Attitude of

Physical Activity

Increase in knowledge of Healthy Behaviors

Obtain Horticulture

Skills

Intermediate Outcomes

Creation of an environment supporting

healthy behaviors

Increase in Physical Activity

Increase in Fruit and Vegetable

Consumption

Long Term

Outcomes

Healthier community

Reduce the Risk of Pre-

diabetes and Diabetes and

Prevent Complications

of Diabetes

7

Program Planning

8

STOP: Check List for Program Planning

Do you and the stakeholders know what is going to be done, with whom, to whom/what?

Is the goal statement outcome oriented?

Are the expectations of stakeholders known to all?

Can the pieces of the logic model be measured?

Are there missing pieces or “gaps” in the logic model/plan?

Is the impact of the program important? Is there evidence?

10

Keep In Mind

• Key Questions Grantees Need to Answer About their Programs:– What role, if any, did my program play in the results?– What role, if any, did the moderators play?– Were there any unintended outcomes?– What will happen if I do not do something?

12

The Steps to Good Program Evaluation

1 •Engage Stakeholders

2 •Describe the Program

3 •Focus the Evaluation

4 •Gather Credible Evidence

5 •Justify Conclusions

6 •Use Lessons Learned

13

Moderators

Political Economical

Social Technological

14

Logic Model

Resources

Grant Funding

Volunteers

Church Facilities

Activities

Construct Walking

Trail

Conduct Sessions

Community Garden Created

Outputs

Number of Miles Walked

Number of

Attendees

Number of Seed

Hills

Short Term Outcomes

Improved Attitude of

Physical Activity

Increase in knowledge of Healthy Behaviors

Obtain Horticulture

Skills

Intermediate Outcomes

Creation of an

environment

supporting healthy behaviorsIncrease

in Physical ActivityIncrease in Fruit

and Vegetable Consump

tion

Long Term Outcomes

Healthier commun

ity

Reduce the Risk of Pre-

diabetes and

Diabetes and

Prevent Complicatio

ns of Diabetes

Moderators: Weather (drought, heat advisory, snow), safety (snakes/dogs), cost of gas (less volunteers/low attendance), neighborhood safety, inexperienced personnel, local habitat policies.

15

Measures

Each activity and outcome should have a measure.

A good measure is: Specific Observable Measurable

When choosing measures consider the following:• Quality of data• Quantity of data• Logistics • Sources of data collection• Primary vs. Secondary

16

Data Management

17

Data Sources

Primary• Collecting new data• Data is from the people you

work with• Common forms:

– Group discussions– Observation– Document review: logs,

journals, meeting minutes, sign-in sheets, etc.

– Surveys: telephone, e-mail, personal, etc.

Secondary• Existing data sources• Before using Secondary

data ensure that they will meet the evaluation needs

• Large ongoing surveillance systems

• Routinely collected• Not flexible

18

Data Sources

• Program participants and non-participants• Key members• Program staff

Who would you interview?

• Meetings• Special events / activities

What might you observe?

• Meeting minutes• Strategic plans• Registration form• Photos

Which Documents might you review?

19

Success Story•Catchy•Action verbTitle•Importance of the problem•EvidenceIssue•Describe program •Partner SupportProgram•Point out the most important measured outcomes/benefits of the programImpact

20

Success Story

• Content– Contact Information– Photos and Logos– Quotes

• Format– Brief– One Page– Bullets

• Possible Inclusions:– Testimonials– Promising practices– Lessons Learned– Partner Success– New Partners

Make it your own!

21

Issue

Contact

Impact

Title

Program

Photo

22

Types of data• Make sure you are choosing the right kind of

data for your specific evaluation questions.

• Qualitative• Quantitative– Continuous– Categorical• Ordinal• Nominal

23

Data Management

• Physical Records– De-identify records whenever possible– Store records in a safe locked place– Shred personal data upon completion of evaluation

• Electronic Records– Use unique identification numbers whenever possible– Use a password on your computer– Erase (not delete) personal data upon completion of evaluation.– Back up data.

24

Data Management

Id# Age County Variable 3 Variable 4

00100 23 Richland

00101 34 Fairfield

00102 45 Orangeburg

00103 34 Lexington

Id# Age County Variable 3 Variable 4

00100 23 Richland

00101 34 Fairfield

00102 45 Orangeburg

00103 34 Lexington

Id# Age County Variable 3 Variable 4

00100 23 Richland

00101 34 Fairfield

00102 45 Orangeburg

00103 34 Lexington

25

Everyday Program Evaluation

Thank You!

Sheena Cretella MSPHSC DHEC Diabetes Division

Program Evaluator cretels@dhec.sc.gov

803-545-4488