Program Evaluation How to Effectively Evaluate

Your Program

Carol Pilcher Department of Entomology

Iowa State University

Logic ModelUniversity of Wisconsin-Extension

Purpose of Logic Model

• Visual display to connect inputs, outputs and outcomes

“Helps people get on the same page”“Describes a program and why it works”“Roadmap to help you know where you

are going”

Logic Model

Inputs Outputs Outcomes-Impact

Situation

Assumptions External Factors

Priorities

What We

Invest

What We Do

Who We

Reach

What the

Short Term

Results Are

What the

Medium Term

Results Are

What the

Ultimate Impact

Is

Adapted From University of Wisconsin-Extension

Situation

• What is the problem or issue?– Background information, grant

proposal

• Why is this a problem?– Background information, grant

proposal

• Who has a stake in the problem?– Identify stakeholders

Priorities

• Determine the focus of your program– Establish priorities based on:

• Resources• Experience• Situation

– Objectives and Timeline, grant proposal

Logic Model

Inputs Outputs Outcomes-Impact

SITUATION

Assumptions External Factors

PRIORITIES

What We

Invest

What We Do

Who We

Reach

What the

Short Term

Results Are

What the

Medium Term

Results Are

What the

Ultimate Impact

Is

Inputs

• What resources will you invest?– Staff

• Number of staff and hours available

– Time• Hours reported

– Money• Allocated funds

Logic Model

Inputs Outputs Outcomes-Impact

SITUATION

Assumptions External Factors

PRIORITIES

What We

Invest

What We Do

Who We

Reach

What the

Short Term

Results Are

What the

Medium Term

Results Are

What the

Ultimate Impact

Is

Outputs-Activities

• What will you do?– Conduct workshops, meetings– Deliver services– Develop products, resources

Outputs-Activities

• Examples

Thirty-nine food safety programs were held including hand-washing demonstrations and food safety during the RAGBRAI intra-state cycling trek

Starting with a pilot of 4 districts during the 2001-2002 school year, the School Integrated Pest Management Program has trained more than 30 districts statewide.

Output-Participation

• Who will you reach?– Participants– Clients– Agencies– Decision makers– Customers

Output-Participation

• Examples

Attendance at Crop Advantage Meetings2002—933 2003—720 2004—1735

5,982 individuals were involved in parent education programs

Output-Reactions

• Reactions about course materials• Reactions about program

Output-Reactions

• Example

Of the participants that completed the survey (n=120), 98% indicated that the information presented was useful

for their farm operation

Strongly Agree

Agree Undecided Disagree Strongly Disagree

The information presented today was useful for my farm operation.

___ ___ ___ ___ ___

Logic Model

Inputs Outputs Outcomes-Impact

SITUATION

Assumptions External Factors

PRIORITIES

What We

Invest

What We Do

Who We

Reach

What the

Short Term

Results Are

What the

Medium Term

Results Are

What the

Ultimate Impact

Is

Outcomes-ImpactsShort Term Results

• Amount of learning– How much was learned during the

activity?– KASA

• Knowledge, Attitudes, Skills, Awareness

Outcomes-ImpactsKASA

• Example

Of those who returned evaluation surveys, 78% reported moderate to significantly improved understanding of the

factors affecting groundwater and surface water contamination of pesticides.

Please circle the best response for the following questions

Low Level

1

2 3 4 High Level

5

Prior to today’s program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides

1 2 3 4 5

After today’s program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides

1 2 3 4 5

Outcomes-ImpactsMedium Term Results

• Changes in actions– Behavioral changes– Changes in practices– Changes in decision making

• ExampleYear One

As a result of today’s training, will you use a drift-reduction nozzle in your sprayer to assist in reducing drift

–Adopted Prior To Training, –Plan to Adopt, –Choose Not to Adopt, –Does Not Apply

Year Two

As a result of last year’s program, do you use a drift-reduction nozzle in your sprayer to assist in reducing drift

–I Adopted –I Did Not Adopt –Does Not Apply

Outcomes-ImpactsChange in Actions

Outcomes-ImpactsChange in Actions

• Examples

As a result of the program, 50% of the participants said they would use drift reduction nozzles in their sprayers to reduce drift.

This post-training evaluation examined if participants had indeed successfully implemented new pesticide safety activities. According to the respondents, 85% of the participants said that they now use drift reduction nozzles in their sprayers.

Ninety-eight percent of the food safety program participants reported adopting one of more of the recommended food handling practices.

Outcomes-ImpactsUltimate Impacts

• Changes in conditions– Social/Health– Environment– Economic

Outcomes-ImpactsChanges in economic conditions

• Example

Approximately, 4.3 million acres were managed with the IPM practices of improved disease resistant varieties, use of safe and effective fungicides as recommended by a forecasting model, and use of the wheat midge risk map and midge scouting. A total of $105.7 million was gained by producers using these IPM practices in 2003.

Outcomes-ImpactsUltimate Impacts

IPM Focus Areas

Production Agriculture

Residential and Public Areas

Natural Resources and Recreational Environments

IPMImpactAreas

Environmental Impacts (Reduce environmental

risk)

Health Impacts(Reduce risks to health)

Economic Impacts(Cost/Benefit)

Logic Model

Inputs Outputs Outcomes-Impact

SITUATION

Assumptions External Factors

PRIORITIES

What We

Invest

What We Do

Who We

Reach

What the

Short Term

Results Are

What the

Medium Term

Results Are

What the

Ultimate Impact

Is

Evaluation

Answers the “so what” question.

What difference does your program make for the participants?

Determines the worth of a program.