step 3: focus evaluation design luann c. d’ambrosio, med

Step 3: Focus Evaluation Design

Luann C. D’Ambrosio, MEd

• Understand how to focus the scope of your evaluation

• Describe the difference between process and outcome aims

• Develop questions to ask for gathering information

• Understand how to develop standards and indicators to measure progress

• Develop an evaluation plan for your own program(s)

Objectives

CDC Evaluation Framework

When Designing Your Evaluation, Consider

Plan Focus

Process

(means, resources/activities/outputs on

your logic model)

The number of inspections

conducted by food inspection staff .

Process

(means, resources/activities/outputs on

your logic model)

The number of inspections

conducted by food inspection staff .

Outcome

(ends, outcomes/goal on your logic model)

Reduction in critical item food safety

violations.

Outcome

(ends, outcomes/goal on your logic model)

Reduction in critical item food safety

violations.

How Do I Know What Questions to Ask?

• Preventing disease• Reducing health risk• Promoting positive health

behaviors• Compiling with legal

mandates• Responding to public

complaints• Addressing current impact/

threats• Anticipating future

impact/threats

How Do I Know What Questions to Ask?

• Maximizing use of resources• Demonstrating effective

public relations• Ensuring program

sustainability• Maintaining program control• Improving quality• Managing costs• Effectively communicating

with the public

Outcome or Process?

Hint: Ask the “Why?” question

Preventing Disease

Effectively communicating with the public

Promoting positive health behaviors

Outcome Process

Using Questions to Develop Standards

Use your questions to describe what should exist in order for the program to operating at an optimum level – the “bar.”

Process Question Example

Standard: Each EH specialist should be inspecting 100 food service establishments per month.

Outcome Question Example

Standard: The average number of critical item violations for food service establishments should be no greater than 2.5.

Finding Standards

• Your stakeholders

• National, state, local standards

• Program standards

• Other jurisdictions

• Existing

• Identify a process question, then turn it into a process standard

• Identify an outcome question, then turn it into an outcome standard

Group Practice

How Do I Measure Differences?

Indicators

Ask “What information do I need to determine if my standard is being met?

Example

Standard: Each EH Specialist should be inspecting 100 food service establishments per month.

Indicators:

•# inspections performed per month

•# FTE’s in food program conducting inspections

•Average time to conduct an inspection

•# of food service establishments in jurisdiction

Group Practice

Home grown tomatoes to meet your family’s needs

1. Identify family’s needs (standard)

2. Develop an Outcome and Process standard

3. Outcome and Process Indicators

Muf

fet

Group Practice

Are we able to grow tomato plants for our salad, canning, and juice needs this year?Are we able to grow tomato plants for our salad, canning, and juice needs this year?

QuestionQuestion

Raise 8 tomato plants to maturity by Sept 15. Raise 8 tomato plants to maturity by Sept 15.

Process standardProcess standard

• # plants reaching maturity by Sept 15

• # plants producing green tomatoes by July 15

• # times fertilized per month

• # times watered per week

• # times plants pruned

• # plants reaching maturity by Sept 15

• # plants producing green tomatoes by July 15

• # times fertilized per month

• # times watered per week

• # times plants pruned

IndicatorsIndicators

Group Practice

Are we able to produce sufficient, eatable tomatoes for our salad, canning, and juice needs this year?Are we able to produce sufficient, eatable tomatoes for our salad, canning, and juice needs this year?

QuestionQuestion

50 lbs of ripe, edible tomatoes harvested by Sept 1550 lbs of ripe, edible tomatoes harvested by Sept 15Outcome standardOutcome standard

• # of ripe, eatable tomatoes per plant

• Pounds of eatable tomatoes picked by Aug 30

• Pounds of eatable tomatoes picked by Sept 15

• Pounds of tomatoes discarded by Sept 15

• # of jars of tomatoes canned by Sept 15

• # jars of homemade tomato juice available by Sept 15

• # of salads containing tomatoes consumed by Sept 15

• # of ripe, eatable tomatoes per plant

• Pounds of eatable tomatoes picked by Aug 30

• Pounds of eatable tomatoes picked by Sept 15

• Pounds of tomatoes discarded by Sept 15

• # of jars of tomatoes canned by Sept 15

• # jars of homemade tomato juice available by Sept 15

• # of salads containing tomatoes consumed by Sept 15

IndicatorsIndicators

Important Terms to Remember

• Process

• Outcome

• Standards

• Indicators

Questions?