program evaluation the use of scientific methods to judge and improve the planning, monitoring,...

26
Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and other human service programs

Upload: madeline-martin

Post on 04-Jan-2016

217 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Program Evaluation

The use of scientific methods to judge and improve the planning, monitoring,

effectiveness, and efficiency of health, nutrition, and other human service programs

Page 2: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Why Evaluate a Program?

• See table 10-1, page 309, Boyles and Morris

Page 3: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Types of Program Evaluation

• Process evaluation

• Impact or outcome evaluation

• Fiscal or efficiency evaluation

Page 4: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Process Evaluation

• Evaluate process objectives

• Provides information for why program may or may not have reached its outcome objectives

• If program is delivered from a variety of sites, provides information on why some sites may have been more successful than others

Page 5: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Six Steps for Program Evaluation

• 1. Determine objectives of program

• Evaluate for:

• a. Appropriateness of objectives

• b. Effectiveness in meeting objectives

• c. Efficiency of program

• d. Side effects of program

Page 6: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Steps in Program Evaluation

• 2. Determine characteristics to be measured

• Measurements should be:

• Valid

• Reliable

• Precise

Page 7: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Steps in Program Evaluation

• 3. Measure characteristics

Page 8: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Steps in Program Evaluation

• 4. Make comparisons

• May use:

• Control groups

• Similar groups

• Standards

• Pre vs post measurements

Page 9: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Steps in Program Evaluation

• 5. Draw conclusions

• 6. Make recommendations

Page 10: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Common Biases Introduced During Evaluations

• Selection

• Testing

• History

• Maturation

• Halo effect

Page 11: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Evaluation Design

• 1. Experimental design

• 2. Quasi-experimental design

• 3. Non-experimental design

Page 12: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Steps for experimental design

• 1. Experimental and control groups randomly assigned

• 2. Each group measured

• 3. Intervention or program provided

• 4. Groups measured again--if experimental group improved more than control, program was successful

Page 13: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Examples of Designs of True Experiments

• Pre-test post-test control group design

• R O X O

• R O O

Page 14: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Examples of Designs of True Experiments

• After only control group

• R XO

• R O

Page 15: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Examples of Experimental Design

• Solomon 4 group

• R O X O

• R O O

• R X O

• R O

Page 16: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Quasi-experimental design

• Steps similar to experimental, but rigid control not met.

• Random selection may not be done

• Subjects may be volunteers

• Nonequivalent control groups may be used

Page 17: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Nonexperimental design

• Random selection not used

• No control group or nonequivalent control group used

Page 18: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Examples of Non-experimental design

• After only or one-shot case study

• X O

• Nonequivalent control group study

• X O

• O

• Pre-test-Post-test design

• O X O

Page 19: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Fiscal or Efficiency Evaluations

• Cost-benefit analysis

• Cost-effectiveness analysis

Page 20: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Cost benefit analysis

• Decision making framework used in allocating resources among competing uses.

• Both costs and benefits are expressed in dollars

Page 21: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Costs

• Direct Costs --Cash expenditures

• Indirect Costs

– All other costs such as

– Spillover effects

– Costs to client

– Costs to organization not covered by program

• Opportunity costs

• Intangible costs--grief, suffering pain

Page 22: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Benefits

• All costs that would be avoided if the program were in effect

• Direct benefits--values of resources which the program saves

• Negative benefits

• Indirect benefits--other costs averted

• Intangible benefits--happiness, bonding from breastfeeding

Page 23: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Discount rate

• Based on deferred benefits

Page 24: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Cost-Benefit Analysis of Attending School

• Direct costs

• Indirect costs

• Intangible costs

• Direct benefits

• Indirect benefits

• Intangible benefits

• Discount rates

Page 25: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Cost effectiveness analysis

• Determines the most efficient way of meeting a predetermined set of objectives

• Costs measured in dollars

• Effectiveness measured by outcomes, e.g.. lives saved, increase in birth weight, etc

Page 26: Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and

Communicating Evaluation Results

• See pages 322-326