Welcome to Program Evaluation
Overview of Evaluation Concepts
Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlikeLicense.
Introduction
• Objectives of course, • What is Program Evaluation• Types of evaluation (general)• Example
Objectives of course
• By the end of the course, you will be able to:• (1) Explain major concepts in program
evaluation:– Type of evaluation and their purpose– Levels of measurement– Sources of data– Study designs, threats to validity
Objectives of course
• (2) Perform skills required in program evaluation:– Design a conceptual framework– Develop objectives, indicators– Conduct a focus group– Pretest a survey– Process statistics– Use participatory evaluation techniques
• (3) Write an evaluation plan
What’s not covered?
• How to do multivariate analysis required to analyze the designs discussed in class (e.g., randomized trial)
• Calculations of sample size, power• Surveillance techniques• Additional qualitative methods• Actual implementation - ECUR 810
Tell me more Jay.
Overview of Program Evaluation
What is program evaluation?
“the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.”
(Wiess, 2009)
Which really means…
Telling some group how well the program is doing what it is supposed to be doing.
“There is room for judgment.”Brian Noonan
Why do we do program evaluation?
• To determine the effectiveness of a program:– Did it achieve its objectives?– Were effects similar across subgroups?
• To identify ways of improving on existing program design.
• Continue or terminate a program.• To satisfy financial considerations, requirements. • For ‘political’ reasons, PR, staff morale,
communication.
Initially you need to think about and ask:
• What is the problem?• Why am I doing this?• What exactly do I want to know?• Does an answer already exist?• How do I find out?• Who is involved?• What will this cost?• What will I do with the data?• What happens when the evaluation is finished?
Tailoring evaluation to a specific context
• What questions will the evaluation answer?• What methods/procedures will be used?• What will be the evaluator-stakeholder
relationship? – Independent evaluation– Participatory/collaborative evaluation– Empowerment evaluation
• What do you do with the answers?
Other key factors: program structure and circumstances
• Stage of program development• Political context (conflict over goals)• Structure of the program– Scope of activities, type of services – Number and location of service sites– Characteristics of intended audience
• Resources available: – Human, $$$, support of admin or management
Challenges in program evaluation
• Dealing with the ‘politics’ of programs• Having the program design change mid-course• Balancing the tensions between scientific
soundness and practicality (utility for decision-makers)
• Obtaining $$$ and support for strong designs
Primary types of evaluation
• Formative– Needs assessment/diagnostic– Pretesting
• Process– Implemented well? As planned?– Input analysis– Output, effectiveness, user satisfaction
• Summative– Monitoring of program utilization– Monitoring of behavior or health status– Impact/outcome assessment
• Cost effectiveness, cost-benefit• Benchmarks, standards
Characteristics of PE(Powell, 2006)
• Used for decision making.• Deals with research questions about a program.• Takes place in the real world of the program.• Usually represents a compromise between pure and
applied research.• Should be carefully planned.• Have a purpose not an end in itself.• Must be hope for action or change.• Can be used to raise internal as well as external
awareness
Pallet Cleanser
An example…
• Two ways a programs can be studied.• Two ways to determine what is working or not
working with a program.
Implementation failure vs. Theory failure
• Implementation failure– Program is not implemented as planned
• Theory failure– Program is implemented as planned– Intervention does not produce intermediate
results and/or – They do not produce desired outcome
Measures of outcome (Rossi, Lipsey, Freeman, 2004)
• Initial• Intermediate• Long-term
Example: mass media program to reduce HIV transmission
• Initial– Antecedents to behavior– Ex: knowledge, attitudes, practice
• Intermediate– Behaviors– Ex: contraceptive or condom use
• Long-term– Health status– Ex: fertility rates, HIV prevalence
Example: conceptual model of program impact
Implementation failure
Theory failure
Theory failure
What????