welcome to program evaluation overview of evaluation concepts copyright 2006, the johns hopkins...
DESCRIPTION
Objectives of course By the end of the course, you will be able to: (1) Explain major concepts in program evaluation: – Type of evaluation and their purpose – Levels of measurement – Sources of data – Study designs, threats to validityTRANSCRIPT
![Page 1: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/1.jpg)
Welcome to Program Evaluation
Overview of Evaluation Concepts
Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlikeLicense.
![Page 2: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/2.jpg)
Introduction
• Objectives of course, • What is Program Evaluation• Types of evaluation (general)• Example
![Page 3: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/3.jpg)
Objectives of course
• By the end of the course, you will be able to:• (1) Explain major concepts in program
evaluation:– Type of evaluation and their purpose– Levels of measurement– Sources of data– Study designs, threats to validity
![Page 4: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/4.jpg)
Objectives of course
• (2) Perform skills required in program evaluation:– Design a conceptual framework– Develop objectives, indicators– Conduct a focus group– Pretest a survey– Process statistics– Use participatory evaluation techniques
• (3) Write an evaluation plan
![Page 5: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/5.jpg)
What’s not covered?
• How to do multivariate analysis required to analyze the designs discussed in class (e.g., randomized trial)
• Calculations of sample size, power• Surveillance techniques• Additional qualitative methods• Actual implementation - ECUR 810
![Page 6: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/6.jpg)
Tell me more Jay.
![Page 7: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/7.jpg)
Overview of Program Evaluation
![Page 8: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/8.jpg)
What is program evaluation?
“the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.”
(Wiess, 2009)
![Page 9: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/9.jpg)
Which really means…
Telling some group how well the program is doing what it is supposed to be doing.
“There is room for judgment.”Brian Noonan
![Page 10: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/10.jpg)
Why do we do program evaluation?
• To determine the effectiveness of a program:– Did it achieve its objectives?– Were effects similar across subgroups?
• To identify ways of improving on existing program design.
• Continue or terminate a program.• To satisfy financial considerations, requirements. • For ‘political’ reasons, PR, staff morale,
communication.
![Page 11: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/11.jpg)
Initially you need to think about and ask:
• What is the problem?• Why am I doing this?• What exactly do I want to know?• Does an answer already exist?• How do I find out?• Who is involved?• What will this cost?• What will I do with the data?• What happens when the evaluation is finished?
![Page 12: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/12.jpg)
Tailoring evaluation to a specific context
• What questions will the evaluation answer?• What methods/procedures will be used?• What will be the evaluator-stakeholder
relationship? – Independent evaluation– Participatory/collaborative evaluation– Empowerment evaluation
• What do you do with the answers?
![Page 13: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/13.jpg)
Other key factors: program structure and circumstances
• Stage of program development• Political context (conflict over goals)• Structure of the program– Scope of activities, type of services – Number and location of service sites– Characteristics of intended audience
• Resources available: – Human, $$$, support of admin or management
![Page 14: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/14.jpg)
Challenges in program evaluation
• Dealing with the ‘politics’ of programs• Having the program design change mid-course• Balancing the tensions between scientific
soundness and practicality (utility for decision-makers)
• Obtaining $$$ and support for strong designs
![Page 15: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/15.jpg)
Primary types of evaluation
• Formative– Needs assessment/diagnostic– Pretesting
• Process– Implemented well? As planned?– Input analysis– Output, effectiveness, user satisfaction
• Summative– Monitoring of program utilization– Monitoring of behavior or health status– Impact/outcome assessment
• Cost effectiveness, cost-benefit• Benchmarks, standards
![Page 16: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/16.jpg)
![Page 17: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/17.jpg)
Characteristics of PE(Powell, 2006)
• Used for decision making.• Deals with research questions about a program.• Takes place in the real world of the program.• Usually represents a compromise between pure and
applied research.• Should be carefully planned.• Have a purpose not an end in itself.• Must be hope for action or change.• Can be used to raise internal as well as external
awareness
![Page 18: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/18.jpg)
Pallet Cleanser
![Page 19: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/19.jpg)
An example…
• Two ways a programs can be studied.• Two ways to determine what is working or not
working with a program.
![Page 20: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/20.jpg)
Implementation failure vs. Theory failure
• Implementation failure– Program is not implemented as planned
• Theory failure– Program is implemented as planned– Intervention does not produce intermediate
results and/or – They do not produce desired outcome
![Page 21: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/21.jpg)
Measures of outcome (Rossi, Lipsey, Freeman, 2004)
• Initial• Intermediate• Long-term
![Page 22: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/22.jpg)
Example: mass media program to reduce HIV transmission
• Initial– Antecedents to behavior– Ex: knowledge, attitudes, practice
• Intermediate– Behaviors– Ex: contraceptive or condom use
• Long-term– Health status– Ex: fertility rates, HIV prevalence
![Page 23: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/23.jpg)
Example: conceptual model of program impact
![Page 24: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/24.jpg)
Implementation failure
![Page 25: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/25.jpg)
Theory failure
![Page 26: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/26.jpg)
Theory failure
![Page 27: Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This](https://reader036.vdocument.in/reader036/viewer/2022062412/5a4d1aed7f8b9ab05997c26a/html5/thumbnails/27.jpg)
What????