![Page 1: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/1.jpg)
Program Evaluation In A Nutshell
1
Jonathan Brown, M.A.
![Page 2: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/2.jpg)
Two Types of Evaluation
• At the most basic level there are two types of evaluation:
Formative: primary purpose is to provide information to improve a program.
Summative: concerned with providing information about program adoption, continuation, or expansion.
2
![Page 3: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/3.jpg)
Two Types of Evaluation
3
Rel
ativ
e E
mp
has
is
Program Life
Summative Evaluation
Formative Evaluation
![Page 4: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/4.jpg)
What Are You Assessing?
The evaluation plan must consider the situation or context of the evaluation.
• The purpose of the evaluation.
• Improvement (formative evaluation).
• Accountability (summative evaluation).
• Knowledge generation.
• Hidden agendas.
4
![Page 5: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/5.jpg)
What Are You Assessing?
The evaluation plan must consider the situation or context of the evaluation.
• The program’s structure and context.
• The stage of program development.
• Administrative and political context.
• Conceptual and organizational structure of the program.
5
![Page 6: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/6.jpg)
What Are You Assessing?
The evaluation plan must consider the situation or context of the evaluation.
• The resources available for the evaluation.
• Personnel
• Equipment
• Facilities to support data collection, analyses, and reporting.
• Is specialised expertise needed?
6
![Page 7: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/7.jpg)
What Are You Assessing?
Typically, evaluation involves assessing one or more of the five following areas:
1. The need for a program.
2. The program’s design.
3. The program’s implementation or process.
4. The program’s impacts or outcomes.
5. The efficiency of a program.
7
![Page 8: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/8.jpg)
Needs Assessment
• Purpose of a social program is to alleviate a social problem.
• Needs assessments assess:
• The nature, extent, and distribution of a social problem.
• How these features will impact the design of an intervention program.
8
![Page 9: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/9.jpg)
Needs Assessment
These assessments can be conducted when…
• Planning a new program.
• Restructuring an existing program.
• Determining if an existing program is responsive to the current needs of the target population.
9
![Page 10: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/10.jpg)
Assessing Program Theory
• Program theory evaluation is an evaluation guided by a program’s explicit theory of how it causes intended outcomes.
• Program theory outlines the following:
•Target population
•Resources
•Activities
•Outcomes
• A program’s theory is represented by Theory of Change Models (TCMs) and Program Logic Models (PLMs).
10
![Page 11: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/11.jpg)
Assessing Program Theory
11
Conditions before the project
• With no information about the program, it remains a mysterious black box.
• The evaluation will also not be able to attribute the final conditions to the specific aspects of the program.
Conditions after the project
The Program is a
![Page 12: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/12.jpg)
Assessing Program Theory
• Assessing program theory can be done by applying SMART principles (Knowlton & Phillips, 2009).
• Specific – the program is clear enough to implement and evaluate.
• Measurable – indicators can be qualified and/or quantified.
• Action oriented – activities will provoke the desired change in targets.
• Realistic – the program is plausible and feasible.
• Timed – duration of activities and intended outcomes are specified.
12
![Page 13: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/13.jpg)
Process Evaluation
• Given that a valid theory on how to intervene on a diagnosed social problem has been done, the program must be implemented well to be successful.
• Two forms:
• Process evaluation
• Program monitoring
13
![Page 14: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/14.jpg)
Process Evaluation
• Process evaluation looks at service utilization and program organization.
Service Utilization
• The extent to which the intended targets receive services.
• Important when participation is voluntary, or participants must learn new procedures, or habits.
• Coverage
• Bias
14
![Page 15: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/15.jpg)
Process Evaluation
• Process evaluation looks at service utilization and program organization.
Program Organization
• How well the program is in terms of managing its efforts and using its resources to accomplish its activities.
• Service delivery:
• None or not enough intervention
• Wrong intervention
• Unstandardized intervention15
![Page 16: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/16.jpg)
Outcome Evaluation
• Also known as an impact assessment.
• Assesses the extent to which a program produces the outcomes it intends.
• But the desired changes could be caused by factors unrelated to the program.
• Interested in the changes produced by the program above and beyond any external elements.
16
![Page 17: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/17.jpg)
Outcome Evaluation
Program outcomes
• The states or conditions of the target that changed as a
result of a program.
1. Observed characteristics of the target population or social
conditions.
2. Occur to different degrees.
17
![Page 18: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/18.jpg)
Outcome Evaluation
Program outcomes
• Outcome level – status of an outcome at a point in time.
• Outcome change – difference between outcome levels at different points in time.
• Program effect – the unique portion of an outcome that can be attributed to only the program.
18
![Page 19: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/19.jpg)
Outcome Evaluation
19
Out
com
e V
aria
ble
Before Program
During Program
After Program
Outcome status without program
Outcome status with program
Program effect
Post-program outcome level
Pre-program outcome level
Outcome Change
(Rossi, Lipsey, & Freeman, 2004)
![Page 20: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/20.jpg)
Efficiency Assessment
• Due to limited resources, program accomplishments must be judged against program costs.
• Two types of efficiency assessments:
• Cost-benefit analysis
• Cost-effectiveness analysis
• Such assessments require placing a dollar amount on program activities and benefits.
20
![Page 21: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/21.jpg)
Typical Evaluation Questions
Needs Assessment
• What is the nature and magnitude of the problem?
• What are the characteristics of the population in need?
• What are the needs of the population?
• What services are needed?
• How much service is needed and over what period of time?
• What service delivery arrangements are needed to provide services to the population?
21
![Page 22: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/22.jpg)
Typical Evaluation Questions
Program Theory Assessment
• Is the program doing the right thing?
• How could the program better address the needs of participants?
• How should the program be organized?
• What are the best delivery systems for the services?
• Is the program specific enough to measure?
22
![Page 23: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/23.jpg)
Typical Evaluation Questions
Process Evaluation
• How many persons are receiving services?
• Are administrative and service objectives being met?
• Are the intended targets receiving services?
• Are the necessary program functions being performed
adequately?
• Are resources being used effectively and efficiently?
23
![Page 24: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/24.jpg)
Typical Evaluation Questions
Outcome Evaluation
• Are the goals and outcomes being achieved?
• Do the services have adverse side effects?
• Are some participants affected more by the services than others?
• Is the problem the services are intended to address made better?
24
![Page 25: Program Evaluation In A Nutshell 1 Jonathan Brown, M.A](https://reader036.vdocument.in/reader036/viewer/2022062715/56649d7b5503460f94a5f5a1/html5/thumbnails/25.jpg)
Typical Evaluation Questions
Efficiency Assessment
• Are resources used efficiently?
• Is the cost reasonable in relation to the magnitude of the
benefits?
• Would alternative approaches yield equivalent benefits at less
cost?
25