11 educ 612 program evaluation_slides

8/2/2019 11 EDUC 612 Program Evaluation_slides

1/32

PROGRAM EVALUATIONEDUC 612, March 12, 2012

Rohanna Buchanan, Ph.D


2/32

Are we intuitive evaluators?

Can we be objective evaluators?

Evaluation Research Principles


3/32

What do we evaluate?

People/Animals Weather Consumer Products Restaurants Advertiser Messages Political Arguments Programs (social/educational)


4/32

Evaluation Research

Important for guiding decision-making and policydevelopment

Can be used for program improvement or assessingprogram impact (i.e., formative or summative

evaluation)


5/32

Evaluation vs.

Social Science Research

The methodologies involved in evaluation and socialscience research are quite similar

The purpose and use of findings tends to differ: Decision making vs. Knowledge generation Generalizability Value/merit judgments


6/32

Program Evaluation Defined

Program evaluation:A description of program performanceA judgment about program efficacy

Uses social research methods Examines social intervention programs Adaptation of research to the political and

organizational context Informs social action to improve social conditions


7/32

1. Needs Assessment: Is there a need for a program?

2. Program Theory: Is the program logical and well-

designed?

3. Program Monitoring: Is the program being

implemented effectively and with fidelity?

4. Program Impact: Did the program work? Did it effectthe intended population?

5. Program Efficiency: Was the program cost-effective?

Program Evaluation Domains


8/32

1. Needs Assessment

Identifying & Defining a Social Problem Identifying & Defining a Target Population Reconciling Need with Resource Availability in

Program Design


9/32

Assessing Program Need

Needs Assessment: a systematic approach toidentifying social problems, determining their

extent, and accurately defining target populations

A needs assessment can be used to aid in the designof new programs or to review the protocols of

established programs

Helps prevent implementation of inappropriate orunneeded services (challenges assumptions)


10/32

What is a Social Need/Problem?

A social need is not an objective phenomena A social phenomena becomes a social need when

stakeholders become dissatisfied with existing

policy

Social needs/problems tend to be hot buttontopics that are politically charged

Drug use, gambling, prostitutionTeen pregnancy, child literacy, hazingDUI, seat belt/helmet use, kids/pets left in carsSchool uniforms, second-hand smoke, homelessness


11/32

Class Activity:

Identify a population and apotential social or educational

programDescribe the need


12/32

2. Program Theory

The set of assumptions and expectations about howa program is designed to work and the connections

presumed between its various activities, functions,

and outcomes Does the conceptualization and design of the

program reflect valid assumptions about the natureof the target problem and represent a well-

founded and feasible approach to resolving it?


13/32

Program Theory

Assumptions and expectations about programoperations:

Program

Theory

ProcessTheory

(Function)

ImpactTheory

(Causality)

OrganizationalPlan

Service Delivery


14/32

Organizational Plan

Details the programs service objectives and theresources that are necessary to carry out those

objectives

i.e., how program management decides to garner, configure,and deploy resources and organize program activities so that

the intended service delivery system is developed andmaintained

Necessary conditions: what type of service, who is to receive services (target

population), how many and what type of employees,

educational and work experience qualifications, location and

hours of service, etc.


15/32

Organizational Plan (continued)

After School Mathematics Enrichment Example:Program location and timesResources: facilities, equipment, program materialsHuman capital (teachers, support staff): application

process, qualifications, stipend, professional

development/training requirements, number needed

Students: application process; selection criteria


16/32

Services Utilization Plan

The assumptions and expectations about how thetarget population will make initial contact with

the program and be engaged with it through the

completion of the intended services

Assumptions and Expectations:beliefs about how the intended target population is to

become aware of (e.g., information leaflets, public

service announcements, mailings, billboards) and

receive the intended amount of the intervention


17/32

Services Utilization Plan (continued)

After School Mathematics Enrichment Example:Contact: letters to parents, advertisements in schools,

announcements by teachers, counselors, principal

Engagement: arrange transportation, provide snacks,provide incentives & awards for participation and

completion of the program


18/32

Impact Theory

The beliefs, assumptions, and expectationsinherent in a program about the nature of the

change to be brought about by program action

and how the change is to result in the intendedimprovement in social conditions

Causal Theory:the hypothesized cause-effect sequence between

program activities and the programs proximal and

distal outcomes


19/32

Impact Theory (continued)

After School Mathematics Enrichment Example:

Problem: Low mathematics achievement Solution: After-school mathematics program Activities: Focused instruction in computation, problem-

solving, logical reasoning

Proximal Outcomes: Increased self-efficacy, improvedattitudes toward mathematics, improved mathematics

achievement

Distal Outcomes: Increased mathematics course-taking,increased pursuit of math/science degrees in college


20/32

Using Logic Models to

Represent Program Theory

Logic Model:

A logic model is a schematic representation ofprogram theory

The logic model provides a road map of theprogram, highlighting how it is expected to work,

what activities need to come before others, and how

desired outcomes are achieved Logic models facilitate communication with

stakeholders, and serve as an effective

management tool


21/32

Logic Model Example

Problem:

Low Math Achievement

Distal Outcomes:

Increased math course taking

in H;, increased math/science

degrees in college

Proximal Outcomes:

Improved attitudes, self-

efficacy, math achievement

Instructional Focus:

Problem solving, logical

reasoning, computation

Solution:

Afterschool Math Program


22/32

Class Activity:

Describe the program theoryProcess (organizational plan, service

delivery) or causality (impact theory)


23/32

3. Program Monitoring

A form of evaluation to answer questions about program operationsand service delivery (follows logically from program theory)

Is the program being implemented according to the intended design? Is the program staffed and well-organized? Is the program reaching the intended target population? Is the program in compliance with federal law, state statutes, and professional

standards?

Program monitoring can be thought of as the systematic study of keyaspects of program performance (a process or implementation

evaluation)

Program monitoring focuses on documenting the extent to which anintervention is actually delivered, received, and adhered to bytargets


24/32

Monitoring Program Processes

Program monitoring is used to provide quality assuranceinformation (e.g., accountability), identify impediments to

service delivery, and document the processes by which a

program works

Program monitoring can also uncover service delivery breakdowns (e.g., program materials not received or

utilized)

unwanted side effects (e.g., NCLB: narrowing of content, teaching to thetest)

and confounds (e.g., extra treatments) early in the service deliveryprocess

Program monitoring thus has great diagnostic value for theimpact evaluation


25/32

Implementation Fidelity

The monitoring of program processes is the means by whichimplementation fidelity is assessed

Definition: The degree to which a program is delivered as intended(adheres to program protocol)

Synonyms: treatment integrity, treatment fidelity, programadherence, protocol compliance

Implementation fidelity is a relatively new concept (emerged inthe 1970s) and is an often overlooked aspect of the

evaluation process increasingly common Reasons for not closely considering implementation fidelity:

the general lack of clearly defined treatment models, resource

costs, and attitudes about the professional discretion of

providers

Th I t f


26/32

The Importance of

Implementation Fidelity

When null results are obtained, close monitoring of the degreeto which a program is implemented as designed enables one

to distinguish between implementation and theory failure

Implementation failure: the failure of the program to adequatelydeliver the service or perform the actions to elicit change (i.e., theintervention is delivered inconsistently or not at all)

Theory failure: a failure of the programs conceptual model(correctly implemented program services do not bring about the

expected proximal or distal outcomes) When positive results are obtained, fidelity data can be used

to ascertain whether the results were due to program or the

non-program activities of providers


27/32

Class Activity:

How will you conduct programmonitoring?

i.e., measure fidelity?


28/32

4. Program Impact

Evaluation to answer questions about programeffectiveness (i.e., did the program have the

intended effect?)

Need to separate the effects of program fromthe intake characteristics & natural maturation of

program participants

Social science research methods provide the key


29/32

Isolating Program Effects

0

10

20

30

40

50

60

Pre Program Post Program


30/32

Isolating Program Effects

0

10

20

30

40

50

60

Pre Program Post Program

ControlGroup

ProgramEffect

TreatmentGroup


31/32

5. Program Efficiency

Evaluation to answer questions about program costand cost effectiveness (i.e., did the ends justify the

means?)

Follows logically from the evaluation of programimplementation and impact necessary to justify

continued support (forces consideration ofalternatives)


32/32

Class Activity:

What are important considerations related toprogram impact and program efficiency?

11 educ 612 program evaluation_slides

Documents