Overview of Evaluation

ED. 883.721

Session 1: 01/28/10

Assignment 1

HandoutsReview your problem and program for usDiscussion

Clear problem statementsDefining termsClear program or policyScope and scale

“A program is a theory and an evaluation is its

test.”

(Rein, 1981 p141)

What is a program theory?

Theory of action – why should the program work?Explains why the program does what it does and

provides the rationale for expecting that doing so will achieve the desired results

Explicit or implicit in all programs and policies

X Y

Why would we want to know the program’s theory of action?

Develop knowledge of issuesAid to identify evaluation questionsFramework for data collectionHelp to understand the dataInterpret the evidenceMake sound recommendationsBetter reportingObject of evaluation itself

Social programs are complex thingsCan be clearly defined

In terms that are clear and reproducibleInvolves a mechanism that is well understood or visible

or “fuzzy”

Programs are not “Black Boxes”

Y

Conceptualizing a Program

Define the Boundaries of the Program

Program GoalsOfficial, Stated GoalsAs perceived by staffDistinct from objectivesProblems?

Assumptions

X Yx

y

Important Components

Program Goals and ObjectivesCan they be realistically be obtained?

Program Functions, Components and Activities

Linking Functions, Components and ActivitiesHow does everything fit together?

Finding out about a Program

Review program documentsInterview stakeholdersSite visits and observationsLiterature review

Corroborate it

Program Theory

Target Population

Program

Outcomes

Impact Theory

Assumptions about the change process actuated by the program

Assumptions about the improved outcomes

Target Population

Program

Outcomes

Impact Theory

Causal – cause and effect relationshipDiagrams of the chain of causes and effectsProximal versus Distal Outcomes

X Yz

y

Service Utilization Theory

How to reach targetsProvision and sequence servicesHow to conclude services

Target Population

Program

Outcomes

Service Utilization Theory

Focus on assumptions how and why recipients will become engaged in the program and follow through to desired outcomes

Flow charts of paths through program

Organizational Theory

Program resources, personnel, administration, organization

Factors that make it possible to develop and maintain service delivery

Target Population

Program

Outcomes

Organizational Theory/Plan

Who is in charge of what?What resources are necessary at critical points?Charts preconditions for each critical pointInputs and activities

Inputs/Factors

Positive and NegativeResources and BarriersTangible

Activities

Processes, tools, events, actionsProducts

Curricula, materials

ServicesEducation, trainingCounseling

InfrastructureRelationships, capacities

Outputs

DIRECT results of program activitiesSpeak to size and scopeData about activities

Number of classes taughtParticipation ratesHours of services provided

Outcomes & Impacts

OutcomesWhat is expected to result from activitiesIndividual level

ImpactsHigher level changes that result from program activities

Problem Statements

Feb 11: Identifying Problems and Asking Researchable Questions

RLF Chapters 3 & 4Light, R., Singer, J., & Willet, J. (1999). What are

your questions? In By design: planning research on higher education (pp. 12-40). Cambridge, MA: Harvard University Press.

Kerlinger, F. (1986). Problems and hypotheses. In Foundations of behavioral research (pp. 15-25). New York: Harcourt Brace Jovanovich.