Chapter 6: Program-Oriented Approaches

Presentation by Jay Kerstetter, Amanda Brown, Jody Yoos, & Jane Lightner

J.K.

Orienting Questions: #1 What are the key concepts of the objectives-oriented evaluation

approach?#2 How has this approach influenced evaluation?

What is it? Objectives-oriented evaluation approach helps determine

whether some or all of the program objectives are achieved and, if so, how well they are achieved.

Evaluators may work with stakeholders to establish if program objectives are met.

Information from this approach can assist with deciding to maintain, terminate, or change approaches within the program.

J.K.

Tylerian Evaluation Approach

1. Ralph Tyler is credited with initiating this approach in the 1930’s

2. Tyler began to formulate his views on education and evaluation.

3. His approach included the following steps: Establish goals or objectives Classify the goals or objectives Define objectives in behavioral terms Find situations in which achievement of objectives can be shown Develop or select measurement techniques Collect performance data Compare performance data with behaviorally stated objectives

4. This approach was readily adoptable by evaluators and had great influence on evaluation theorists.

J.K.

Provus’s Discrepancy Evaluation Model:

1. Developed by Malcolm Provus; viewed evaluation as a continuous information management process.

2. Provus stemmed key characteristics of his proposal from Tyler.3. Provus viewed evaluation as a process.4. This process was called DEM; Discrepancy Evaluation Model, which

are broken into four developmental stages.◦ Definition ◦ Installation ◦ Process◦ Product◦ Cost-benefit analysis (optional)

The DEM was designed to facilitate the development of programs in large public school systems and later applied to state evaluations by federal bureau.

The DEM was one of the earliest approaches to evaluation and elements of it can still be found today in many evaluation. J.K.

Orienting Question #3: How is the objective-oriented evaluation approach used today?

Standards-based testingAccountability in educationPerformance monitoring systems used in many government programs

Many refinements to the system since the 1930’s

J.Y.

Ralph W. Tyler: Tylerian Evaluation Approach

Influenced the Elementary and Secondary Education Act (ESEA) of 1965, first act to require evaluation of educational programs

Started the National Assessment of Educational Progress (NAEP), only way to examine educational achievement of al 50 states.

J.Y.

Malcolm Provus: Provus’s Discrepancy Evaluation Model

Approach evaluated in Pittsburgh public schools

Viewed evaluation as a continuous information- management process designed to serve as “the watch-dog of program-management” and the “handmaiden of administration in the management of program development through sound decision making”

J.Y.

Orienting Question #4a: How are Logic Models used in evaluation?

Developed as an extension of objectives-oriented evaluation

Designed to fill in those steps between the program and its objectives

Program planners/evaluatorIdentify programo InputsoActivitiesoOutputs (immediate

program impacts)oOutcomes (long-term

objectives/goals)

J.Y.

Today Logic Models are used in program planning/evaluation

Help program staff articulate /discuss how program might achieve goals

What elements are important to evaluate at any given time

Build internal evaluation capacity or think in an evaluative way

Example Companies:United way of AmericaWK Kellogg FoundationAnnie E Casey Foundation

J.Y.

Orienting Question #4b: How are Program Theories used in evaluation?

Theory-Based evaluationIs used by evaluators to gain a better understanding of

the programBetter define the evaluation questions the study should

addressTo aid their choices of what concepts to measure and

when to measure themTo improve their interpretation of results and their

feedback to stakeholders to enhance use.

J.Y.

Steps in Theory Based Evaluation

1) Engage Relevant stakeholder2) Develop a first draft of program theory

evaluator or evaluation team3) Present draft to stakeholders for further

discussion, reaction, and input4) Conduct a plausibility check5) Communicate findings to key stakeholders6) Probe arrows for model specificity7) Finalize program impact theory

J.Y.

Theory Driven Evaluation

Work with stakeholders to identify key questions to be answered

in the evaluation and the appropriate designs and methods for

answering those questions

Emphasis is on testing the program model

Provide guidance as to what to measure and when to measure it

The selection of the evaluation questions to be addressed depend

on the stage of the program and what stakeholders hope to learn

Provides evaluator with critical information that can be used

throughout the evaluation

J.Y.

Orienting Question #5: How do theory-based evaluation and objectives approaches differ?

Objectives Approach:1. The objectives are

identified by group looking for the evaluation.

2. Purpose of activity specified and then those purposes are being acheived.

3. Individual credited-Ralph W. Tyler

Theory-Based1. The evaluator discusses

goals, basics, and objectives of program w/the stakeholders.

2. The evaluator decides how the program should work and then sees if it works that way.

3. Huey Chen and Leonard Bickman helped to develop approaches to theory based.

4. Science based and quantitative.

A.B.

Question #6: What are the strengths and limitations of program oriented

evaluation approaches?

Strengths:Objective oriented – the simplicity of the

concept and program makes it easy to understand, follow and possibly implement. ◦Face validity – the evaluator is being held

accountable for what is being asked to be evaluated. They want to see if what is currently happening or working based on their own goals and objectives for the program.

A.B.

Strengths (con’t)

Theory oriented – the chance for the evaluator to partake in dialogue with the stakeholders is a strength of the program because it helps them to expand their knowledge on the topic. ◦This then leads to the evaluator to have a clear

understanding of the program so they know how to properly evaluate the program.

A.B.

Weaknesses:

Objective oriented – the evaluator can have a single minded focus on the objectives which will cause them to overlook the complications, elements and factors contributing to that programs success or failure.a. This approach does not ask for the evaluator to gain

an understanding of the context in which the program operates this could then be what is affecting the program’s success or failure.

b. Evaluator could ignore the actual value of the objective. Since the evaluator is told what the objectives are they are not asked to evaluate whether that objective even fits the program.

A.B.

Weaknesses (con’t)

Theory oriented – this like the objective oriented can cause the evaluator to ignore certain aspects of the program that are important because the evaluator is so focused on the theory of how it runs and not how it is actually running.a. Evaluators may ignore the needs or values of

stakeholders involved with the program.b. This approach may also oversimplify the

complexity of the program making it feel easier to evaluate than it really is because not all of the surrounding factors are accounted for in the process.

A.B.

Orienting Question # 7: What is goal-free evaluation?

Rationale: “Goals should not be taken as givens.”

Developed by Scriven (1972)---believes the most important function of a goal-free evaluation is to reduce the bias that occurs from knowing program goals

Thus…increase objectivity in judging the program as a whole.

J.L.

Orienting Question #8: What does it teach us about conducting an evaluation?

Goals can act as “blinders” causing us to possibly miss the most important outcomes not related to the goals.

Goal-free evaluation was proposed to primarily indentify the unanticipated side effects that an objectives-oriented evaluation might miss

J.L.

Major Characteristics of a Goal-Free Evaluation

1. Evaluator purposely avoids

becoming aware of program goals.

2. Predetermined goals are not

permitted o narrow the focus of the

evaluation study.

3. Goal-free evaluation focuses on actual outcomes rather than intended program outcomes.

4. The goal-free evaluator has

minimal contact with the program

manager and staff.

5. Goal-free evaluation increases the likelihood that unanticipated side

effects will be noted.

J.L.

Goal Directed + Goal-Free= can work together

Internal Goal Evaluator vs. External Goal-Free Evaluator

Goal-directedHow well is program

meeting goalsProvide information to

administrator

What does the program do?

Looking at ALL the programs outcomes, intended or not

J.L.

Information taken from Program Evaluation:

Alternate Approaches and Practical Guidelines

p. 153-171