program evaluation

Program EvaluationJoy Anne R. Puazo

Marie Buena S. Bunsoy

Evaluation

Evaluation

◦The systematic gathering of information for purposes of making decisions

◦- Richards et al. (1985. p. 98)

Evaluation

“Systematic educational evaluation consists of a formal assessment of

the worth of educational phenomena.”

- Popham (1975, p.8)

Evaluation

“Evaluation is the determination of the worth of a thing. It includes obtaining information for use in judging the worth of the program,

product, procedure, or object, or the potential utility of alternative approaches designed to attain specified objectives.”

- Worthen and Sanders (1973, p.19)

Evaluation

“Evaluation is the systematic collection and analysis of all relevant information

necessary to promote the improvement of the curriculum and assess its

effectiveness within the context of the particular institutions involved.”

- Brown

Evaluation

Testing◦Procedures that are based on test, whether

they be criterion-referenced or norm-referenced in nature.

Measurement◦Testing is included; other types of

measurements that result in quantitative data such as attendance records, questionnaires, teacher-student ratings

Evaluation

Evaluation◦More qualitative in natureCase studies, classroom observations, meetings, diaries, and conversations

Approaches to Program Evaluation

Product-oriented approaches◦Focus: goals and instructional objectives

◦Tyler, Hammond, and Metfessel and Michael

Product-oriented approaches

The programs should be built on explicitly defined goals, specified in terms of the society, the students, the subject matter, as well as on measurable behavioral objectives.

Purpose: To determine whether the objectives have been achieved, and whether the goals have been met.

- Tyler


Five steps to be followed in performing a curriculum evaluation: Hammond (Worthen and Sanders 1973, p. 168)

1. Identifying precisely what is to be evaluated

2. Defining descriptive variables3. Stating objectives in behavioral terms4. Assessing the behavior described in the

objectives5. Analyzing the results and determining the

effectiveness of the program


8 Major Evaluation Process: Metfessel and Michael (1967)

1. Direct and indirect involvement of the total school community

2. Formation of a cohesive model of broad goals and specific objectives

3. Transformation of specific objectives into communicable form


4. Instrumentation necessary for furnishing measures allowing inferences about program effectiveness

5. Periodic observation of behaviors6. Analysis of data given by status and change

measures7. Interpretation of the data relative to specific

objectives and broad goals8. Recommendations culminating in further

implementations, modifications, and in revisions of broad goals and specific objectives


Static-Characteristic Approaches◦Conducted by outside experts who inspect

outside the program by examining various records

Accreditation◦Process whereby an association of institutions

sets up criteria and evaluation procedures for the purposes of deciding whether individual institutions should be certified as members in good standing of that association

Static-Characteristic Approaches

“A major reason for the diminishing interest in accreditation conceptions of evaluation is the

recognition of their almost total reliance on intrinsic rather than extrinsic factors. Although there are some

intuitive support for the proposition that these process factors are associated with the final outcomes

of an instructional sequence, the scarcity of the empirical evidence to confirm the relationship has

created growing dissatisfaction with the accreditation approach among the educators.”

- Popham (1975, p. 25)


Process-Oriented Approaches (Scriven and Stake)

Scriven’s Model/Goal-free evaluation◦Limits are not set on studying the expected effects of the program vis-à-vis the goals

Process-Oriented Approaches

Countenance model (Stake; 1967)

1. Begin with a rationale2. Fix on descriptive operations3. End with judgmental operations at 3 levels:

Antecedents Transactions Outcomes


Decision-Facilitation Approaches◦Evaluators attempt to avoid making judgments

◦Gathering information that will help the administrators and faculty in the program make their own judgments and evaluation

◦Examples: CICP, CSE, Discrepancy model

Decision-Facilitation Approaches

CIPP (Context, Input, Process, Product)

4 key elements in performing program evaluation: Stufflebeam (1974)

1. Evaluation is performed in the service of decision making, hence it should provide information that is useful to decision makers.

2. Evaluation is a cyclic, continuing process and therefore must be implemented through a systematic program.

3. The evaluation process includes 3 main steps of delineating, obtaining, and providing. (methodology)

4. The delineating and providing steps in the evaluation process are interface activities requiring collaboration.


CSE (Center for the Study of Evaluation)

5 different categories of decisions (Alkin; 1969)

1. System assessment2. Program planning3. Program implementation4. Program improvement5. Program certification


Discrepancy Model (Provus; 1971)

“Program evaluation is the process of (1) defining program standards; (2)

determining whether the discrepancy exists between some aspect of program

performance and the standards governing the aspect of the program; and (3) using discrepancy information either to change

performance or to change program standards.”


5 stages that shows that discrepancy model is a process-oriented approach:

1. Program description stage2. Program installation stage3. Treatment adjustment stage4. Goal achievement analysis stage5. Cost-benefit analysis

Three Dimensions that Shape Point of View on Evaluation:

1. Formative vs. Summative2. Process vs. Product3. Quantitative vs. Qualitative

Three Dimensions

Purpose of InformationFormative evaluation

◦During the ongoing curriculum dev’t process◦To collect and analyze information that will help

in improving the curriculum

Summative evaluation◦End of a program◦To determine the degree to which the program

is successful, efficient, and effective.

Weakness of Summative Evaluation

Most language program are continuing institutions that

do not conveniently come to an end so that such

evaluation can be performed

Benefits of Summative Evaluation

Identifying the success and failure of the program

Provides an opportunity to stand back or consider what has been achieved in the longer view

Combination of F and S:◦Can put the program and its staff in a strong

position for responding any crises that might be brought on by the evaluation from outside the program

Three Dimensions

Types of InformationProcess evaluation

◦Focuses on the workings of a program

Product evaluation◦Focus on whether the goals of the program are being achieved

Three Dimensions

Types of Data and AnalysesQuantitative data

◦Countable bits of information which are usually gathered using measures that produce results in the form of language

Qualitative Data◦Consist of more holistic information based on

observations

Doing Program Evaluation

Instruments and Procedures

Quantitative Qualitative

Existing information

Records Analysis X X

Systems X X

Literature Review X

Letter Writing X



Tests

Proficiency X

Placement X

Diagnostic X

Achievement X



Observations

Case Studies X

Diary Studies X

Behavior Observation X

Interactional Analyses X

Inventories X



Interviews

Individual X X

Group X X



Meetings

Delphi Technique X

Advisory X

Interest group X

Review X



Questionnaires

Biodata Survey X

Opinion Survey X X

Self-ratings X

Judgmental ratings X

Q-sort X

Gathering Evaluation Data

Quantitative Evaluation StudiesQualitative Evaluation StudiesUsing Both Quantitative and

Qualitative Methods


Quantitative data Bits of information that are countable and are gathered using measures that produce results in the form of numbers.


The importance of using quantitative data is not so much in the collection of those data, but rather in the analysis of the data, which should be carried out in such a way that patterns emerge.


A classic example of what many people think an evaluation study ought to be is a quantitative, statistics-based experimental study designed to investigate the effectiveness of a given program.


Key Terms:Experimental Group

One that receives the treatment

Control GroupReceives no treatment


Treatment is something that the experimenter does to the experimental group or rather than an experience through which they go (as in learning experience).


The purpose of giving treatment to the experimental group and nothing to the control group is to determine whether the treatment has been effective.


Qualitative Evaluation Studies

Consists of information that is more holistic than quantitative data.


Goal Of Qualitative Research

To collect data in order to analyze them in such a way that patterns emerge so that sense can be made of the results and the quality of the program can be evaluated.


Using Both Quantitative and Qualitative Methods

Both types of data can yield valuable information in any evaluation, and therefore ignoring either type of information would be pointless and self-defeating. Sound evaluation practices will be based on all available perspectives so that many types of information can be gathered to strengthen the evaluation process and ensure that the resulting decisions will be as informed, accurate, and useful as possible.

Program Components as Data Sources

The purpose of gathering all this information is, of course, to determine the effectiveness of the program, so as to improve each of the components and the ways that they work together.


The overall purpose of evaluation is to determine the general effectiveness of the program, usually for purposes of improving it or defending its utility to outside administrators or agencies.

Naturally, the curriculum components under discussion are needs analysis, objectives, testing, materials, teaching, and the evaluation itself.


Effective?

Quantitative study that demonstrates the students (who receive a language learning treatment) significantly outperformed a control group (who did not receive the treatment) is really only showing that the treatment in question is better than nothing.

Lynch (1986)

Questions Primary Data Sources

N AE NE AD LS Y S I S

Which of the needs that were originally identified turned out to be accurate (now that the program has more experience with students and their relationship to the program) in terms of what has been learned in testing, developing materials, teaching and evaluation?

All original needs analysis documents

OBJECTIVES

Which of the original objectives reflect real student needs in view of the changing perceptions of those needs and all of the other information gathered in testing, materials development, teaching, and evaluation?

Criterion-Referenced Tests (Diagnostic)

TEST ING

To what degree are the students achieving the objectives of the courses? Were the norm-referenced and criterion-referenced tests valid?

Criterion-referenced tests (achievement), Test Evaluation Procedures

MATERIALS

How effective are the materials (whether adopted, developed, or adapted) at meeting the needs of the students as expressed in the objectives?

Materials Evaluation Procedure

TEACHING

To what degree is instruction effective?

Classroom observations and student evaluations

Program Component as Data Sources

More detailed questions:What were the original perception of the students’

needs?How accurate was this initial thinking (now that

we have more experience with the students and their relationship to the said program)?

Which of the original needs, especially as reflected in the goals and objectives, are useful and which are not useful?

What newly perceived needs must be addressed? How do these relate to those perceptions that were found to be accurate? How must be the goals and objectives be adjusted accordingly?


Efficient?Evaluators could set up a study to investigate the degree to which the amount of time can be compressed to make the learning process more efficient.


N AE NE AD LS Y S I S

Which of the original student needs turned out to be the most efficiently learned? Which were superfluous?

Original needs analysis documents and criterion referenced tests (both diagnostic and achievement)

OBJECTIVES

Which objectives turned out to be needed by the students and which they already know?

Criterion-referenced tests (diagnostic)

TESTING

Were the norm-referenced and criterion-referenced tests efficient and reliable?

Test Evaluation Procedures

MATERIALS

How can material resources be reorganized for more efficient use by teachers and students?

Materials blueprint and scope-and-sequence charts

TEACHING

What types of support are provided to help teachers and students?

Orientation documentation and administrative support structure


AttitudesThe third general area concern in language program evaluation will usually center on the attitudes of the teachers, students, and administrators regarding the various components of the curriculum as they were implemented in the program.


Needs Analysis What are the students’, teachers’, and

administrators’ attitude or feelings about the situational and language needs of the students? Before program?

After?

Needs analysis questionnaires and any

resulting documents

Objectives What are the students’, teachers’, and

administrators’ attitude or feelings about the usefulness of the objectives as originally formulated? Before program?

After?

Evaluation interviews and questionnaires

Testing What are the students’, teachers’, and

administrators’ attitude or feelings about the

usefulness of the tests as originally developed?

Before? After?

Evaluation interviews, meetings, and questionnaires

Materials What are the students’, teachers’, and


usefulness of the materials as originally adopted, developed

and/or adapted? Before? After?


Teaching What are the students’, teachers’, and


usefulness of the teaching as originally

delivered? Before? After?


program evaluation

Education

evaluation evaluation

evaluation procedures

curriculum evaluation

evaluation examples

major evaluation process

program evaluation joy

evaluation testing procedures

systematic program