Alternative Approaches to EvaluationDzulkifli Awang, PhDUniversiti Teknologi Malaysia(courtesy of Dr. Mary Schutten, Western Michigan University)

Review: IntroductionDefine EvaluationHow do formal/informal evaluation differ?What are two uses of evaluation in education?What are the pros/cons of using an external evaluator?

Alternative ApproachesStakeholders: individuals and groups who have a direct interest in, and may be affected by, evaluation; should be involved early, actively & continuously

Program: activities that are provided on a continuing basis; typically what is evaluated

There are a variety of alternative, often conflicting, views of what evaluation is and how it should be carried out

Why so many alternatives?The way one views evaluation directly impacts the type of activities/methods used

Origins of alternative models stem from differences in:Philosophical & ideological beliefsMethodological preferencesPractical choices

Philosophical & Ideological BeliefsEpistemologies (philosophies of knowing)Objectivism (social science base of empiricism; replicate)Subjectivism (experientially-based; tacit knowledge)Pros/Cons of each?Principles for assigning value (parallel obj/subj)Utilitarian: focus on group gains (avg scores); greatest good for the greatest numberIntuitionist-pluralist: value is individually-determinedRoom for both or are these dichotomous?Philosophical purists are rare (impractical?)Choose the methods right for THAT evaluationUnderstand assumptions/limitations of different approaches

Methodological PreferencesQuantitative (numerical) Qualitative (non-numerical)

Evaluation is a transdiscipline; crosses paradigmsLaw of the instrument fallacyWith hammer/nails, all appears to need hammering

Identify what is useful in each evaluation approach, use it wisely & avoid being distracted by approaches designed to deal w/ different needs

Practical ConsiderationsEvaluators disagree whether/not intent of evaluation is to render a value judgment Decision-makers or evaluator render judgment?Evaluators differ in views of evaluations political roleAuthority? Responsibility? These dictate eval styleInfluence of evaluators prior experienceWho should conduct the evaluation and nature of expertise needed to do soDesirability (?) of having a wide variety of evaluation approaches

Classification Schema for Evaluation ApproachesConceptual approaches to evaluation, NOT techniquesObjectives-oriented: focus on goals/objectives & degree to which they are achievedManagement-oriented: identifying and meeting informational needs of decision makersConsumer-oriented: generate information to guide product/service use by consumersExpertise-oriented: use of professional expertise to judge quality of evaluation objectParticipant-oriented: stakeholders centrally involved in process

Objectives-oriented ApproachPurposes of some activity are specified and then evaluation focuses on the extent to which these purposes are achievedRalph W. Tyler popularized this approach in education (criterion ref test)Tylerian modelsMetfessel & Michaels paradigm (enlarged vision of alternative instruments to collect evaluation data)Provuss Discrepancy Evaluation Model (agree on stds, det if discrepancy exists btwn perf/std, use discrepancy info to decide to improve, maintain, terminate program)Logic modelsDetermine long-term outcomes & backtrack to today

Objectives-oriented StepsEstablish broad goals or objectives tied to mission statementClassify the goals or objectivesDefine objectives in behavioral termsFind situations where achievement of objectives can be shownSelect/develop measurement techniquesCollect performance dataCompare data with behaviorally stated objectives

Objectives-oriented Pros/ConsStrengths: simplicity, easy to understand, follow and implement; produces information relevant to the mission

Weakness: can lead to tunnel visionIgnores outcomes not covered by objectivesNeglects the value of the objectives themselvesNeglects the context in which evaluation takes place.

Goal Free EvaluationThis is the opposite of objectives-oriented evaluation, but the two supplement one another Purposefully avoid awareness of goals; should not be taken as given, goals should be evaluatedPredetermined goals not allowed to narrow focus of evaluation studyFocus on actual outcomes rather than intendedEvaluator has limited contact with program manager and staffIncreases likelihood of seeing unintended outcomes

Management-oriented ApproachGeared to serve decision makersIdentifies decisions administrator must makeCollects data re: +/- of each decision alternativeSuccess based on teamwork between evaluators and decision makersSystems approach to education in which decisions are made about inputs, processes, and outputsDecision maker is always the audience to whom evaluation is directed

CIPP Evaluation Model (Stufflebeam)Context Evaluation: planning decisionsNeeds to address? Existing programs?Input Evaluation: structuring decisionsAvailable resources, alternative strategies?Process Evaluation: implementing decisionsHow well is plan being implemented? Barriers to success? Revisions needed?Product Evaluation: recycling decisionsResults? Needs reduced? What to do after program has run its course?

CIPP StepsFocusing the EvaluationCollection of InformationOrganization of InformationAnalysis of InformationReporting of InformationAdministration of Evaluation (timeline, staffing, budget etc)

Context EvaluationObjective: define institutional context, target population and assess their needs

Method: system analysis, survey, hearings, interviews, diagnostic tests, Delphi technique (experts)

For deciding upon the setting to be served, the goals associated with meeting needs and objectives for solving problems

Input EvaluationObjective: identify and assess system capabilities, procedural designs for implementing the strategies, budgets, schedules

Method: inventory human and material resources, feasibility, economics via literature review, visit exemplary programs

For selecting sources of support, solution strategies in order to structure change activities, provide basis to judge implementation

Process EvaluationObjective: identify or predict defects in the process or procedural design, record/judge procedural events

Method: monitoring potential procedural barriers, continual interaction with and observation of the activities of the staff

For implementing and refining the program design and procedure (a.k.a., process control)

Product EvaluationObjective: collect descriptions and judgments of outcomes and relate them to CIP, interpret worth/merit

Methods: measure outcomes, collect stakeholder information, analyses of data

For deciding to continue, terminate, modify, or refocus an activity and to document the effects (whether intended or unintended)

Uses of Management-oriented Approaches to EvaluationCIPP has been used in school districts, state and federal government agenciesUseful guide for program improvementAccountabilityFigure 5.1 (p. 94)Formative and summative aspects of CIPP

Management-oriented Pros/ConsStrengths: appealing to many who like rational, orderly approaches, gives focus to the evaluation, allows for formative and summative evaluation

Weaknesses: preference given to top management, can be costly and complex, assumes important decisions can be identified in advance of the evaluation

REVIEW/QsWhy are there so many alternative approaches to evaluation?What two conceptual approaches to evaluation did we discuss tonight? What are their +/-?Which, if either, of these approaches do you think will work for your evaluation object?

Identify your most likely evaluation object