formative evaluation models 1

FORMATIVE EVALUATION MODELS

By

Olorunlero Jumoke I.

01/250C187

Educational Research Method and Evaluation (M.Ed)

1 | P a g e F O R M A T I V E E V A L U A T I O N M O D E L S

The areas of discussion in this paper are sectioned as follows: Keywords defined Description of Formative Evaluation technique Formative Evaluation Models Types of Models Summary of Major Evaluation Models: Strength, necessary cautions and other notes. Literature review on Evaluation Models Conclusion

Keywords definedTo adequately understand Formative Evaluation Models, the following key words should be understood.

Keyword 1: EvaluationAs defined by Daniel 2003, generally; an evaluation is a systematic investigation of the value of a program or other evaluand. More specifically, he described evaluation as a process of delineating, obtaining, reporting, and applying descriptive and judgmental information about some object’s merit, worth, probity, and significance.

Ron O. n.d. also defines evaluation as the process of gathering information about the merit or worth of a program for the purpose of making decisions about its effectiveness or for program improvement.

Keyword 2: ModelA model is a representation of a system that allows for investigation of the properties of the system and, in some cases, prediction of future outcomes. Models are often used in quantitative analysis and technical analysis, and sometimes also used in fundamental analysis. REF

Scrieven, 1967 also defines “models” as a term loosely used to refer to a conception or approach or sometimes even a method (e.g., naturalistic, goal-free) of doing evaluation. Models are to paradigms as hypotheses are to theories, which means less general but with some overlap.

Keyword 3: Formative EvaluationSeels & Glasgow, 1990 (as sited by Michael Scriven 1967), coined the term "formative evaluation" which refers to the type of evaluation that occurs during the developmental stage of the instructional design process The instructional designer evaluates materials during the process of instructional development to determine where there are weaknesses in the instruction so that revisions can be made (Smith & Ragan, 1999).

Description of Formative Evaluation techniqueAs opined in the internet based discussion on Evaluating Socio Economic Development; Formative evaluation seeks to strengthen or improve a programme or intervention by examining, amongst other things, the delivery of the programme, the quality of its implementation and the organizational context, personnel, structures and procedures. As a change oriented evaluation approach, it is especially attuned to assessing in an ongoing way, any discrepancies between the expected direction and outputs of the programme and what is happening in reality, to analyzing strengths and weaknesses, to uncovering obstacles, barriers or unexpected opportunities, and to generating understandings about how the programme could be implemented better. Formative evaluation is


responsive to the dynamic context of a programme, and attempts to ameliorate the messiness that is an inevitable part of complex, multi-faceted programmes in a fluid policy environment.

The discussion further states that formative evaluation pays special attention to the delivery and intervention system, but not exclusively. In formative evaluation, the evaluator also has to analyze the intervention logic, the outcomes, the results and impacts.

Formative evaluation activities include the collection and analysis of data over the lifecycle of the programme and timely feedback of evaluation findings to programme actors to inform ongoing decision-making and action (i.e. it is a form of operational intelligence). It requires an effective data collection strategy, often incorporating routine monitoring data alongside more tailored evaluation activities. Feedback is primarily designed to fine-tune the implementation of the programme although it may also contribute to policy-making at the margins through piecemeal adaptation.

Evaluators conducting a formative evaluation ask many different kinds of questions and use a variety of methods to address them. Questions are commonly open-ended and exploratory, aimed at uncovering the processes by which the programme takes shape, establishing what has changed from the original design and why, or assessing soft organizational factors such as the extent of ‘buy in’ by practitioner staff to the programme’s goals and intended outcomes. . Formative evaluation questions also investigate the relationship between inputs and outcomes, which can involve the formulation and measurement of early or short-term outcome measures. These often have a process flavor and serve as interim markers of more tangible longer term outcomes.

The online discussion also revealed that formative evaluation lends itself most readily to a case study approach, using a qualitative mode of inquiry. There is a preference for methods that are capable of picking up the subtleties of reforms and the complexities of the organizational context and wider policy environment. Methods which might be used include stakeholder analysis, concept mapping, focus groups, nominal group techniques, observational techniques and input-output analysis. Formative evaluation’s concern with the efficiency and effectiveness of project management can be addressed through management-oriented methods like flow charting, PERT/CRM (Programme Evaluation and Review Technique and Critical Path Method) and project scheduling. The measurement of interim or short-term outcome measures, which capture steps in the theory of how change will be achieved over the long term, may involve construction of qualitative or process indicators and use of basic forms of quantitative measurement.

Formative evaluation may be planned and managed in a variety of ways. The prevailing practice has been to prioritize the information needs of central staff (policy makers, programme managers) as those primarily responsible for programme steerage, leaving unspecified the roles that local staff (local site managers, local practitioners) and clients can play in reshaping plans and strategies in response to feedback. Newer conceptions of formative evaluation (for example, the mutual catalytic model of formative evaluation outlined by Chacon-Moscoso et al, 2002) emphasize a more inclusive approach to the involvement of stakeholders, and as well seek to elicit their participation as collaborators in the evaluation process rather than simply as providers of information. The role of evaluator changes from one concerned with gathering data and communicating evaluation findings to one of engaging programme participants in a form of evaluative inquiry. Organizational actors are helped to generate their own data and feedback through collective learning processes.

Formative evaluation involves the following tasks:

identification of evaluation goals, planning data collection,


contributing to methodological choices,

making value judgments and

Generating evaluation findings.

Those undertaking a formative evaluation may need to be specific about the roles each stakeholder group can and should play, and how the knowledge of all stakeholder groups can be brought together in ways that contribute to improved programme performance. Evaluators need also to consider the different needs of stakeholder groups for different kinds of information, and to determine what kinds of feedback mechanisms are appropriate in each case. Programme managers, for example, may want evidence of the progress being made towards objectives and early identification of problematic areas of implementation where intervention is required. Formal presentations, tied into the decision-making cycle, are effective for this purpose. For local site managers and practitioner staff, the kind of evaluation findings they find useful are often those which illuminate the organization’s culture, policies and procedures and how these are impacting on their work, or the extent to which perceptions and experiences of the programme are shared by those delivering the programme and its recipients or beneficiaries.

They look to evaluation to enhance the quality of data that will allow them to make quick decisions in an environment where they are commonly being asked to do more and more with fewer and fewer resources. Evaluators need to create fora that bring local programme actors together to engage in dialogue, drawing on evaluation findings and reflecting on their own knowledge and experience. In essence, the evaluator is seeking to create a collective learning process through which participants share meanings, understand the complex issues of programme implementation, examine linkages between actions, activities and intended outcomes, develop a better understanding of the variables that affect programme success and failure and identify areas where they may need to modify their thinking and behaviors.

Formative Evaluation ModelsHaving defined the key words and described the technique, what is an Evaluation Models? Daniel, 2003 defines an Evaluation Model as an idealized or “model” view for conducting evaluations according to a defensible conceptualization of evaluation. He further states that a sound evaluation model provides evaluators and clients:

a link to evaluation theory a structure for planning evaluations

a framework for collaboration

a common evaluation language

a procedural guide

standards for judging evaluations

Types of Models: Evaluation Models has been categorised by different authors; some of which is discussed as follows:

Payne, 1994 outlined four types of Formative Evaluation Models as outlines below:


1. Management ModelsThe basic idea of the management approach is that the evaluator’s job is to provide information to management to help them in making decisions about programs, products, etc. The evaluator’s job is to serve managers (or whoever the key decision makers are).

One very popular management model used today is Michael Patton’s Utilization Focused Evaluation. Basically, Patton wants evaluators to provide information to primary intended users, and not to even conduct an evaluation if it has little or no potential for utilization. He wants evaluators to facilitate use as much as possible. Patton’s motto is to “focus on intended use by intended users.” He recommends that evaluators work closely with primary intended users so that their needs will be met. This requires focusing on stakeholders’ key questions, issues, and intended uses. It also requires involving intended users in the interpretation of the findings, and then disseminating those findings so that they can be used. One should also follow up on actual use. It is helpful to develop a utilization plan and to outline what the evaluator and primary users must do to result in the use of the evaluation findings. Ultimately, evaluations should, according to Patton, be judged by their utility and actual use.

2. Judicial ModelsJudicial or adversary-oriented evaluation is based on the judicial metaphor. It is assumed here that the potential for evaluation bias by a single evaluator cannot be ruled out, and, therefore, each “side” should have a separate evaluator to make their case. For example, one evaluator can examine and present the evidence for terminating a program and another evaluator can examine and present the evidence for continuing the program. A “hearing” of some sort is conducted where each evaluator makes his or her case regarding the evaluand. In a sense, this approach sets up a system of checks and balances, by ensuring that all sides be heard, including alternative explanations for the data. Obviously the quality of the different evaluators must be equated for fairness. The ultimate decision is made by some judge or arbiter who considers the arguments and the evidence and then renders a decision.

One example, which includes multiple “experts”, is the so called blue-ribbon panel, where multiple experts of different backgrounds argue the merits of some policy or program. Some committees also operate, to some degree, along the lines of the judicial model.

As one set of authors put it, adversary evaluation has “a built-in meta-evaluation” (Worthen and Sanders, 1999). A meta-evaluation is simply an evaluation of an evaluation.

By showing the positive and negative aspects of a program, considering alternative interpretations of the data, and examining the strengths and weaknesses of the evaluation report (meta-evaluation), the adversary or judicial approach seems to have some potential. On the other hand, it may lead to unnecessary arguing, competition, and an indictment mentality. It can also be quite expensive because of the requirement of multiple evaluators. In general, formal judicial or adversary models are not often used in program evaluation. It is, however, an interesting idea that may be useful on occasion.

3. Anthropological ModelsPayne includes under this heading the “qualitative approaches” to program evaluation. Qualitative researches tends to be exploratory, collect a lot of descriptive data; and take an inductive approach to understanding the


world (i.e., looking at specifics and then trying to come up with conclusions or generalizations about the what is observed). Payne points out that you may want to view the group of people involved in a program as forming a unique culture that can be systematically studied.

Payne treats several approaches as being very similar and “anthropological” in nature, including “responsive evaluation” (Robert Stake’s model), “goal-free evaluation” (developed by Scriven as a supplement to his other evaluation approach), and “naturalistic evaluation” (which is somewhat attributable to Guba and Lincoln, who wrote a 1985 book titled Naturalistic Evaluation). Again, what all these approaches have in common is that they tend to rely on the qualitative research paradigm.

In all of these approaches the evaluator “enters the field” and observes what is going on in the program. Participant and non-participant observation are commonly used. Additional data are also regularly collected (e.g., focus groups, interviews, questionnaires, and secondary or extant data), especially for the purpose of triangulation.

The key to Scriven’s goal-free evaluation is to have an evaluator enter the field and try to learn about a program and its results inductively and without being aware of the specific objectives of the program. Note that Scriven’s approach is useful as a supplement to the more traditional goal-oriented evaluation. Goal free evaluation is done by a separate evaluator, who collects exploratory data to supplement another evaluator’s goal-oriented data.

Payne talks about Robert Stake’s specific anthropological model, which is called “Responsive Evaluation.” Stake uses the term responsive because he wants evaluators to be flexible and responsive to the concerns and issues of program stakeholders. He also believes that qualitative methods provide the way to be the most responsive. He uses a somewhat derogatory (I think) term to refer to what he sees as the traditional evaluator. In particular, he labels the traditional evaluation approach pre-ordinate evaluation, which means evaluation that relies only on formal plans and measurement of pre-specified program objectives.

In explaining responsive evaluation, Stake says “an educational evaluation is responsive evaluation if it orients more directly to program activities than to program intents; responds to audience requirements for information; and if the different value-perspectives present are referred to in reporting the success and failure of the program” (Stake, 1975).

Payne also shows Stake’s “events clock” which shows the key evaluation activities and events, while stressing that they do not have to be done in a predetermined or linear order. Flexibility is the key. Go where the data and your emerging conclusions and opportunities lead you. Ultimately, the responsive evaluator prepares a narrative or case study report on what he or she finds, although it is also essential that the responsive evaluator present findings informally to different stakeholders during the conduct of the study to increase their input, participation, “buy-in,” and use of findings. Responsive evaluation is very much a participatory evaluation approach.

4. Consumer ModelsThe last approach discussed by Payne is the consumer approach. The primary evaluation theorist behind this approach is Michael Scriven. Obviously this approach is based on the consumer product metaphor. In other


words, perhaps evaluators can obtain some useful evaluation ideas from the field of consumer product evaluation (which is exemplified by the magazine Consumer Reports). As Payne mentions, the consumer approach is primarily summative. For example, when you read Consumer Reports, your goal is to learn if a product is good or not and how well it stacks up against similar products and whether you want to purchase it. In short, one looks at the merit and worth (absolute and relative) of a particular product. Note, however, that it is much more difficult to evaluate a social or educational program that it is to evaluate, for example, an automobile or a coffee maker. With an automobile or a coffee maker, you can easily measure its specifications and performance. A social program is a much more complex package, that includes many elements and that requires an impact assessment using social science research techniques to determine if the program works and how it works.

Summary of Major Evaluation Models: Strength, necessary cautions and other notes.

Models Description Strengths Cautions and other notes

Goals Based The evaluationidentifies whetherprogram/policy goals/objectives are met.

• Determines whether results align with goals/objectives, important for accountability reasons and to identify change.

• Does not examine unanticipated results.• Requires clearly articulated and measurable goals or objectives.• Does not necessarily identify if achieved goals/objectives are due to the program/policy.• May not examine the validity of the goals/objectives.• May ignore process issues.

Goal Free The evaluationidentifies all results of the program, whether anticipated or unanticipated.

• More likely to identify unanticipated results.

• May not fit with an accountability mandate.• May not provide enough criteria to assess the worth of a program/policy.• In practice goals based and goals free approaches are usually combined, that is, both expected and unexpected results are examined.

Theory Based(logic model)

A logic model is an evaluation tool that provides a way of illustrating a program with a diagram or picture. Usually, boxes and arrows are used to show how the program will be set up, its planned activities, and the results that are expected from it.This evaluation isguided by the program/policy theory, i.e. the logical relationships between all the parts.

• Encourages a greaterunderstanding of thefundamental mechanisms ofthe program/policy, i.e. what works when, where, why and for whom.• Decreases the chance of negative results.• Increases dialogue among avariety of stakeholder groupsand clarifies underlying assumptions.thisapproach may be a diversion orcause antagonism (Scriven 2007)and that it may emphasize thequantitative over the qualitative(Walden & Baxter 2001).

• Requires up front reflection time to identify underlying theories and links between processes, activities and outcomes.• Logic models may not capture all important aspects of a program/policy.• Less common criticisms are that thisapproach may be a diversion orcause antagonism (Scriven 2007)and that it may emphasize thequantitative over the qualitative(Walden & Baxter 2001).

Utilization Evaluation isstructured accordingto who will use the

• Pragmatic and flexible.• Increases chances evaluation results will be followed up on.

• May decrease role of groups other than primary users.


evaluation resultsand how they will use the results.

Collaborative Collaborativeevaluation is a broad category whichincludes a variety oftypes of evaluation,with varying degreesof participation and different purposes.

• Strong capacity building potential. Increases relevance of results.• Increases relevance of results.• Increases chances of resultsbeing followed up onbecause active participantsfeel more invested in the evaluation.• Greater range of perspectives represented in evaluation information.• May give voice toTraditionally

• Requires training of non professionals.• Time consuming.• May require larger amounts of funds.• May lack credibility to some.• Requires strong facilitation becauseof diverse perspectives andsituations.• Attaining equitable participation is challenging because people’s circumstances and degree of power vary so much.

BalancedScorecard

The BalancedScorecard approachto evaluation is basedon four linked areas:financial, customer, business process, learning and growth.

• Systematic.• Increases clarity and consensus around strategy.• Increases alignment between strategic objectives and actions.

• Top-down approach.• Possibility of faulty assumptions leading to negative results if a causal link between the perspectives is assumed.

AppreciativeInquiry (AI)

Appreciative Inquiryuses an asset-based approach to evaluation.

• Increases evaluation participation.• Increases use of results.• Builds learning and change capacity.• Supports and strengthens good performance.

• A debate exists regarding whether AI avoids addressing challenges.• It is not always appropriate.• Requires a special skill set to carry out well.• Program staff are more likely to be included than clients, increasing the power gap.

Externa l An evaluator notassociated with theorganization or program carries out the evaluation.

• Greater credibility.• Efficient.

• May be irrelevant to stakeholders.• May miss or misinterpret information.

Kirkpatrick Kirkpatrick’sapproach is toevaluate four levels sequentially: reaction, learning, behaviour, results.

• Straightforward. • Does not explore the why or how of results.

CIPP CIPP evaluationsfocus on programcontext, input, process, and product.

• Takes into account a rangeof environmental factors from politics to personalities.

• Time consuming.

Literature review on an Evaluation ModelsThere are various evaluation Models. Discussed below is a review of the CIPP Evaluation Model.

Daniel Stufflebeam’s CIPP Model is a typical example of a management Model, (Payne, 1994); which has been around for many years (e.g., see Stufflebeam, et al. 1971), and it has been very popular in education. In


this Model, he opined that evaluators should pay special attention to evaluation’s root term, value by addressing pertinent criteria of:

Merit: something’s excellence Worth: it’s excellence &utility Probity: it’s uncompromising adherence to moral standards Significance: it’s reach, importance, & visibility

Daniel Stufflebeam’s CIPP Model also upheld that evaluations should meet 4 main standards which are: UTILITY: informing intended users FEASIBILITY: maintaining procedural, political, and financial viability PROPRIETY: evaluating legally & ethically ACCURACY: producing valid findings

The CIPP Model originated in 1966. The national context included “Great Society” money & reforms, ESEA & OEO, Titles I and III projects, R & D labs and centres, Services for the handicapped, Congressional requirements for accountability.

New models were required, because traditional approaches didn’t fit the evaluation requirements as: Experimental design wasn’t feasible & didn’t address the right questions. Standardized tests weren’t keyed to needs & performance levels of disadvantaged students & did

not yield timely feedback. Site visits weren’t trusted to deliver unbiased findings.

Therefore, CIPP Evaluation Model was an attempt to align evaluation procedures, data, & feedback with project timetables & local, state, & national information requirements and serve decision-making and accountability needs. Targeted decision areas were:

Goal setting Proposal writing Project implementation Project recycling

The CIPP Model is a simple systems model applied to program evaluation. A basic open system includes input, process, and output. Stufflebeam added context, included input and process, and relabeled output with the term product. Hence, CIPP stands for context evaluation, input evaluation, process evaluation, and product evaluation. These types are typically viewed as separate forms of evaluation, but they can also be viewed as steps or stages in a comprehensive evaluation.

CIPP Evaluation Framework uses the following formative and summative questions:Formative Evaluation Summative Evaluation

Content What needs to be done? Were important needs addresses?Input How should it be done? Was a defensible design employed?Process Is it been done? Was the design well executed?


Product Is it succeeding? Did the efforts succeed?

Context evaluation includes examining and describing the context of the program you are evaluating, conducting a needs and goals assessment, determining the objectives of the program, and determining whether the proposed objectives will be sufficiently responsive to the identified needs. It helps in making program planning decisions.

Input evaluation includes activities such as a description of the program inputs and resources, a comparison of how the program might perform compared to other programs, a prospective benefit/cost assessment (i.e., decide whether you think the benefits will outweigh the costs of the program, before the program is actually implemented), an evaluation of the proposed design of the program, and an examination of what alternative strategies and procedures for the program should be considered and recommended. In short, this type of evaluation examines what the program plans on doing. It helps in making program structuring decisions.

Process evaluation includes examining how a program is being implemented, monitoring how the program is performing, auditing the program to make sure it is following required legal and ethical guidelines, and identifying defects in the procedural design or in the implementation of the program. It is here that evaluators provide information about what is actually occurring in the program. Evaluators typically provide this kind of feedback to program personnel because it can be helpful in making formative evaluation decisions (i.e., decisions about how to modify or improve the program). In general, process evaluation helps in making implementing decisions.

Product evaluation includes determining and examining the general and specific outcomes of the program (i.e., which requires using impact or outcome assessment techniques), measuring anticipated outcomes, attempting to identify unanticipated outcomes, assessing the merit of the program, conducting a retrospective benefit/cost assessment (to establish the actual worth or value of the program), and/or conducting a cost effectiveness assessment (to determine if the program is cost effective compared to other similar programs). Product evaluation is very helpful in making summative evaluation decisions (e.g., what is the merit and worth of the program? Should the program be continued?)

ConclusionIn the 1960s and 1970s, assistance for development focused on projects. Evaluation of these development projects was focused on efficiency and effectiveness. The 1980s showed a greater attention to structural adjustment policies, moving beyond evaluating at the project level to looking at programs. Since the 1990s, the international community has been developing partnership approaches to development assistance. Cooperative approaches include more stakeholders and more complex operations. In addition, donors are communicating with other donors and agreeing to work together to meet development goals, including working cooperatively on interventions. (IPDET Handbook, 2007)Since 2000, the task for evaluation has become even more complex. The Millennium Development Goals (MDGs) were developed and agreed upon by the international community. This new set of standards assists with cooperation among multilateral organisations, donors, and partner countries.


A variety of approaches and strategies have been developed to meet the changing requirements of development evaluation. Some models have been used and tested for many years and continue to be valuable. To meet the demands for fast, flexible, and participatory evaluations, models are recently being used that are somewhat different than traditional program evaluation.

No mater what model is chosen, each model still requires the same planning steps. All models define evaluation questions, identify measures, collect and analyze data, and report and use findings. The choice of evaluation model may be implemented in a variety of locations and may be implemented differently to meet local requirements.


REFERENCES

Barbara Kahan, (2008). Review of Evaluation Frameworks, Saskatchewan Ministry of Education.

Daniel L.S.(2003). The CIPP MODEL FOR EVALUATION Circa 2003: Update Review of Development Checklist to guide implementation.

Evaluation Models, Approaches, and Designs 2004 Chapter 5, pg. 101

Evaluating Socio Economic Development (2003): Methods and Techniques of Formative Evaluation. Retrived from www.evalsed.info.

Farell K., Kratzmann M., McWilliam S., Robinson N., Saunders S., Ticknor J., White K. (2002). Evaluation made very easy, accessible and logical

International Program for Development Evaluation Training (2007): Approaches to Development Evaluation IPDET Handbook Module 7.

Patton, M.Q. (1997). Utilization-focused evaluation: The new century text. Thousand Oaks, CA: Sage.

Payne, D.A. (1994). Designing educational project and program evaluations: A practical overview based on research and experience. Boston: Kluwer Academic Publishers.

Ron, O. (n.d.). Models and Methods for Evaluation York University, Toronto, Canada.

Scrieven, M. (1967). The methodology of evaluation (AERA Monograph series on curriculum evaluation, No. 1). Chicago IL: Rand McNally.

Yee, C. C. and Chen, Y.Y. (2009). World Academy of Science, Engineering and Technology 53: Performance Appraisal System using Multi-factorial Evaluation Model.


http://www.evalsed.info/

formative evaluation models 1

Documents

term outcome measures

local site managers

major evaluation models

primary intended users

cipp evaluation model

formative evaluation models

primary users

cipp model