CHAPTER 10

Designing and Conducting Formative Evaluations

Carolyn Jenkins-Haigler

BACKGROUNDA formative evaluation, evidence of an instructional program’s worth isgathered for use in making decisions about how to revise the program while itis being developed. This is why it is called "formative" evaluation, because theinstruction is in its developmental stages and is not yet "grown up". The ideais to find out if your newly developed course works at teaching the objectivesyou need to teach to the learners who need to learn them, before you present itto your target audience. In any given formative evaluation, you can find outhow to make your instruction more:

Effective

Efficient

Interesting/Motivating

Usable

Acceptable

You do this by carrying out procedures that will provide you with evidenceas to the effectiveness of your instruction. The emphasis is on collecting dataand revising the instruction.

OBJECTIVES

Describe the purposes for and various stages of formative evaluation of instructor-developed materials, instructor- selected materials, and instructor- presented instruction.

Describe the instruments used in a formative evaluation.

Develop an appropriate formative evaluation plan and construct instruments for a set of instructional materials or an instructor presentation.

Collect data according to a formative evaluation plan for a given set of instructional materials or instructor presentation.

THE CONCEPT OF FORMATIVE

EVALUATIONDefinition

The collection of data and information duringthe development of instruction that can be usedto improve the effectiveness of the instruction.

Purpose

To obtain data that can be used to revise theinstruction to make it more efficient andeffective.

THE ROLE OF SUBJECT-MATTER, LEARNING, AND

LEARNER SPECIALISTSIt’s important to have the instruction reviewed by specialists.

SME may be able to comment on the accuracy and currency of the instruction.

Learning specialist may be able to critique your instruction related to what is known about enhancing that particular type of learning

Learner specialist may be able to provide insights into the appropriateness of the material for the eventual performance context.

THE THREE PHASES OF FORMATIVE EVALUATION

I. One-to-One Evaluation

II. Small-Group Evaluation

III. Field Trial

ONE-TO-ONE EVALUATION

Criteria

Selecting Learners

Data Collection

Procedures

Assessments and Questionnaires

Learning Time

Data Interpretation

Outcomes

Purpose

To identify and remove the most obvious errors in the instruction

To obtain initial performance indications and reactions to the content by learners

Criteria

Clarity

Impact

Feasibility

CRITERIADuring the development of the instructional strategy and theinstruction itself, designers and developers make a myriad oftranslations and decisions that link the content, learners, instructionalformat, and instructional setting.

The one- to- one trials provide designers with their first glimpse of theviability of these links and translations from the learners’ perspective.

The three main criteria and the decisions de-signers will make duringthe evaluation are as follows:

1. Clarity: Is the message, or what is being presented, clear toindividual target learners?

2. Impact: What is the impact of the instruction on individuallearner’s attitudes and achievement of the objectives and goals?

3. Feasibility: How feasible is the instruction given the availableresources ( time/ context)?

SELECTING LEARNERS

One of the most critical decisions by the designer in theformative evaluation is the selection of learners toparticipate in the study.

This is not an experiment; there is no need for randomselection of large numbers of learners.

Actually, the designer wants to select a few learnerswho represent the range of ability in the group becauseprior learning or ability is usually one of the majordeterminers of ability to learn new skills andinformation.

DATA COLLECTION The first category, message, relates to how clear the basic message is

to the learner determined by such factors as vocabulary, sentencecomplexity, and message structures. Regardless of whether thelearner reads, hears, or sees the message, he or she must be able tofollow it.

The second category, links, refers to how the basic message is tailoredfor the learner, including contexts, examples, analogies, illustrations,demonstrations, and so forth. When these links are also unfamiliar tothe learner, the basic message will undoubtedly be more complex.

The third area, procedures, refers to characteristics of the instructionsuch as the sequence, the size of segment presented, the transitionbetween segments, the pace, and the variation built into thepresentation. The clarity of instruction may change for the learnerwhen any one of these elements is inappropriate for her or him.

PROCEDURES The typical procedure in a one- to- one evaluation is to

explain to the learner that a new set of instructional materials has been designed and that you would like his or her reaction to them.

You should say that any mistakes that learners might make are probably due to deficiencies in the material and not theirs.

Encourage the learners to be relaxed and to talk about the materials.

You should have the learners not only go through the instructional materials but also have them take the test( s) provided with the materials.

ASSESSMENTS AND QUESTIONAIRES

After the students in the one- to- one trials havecompleted the instruction, they should review theposttest and attitude questionnaire in the samefashion.

After each item or step in the assessment, ask thelearners why they made the particular responses thatthey did.

This will help you spot not only mistakes but also thereasons for the mistakes, which can be quite helpfulduring the re-vision process.

LEARNING TIME One design interest during one- to- one evaluation is

determining the amount of time required for learners tocomplete instruction, which is a very rough estimate,because of the interaction between the learner and thedesigner.

You can attempt to subtract a certain percentage of thetime from the total time, but experience has indicatedthat such estimates can be quite inaccurate.

DATA INTERPRETATION

The information on the clarity of instruction,impact on learner, and feasibility of instructionneeds to be summarized and focused.

Particular aspects of the instruction found to beweak can then be reconsidered in order to planrevisions likely to improve the instruction forsimilar learners.

OUTCOMES The outcomes of one- to- one trials are instruction that

1) contains appropriate vocabulary, languagecomplexity, examples, and illustrations for theparticipating learner;

( 2) either yields reasonable learner attitudes andachievement or is revised with the objective ofimproving learner attitudes or performance duringsub-sequent trials; and

( 3) appears feasible for use with the availablelearners, resources, and setting. The instruction canbe refined further using small group trials.

FIELD TRIAL

PurposeTo determine whether the changes/revisions in the instruction made after the small group stage were effective.

To see whether the instruction can be used in the context for which it was intended.

In the final stage of formative evaluation the instructor attempts to use a learning con-text that closely resembles the intended context for the ultimate use of the instructional materials.

One purpose of this final stage of formative evaluation is to determine whether the changes in the instruction made after the small group stage were effective.

SMALL-GROUP EVALUATION

Purposes

To determine the effectiveness of changes made following

the one-to-one evaluation.

To identify any remaining learning problems that learners

may have.

To determine whether learners can use the instruction

without interacting with the instructor.

FORMATIVE EVALUATION

To determine Weakness(es) in the Instruction

Focusing the design only on the goals and objectives ofthe instruction would be too limited.

Data on learners’ achievement of goals and objectiveswould be insufficient, though important, because thesedata will only provide information about where errorsoccur rather than why they occur.

FORMATIVE EVALUATION

HAS SIX STAGES Design Review

Expert Review

One-To-One

Small Group

Field Trials

Ongoing Evaluation

DESIGN REVIEW

Does the instructional goal match the problem identified in the needs assessment?

Does the learner & environmental analysis match the audience?

Does the task analysis include all the prerequisite skills?

Are the test items reliable and valid, and do they match the objectives?

EXPERT REVIEW

Is the content accurate & up-to-date?

Does it present a consistent perspective?

Are examples, practice exercises, & feedback realistic& accurate?

Is the pedagogy consistent with current instructionaltheory?

Is the instruction appropriate to the audience?

ONE-TO-ONE REVIEW

Is the message clear?

What is the impact on:learner attitudesachievement of objectives & goals

Feasibility of training

SMALL GROUP REVIEW

Look for the effects caused by the changes made in the one-to-one review

Identify any remaining learning problems

FIELD TRIAL REVIEWS

Look for effects in changes made in small group

Can the instruction be used in the context in which it was intended

ONGOING EVALUATION

Project Size

Life span of content

Audiences change

One-To-One

Small Group Tryouts

Field Trials

LEARNER EVALUATION

Do learners understand the instruction?

Do they know what to do during the practice & the tests?

Can they interpret graphics in the text?

Can they read all the material?

How much time does it take?

REFERENCE

SUMMARY

Formative evaluation of instructional materials is conducted to determine theeffectiveness of the materials and to revise them in areas where they areineffective. Formative evaluations should be conducted on newly developedmaterials as well as existing materials that are selected based on theinstructional strategy. Evaluations are necessary for both mediated andinstructor presented materials. The evaluations should be designed toproduce data to pinpoint specific areas where the instruction is faulty and tosuggest how it should be revised. An iterative process of formative evaluationcontaining at least three cycles of data collection, analysis, and revision isrecommended. Each cycle focuses on different aspects of quality. The firstcycle, one- to-one evaluation, is conducted to pinpoint gross errors in thematerials. These errors typically relate to both the clarity of vocabulary,concepts, and examples used, and the motivational value of all fivecomponents of the instructional materials. Evaluations can also be conductedwith content experts and individuals familiar with the characteristics of targetlearners. One- to- one evaluations must be conducted with representatives ofthe target population. An interactive interview process is used so theevaluator can learn what was wrong with the materials and why it waswrong.

THE END

Carolyn Jenkins-Haigler