DESIGNING AND CONDUCTING FORMATIVE

EVALUATIONS

OBJECTIVES

• Describe the purposes for and various stages of formative evaluation of instructor-developed materials, instructor- selected materials, and instructor- presented instruction. .

• Describe the instruments used in a formative evaluation. .

• Develop an appropriate formative evaluation plan and construct instruments for a set of instructional materials or an instructor presentation. .

• Collect data according to a formative evaluation plan for a given set of instructional materials or instructor presentation.

FORMATIVE EVALUATION DESIGNS

• Focusing the design only on the goals and objectives of the instruction would be too limited. Data on learners’ achievement of goals and objec-tives would be insufficient, though important, because these data will only provide information about where errors occur rather than why they occur.

5 GENERAL QUESTIONS THAT SHOULD BE ASKED ABOUT EACH COMPONENT OF THE MATERIALS?

• 1. Are the materials appropriate for the type of learning outcome? Specific prescrip-tions for the development of materials were made based on whether the objectives were intellectual or motor skills, attitudes, or verbal information. You should be con-cerned whether the materials you produced are indeed congruent with suggestions for learning each type of capability. The best evaluator of this aspect of the mate-rials would undoubtedly be an expert in the type of learning involved.

• 2. Do the materials include adequate instruction on the subordinate skills, and are these skills sequenced and clustered logically? The best evaluator for this area of questions would be an expert in the content area.

CONTINUED

• 3. Are the materials clear and readily understood by representative members of the target group? Obviously, only members of the target group can answer these ques-tions. Instructors familiar with target learners may provide you with preliminary information, but only learners can ultimately judge the clarity of the materials.

• 4. What is the motivational value of the materials? Do learners find the materials relevant to their needs and interests? Are they confident as they work through the materials? Are they satisfied with what they have learned? Again, the most appropriate judges of these aspects of the materials are representative members of the target group.

• 5. Can the materials be managed efficiently in the manner they are mediated? Both target learners and instructors would be appropriate to answer these questions.

ONE- TO- ONE EVALUATION WITH LEARNERS

• Criteria• Selecting Learners• Data Collection• Procedures• Assessments and Questionnaires• Learning Time• Data Interpretation• Outcomes

CRITERIA

• During the development of the instructional strategy and the instruction itself, designers and developers make a myriad of translations and decisions that link the content, learners, instructional format, and instructional setting. The one- to- one trials provide designers with their first glimpse of the viability of these links and trans-lations from the learners’ perspective. The three main criteria and the decisions de-signers will make during the evaluation are as follows:

THE THREE MAIN CRITERIA AND THE DECISIONS DE-SIGNERS WILL MAKE DURING THE EVALUATION

• 1. Clarity: Is the message, or what is being presented, clear to individual target learners? • 2. Impact: What is the impact of the instruction

on individual learner’s attitudes and achievement of the objectives and goals? • 3. Feasibility: How feasible is the instruction given

the available resources ( time/ context)?

SELECTING LEARNERS

• One of the most critical decisions by the designer in the formative evaluation is the selection of learners to participate in the study. This is not an experiment; there is no need for random selection of large numbers of learners. Actually, the designer wants to select a few learners who represent the range of ability in the group because prior learning or ability is usually one of the major determiners of ability to learn new skills and information.

DATA COLLECTION

• The first category, message, relates to how clear the basic message is to the learner determined by such factors as vocabulary, sentence complexity, and message structures. Regardless of whether the learner reads, hears, or sees the message, he or she must be able to follow it.

DATA COLLECTION

• The second category, links, refers to how the basic message is tailored for the learner, including contexts, ex-amples, analogies, illustrations, demonstrations, and so forth. When these links are also unfamiliar to the learner, the basic message will undoubtedly be more complex.

DATA COLLECTION

• The third area, procedures, refers to characteristics of the instruction such as the sequence, the size of segment presented, the transition between segments, the pace, and the vari-ation built into the presentation. The clarity of instruction may change for the learner when any one of these elements is inappropriate for her or him.

PROCEDURES

• The typical procedure in a one- to- one evaluation is to explain to the learner that a new set of instructional materials has been designed and that you would like his or her reaction to them. You should say that any mistakes that learners might make are probably due to deficiencies in the material and not theirs. Encourage the learners to be relaxed and to talk about the materials. You should have the learners not only go through the instructional materials but also have them take the test( s) provided with the materials.

ASSESSMENTS AND QUESTIONNAIRES

• After the students in the one- to- one trials have completed the instruction, they should review the posttest and attitude questionnaire in the same fashion. After each item or step in the assessment, ask the learners why they made the particular responses that they did. This will help you spot not only mistakes but also the reasons for the mistakes, which can be quite helpful during the re-vision process.

LEARNING TIME

• One design interest during one- to- one evaluation is determining the amount of time required for learners to complete instruction, which is a very rough estimate, because of the interaction between the learner and the designer. You can attempt to subtract a certain percentage of the time from the total time, but experience has indicated that such estimates can be quite inaccurate.

DATA INTERPRETATION

• The information on the clarity of instruction, impact on learner, and feasibility of instruction needs to be summarized and focused. Particular aspects of the instruction found to be weak can then be reconsidered in order to plan revisions likely to improve the instruction for similar learners.

OUTCOMES

• The outcomes of one- to- one trials are instruction that

• ( 1) contains appropriate vocabulary, language complexity, examples, and illustrations for the participating learner;

• ( 2) either yields reasonable learner attitudes and achievement or is revised with the objective of improving learner attitudes or performance during sub-sequent trials; and

• ( 3) appears feasible for use with the available learners, resources, and setting. The instruction can be refined further using small group trials.

FIELD TRIAL

• In the final stage of formative evaluation the instructor attempts to use a learning con-text that closely resembles the intended context for the ultimate use of the instructional materials. One purpose of this final stage of formative evaluation is to determine whether the changes in the instruction made after the small group stage were effective.

CHAPTER SUMMARY

• Formative evaluation of instructional materials is con-ducted to determine the effectiveness of the materials and to revise them in areas where they are ineffective. Formative evaluations should be conducted on newly developed materials as well as existing materials that are selected based on the instructional strategy. Evaluations are necessary for both mediated and in-structor presented materials. The evaluations should be designed to produce data to pinpoint specific areas where the instruction is faulty and to suggest how it should be revised.

CONTINUED

• An iterative process of formative evaluation con-taining at least three cycles of data collection, analy-sis, and revision is recommended. Each cycle focuses on different aspects of quality. The first cycle, one- to-one evaluation, is conducted to pinpoint gross errors in the materials. These errors typically relate to both the clarity of vocabulary, concepts, and examples used, and the motivational value of all five compo-nents of the instructional materials. Evaluations can also be conducted with content experts and individ-uals familiar with the characteristics of target learn-ers. One- to- one evaluations must be conducted with representatives of the target population. An interac-tive interview process is used so the evaluator can learn what was wrong with the materials and why it was wrong.