DESIGNING AND CONDUCTING SUMMATIVE EVALUATIONS

Angel Jones

BACKGROUND• Formative Evaluation is used to improve the effectiveness of others.

• Summative Evaluation is used to make decision about the continuous use of instruction.

• Did the instruction solve the problem

• Is the instruction effective in solving the performance problem.

• Relationship between the instruction of interest and needs of the organization.

• Field trial is similar to the third phase of formative evaluartion but conducted to determine whether is produces the desired results for the decision maker.

OBJECTIVES• I can describe the purpose of summative evaluation

• I can describe the two phases of summative evaluation and the decisions resulting from each phase.

• I can design a summative evaluation for comparing alternative sets of candidate instructional materials

• I can contrast formative and summative evaluation by purpose and design.

WHAT IS SUMMATIVE EVALUATIONThe design of evaluation studies and the collection of data to verify the effectiveness of instructional materials with target learners.

SUMMATIVE EVALUATION EXPERT JUDGEMENT PHASEOverall Decisions

Do the materials have the potential for meeting the origination's needs?

Specific Decisions

Congruence Analysis: Are the needs and goals of the organization congruent with those in the instructions?

Content Analysis: Are the materials complete, accurate and current?

Design Analysis: Are the principles of learning, instructions and motivation clearly evident in the materials?

Feasibility Analysis: Are the materials convenient, durable, cost-effective, and satisfactory for current users?

SUMMATIVE EVALUATION FIELD TRIAL PHASE Overall Decisions

Are the materials effective with target learners in the prescribed setting?

Specific Decisions

Outcomes Analysis

Impact on Learners: Are the achievement and motivation levels of learners satisfactory following instructions?

Impact on Job: Are learners able to transfer the information, skills and attitudes from instructional setting to the job setting or to subsequent units of related instruction?

Impact on Organization: Are learners’ changed behaviors making positive differences in the achievement of the organization’s mission and goals.

SUMMATIVE EVALUATION FIELD TRIAL PHASE Management Analysis

1. Are instructor and manager attitudes satisfactory?

2. Are recommended implementation proceduers feasible?

3. Are costs related to time, personnel, equipment, and resources reasonable?

EXPERT JUDGEMENT PHASE OF SUMMATIVE EVALUATION

CONGRUENCE ANALYSIS• Obtain a clear description of the organization’s need

• Locate instructional materials that have potential for meeting the organization’s needs. Obtain a clear description of the goals and objectives of the instruction

and the target audience for which it is intended.

RESOURCES• Compare the organizations need vs. needs addressed in the

materials.

• Compare the target group vs. target group for the materials

• Compare the resources vs. requirements for implementing instructions.

QUESTIONS RELATED TO THE DESIGN OF QUALITY MATERIALSQuestions related to the design of quality materials.

1. Are the materials and assessments accurate and complete?

2. Is the instructional strategy adequate for the anticipated types of learning outcomes?

3. Can the materials be used effectively?

4. Are current users of the materials satisfied?

CONTENT ANALYSIS• Content expert may be necessary if you are not a content expert

in the materials you evaluate.

Strategies for content expert.

• Provide the expert with copies of all candidate materials to be judged for accuracy and completeness of the goals.

DESIGN ANALYSIS• Evaluate the adequacy of the components of the instructional

strategy included in the candidate materials.

• Checklist is the most thorough and time saving approach

• List the components of the instructional strategy in the far left column and use the remaining columns for recording information.

• Adopt criteria related to each component based on the type of learning outcomes.

UTILITY AND FEASIBILITY ANALYSIS• Available learners guide or syllabus

• Durability of material

• Special resources such as instructor capabilities, equipment or environments that are required.

• Interviewing the persons in the orgainzations who request the evaluation may help with the design.

SEQUENCE OF STAGES OF ANALYSIS DURING THE EXPERT JUDGEMENT PHASE OF SUMMATIVE EVALUATION

• Perform congruence analysis for organization and instruction.

• Perform content analysis on congruent instruction

• Perform design analysis on congruent, accurate, complete instruction.

• Perform feasibility analysis on well designed instruction

• Perform current user analysis

• Conduct field trials for the materials judged as most promising dureing the export judgement phase of summative evaluation.

FIELD TRIAL PHASE OF SUMMATIVE EVALUATION

CURRENT USER ANALYSIS• Data about the target learners in the other setting.

Entry skills; motivation for studying; pretest and posttest performance level; attitudes about the materials.

• Instructor’s perceptions of the materials

Are the materials easy to use; problems in implementing the materials; resources required; planning on continue using materials

PLANNING• Design Evaluation

• Outcomes Analysis

Describe resources, facilities, equipment needed

Describe ideal entry skills/characteristrics of target group

Describe number of groups and individuals needed

• Management Analysis

Describe skills/capabilities of instructiors or managers

Describe number of instructors needed

Plan and develop any training needed for instructors/managers.

PREPARING• Obtain instruments; Set schedule for instructions and testing;

Create/modify syllabus

• Obtain resources, facilities, equipment

• Select sample; Verify entry skills; Schedule learners

• Select instructors; Verify skills; Schedule instructors

• Provide training for instructors.

IMPLEMENTING/COLLECTING DATA• Adequate?

• Learner performance?;Learner attitudes?;Learner performance in job context

• Validity of implementation? Modifications?

• Training effective?

SUMMARIZING AND ANALYZING DATA• Describe problems by resource facilities, and equipment

• Item by objective analysis for group and individual; Cross test summary by group and individua by objective; Attitude summary

• Describe problems by instructor by objective

• Describe implementation problems by objective and activity

REPORTING RESULTS• Describe limitations of design.

• Recommendation and rationale

• Explanation, recommendations, rationale for learner, job and organization impact.

• Recommendations and rationale

• Recommendation and rationale

COMPARISON OF FORMATIVE AND SUMMATIVE EVALUATION

Formative Evaluation Summative Evaluation

Purpose Locate weaknesses in instruction in order to revise it

Document strengths and weakness in instruction in order to decide whether to maintain or adopt it

Phase or Stages One to OneSmall groupField Trial

Expert judgmentField trial

Instructional Development History

Systematically designed in-house and tailored to the needs of the organization

Produced in house or elsewhere not necessarily following a systems approach

Materials One set of materials One set of materials or several competing sets

Position of Evaluator Member of design and development team

Typically an external evaluator

Outcomes A prescription for revision instructions

A report documenting the design, procedures, results, recommendation, and rationale.

SUMMATIVE EVALUATION CHANGE AGENTAs a change agent , I conduct evaluation to make decisions about whether to maintain or adopt instructions. I will collaborate with the curiculm team to develop evaluation that will verify the effectiveness of the instruction material an target learners. As a change agent I will improve the effectiveness of my instruction with the data summary of the summative evaluation.

THE ENDBy: Angel Jones