Developing Evaluation Instruments

Evaluations

• Formative – How are we doing?

• Summative – How did we do?

• Confirmative – How are we still doing?

Formative Evaluation

• Even excellent plans and concepts can be improved– Gauge success by classroom program

presentation– Informs the instructor how well instructional

package is serving the objectives as it progresses

Formative Evaluation

• Program of instruction is evaluated through– Test results– Learner reactions/comments– General observation of learners during class– Subject-matter expert reviews– Colleague suggestions

• Quality control of the development process

Formative Evaluation

• Questions that may be asked– Level of learning acceptable compared to

objectives?– Learners using knowledge to perform skills?– Time element acceptable?– Activities appropriate?– Tests measuring objectives?

Summative Evaluation

• Measures success of major outcomes

• Usually follows all instruction, projects and testing in a program

Summative Evaluation

• Measures– Learning efficiency (material mastered/time)– Program cost (Development and delivery)– Continuing expenses– Student comments/reactions/evaluations– Long-term benefit of program

Confirmative Evaluation

• Although instruction initially effective, some problems appear over time

• Tracks learner experiences over time to judge program’s validity

• Relies upon numerous data-collection instruments (interviews, performance assessments, etc.)

Confirmative Evaluation

• Questions to investigate– Do learners continue to perform correctly over

time?– Do materials still meet original objectives?– How has technology and/or resources, trends,

and attitudes changed since inception? – How to best meet clients’ needs over time?

Confirmative Evaluation

– Changes needed in materials? What are the costs?

– If instruction not working as well as before:• Should instruction continue as is?

• Should it be revised?

• Should it be terminated?

• What might replace it?

Validity

• The test assesses what it is supposed to measure – For example:

• Performance tests assess processes and outcomes relating to skills or competencies

• Course attitude surveys need to measure reactions to the course

Reliability

• The test will produce consistent results whenever used

• More questions relating to specific objectives = more reliable test

• Standardized delivery of test

• Scoring methods – the less subjective, the better

Pretesting

• Judges the learner’s preparation to study the material (course or topic)

• Determines competencies already mastered

Testing for Prerequisites

• Standardized paper-and-pencil test

• Performance observations

• Questionnaire (Looks at learner background, training, and experiences)

• Review of past work experience

• Interview past supervisors/managers

Objective tests

• Multiple choice

• True or False?

• Matching items

Constructed-Response

• Make learner plan answers and express them in their own words

• Short-answer

• Essay questions

• Problem solving

Skills and Behavior

• Actions observed

• Process, product, or both?

• Constraints/limitations?

• Testing conditions simulated or realistic?

The Rubric

• General assessment of overall product

• Certain elements attain points

• More objective tool

From Designing Effective Instruction 4th Ed., Morrison, Ross, & Kemp, 2004.