Instructional Design and Content Development

Workshop

EVALUATING TRAINING EFFECTIVENESS

Define a program of evaluation Evaluation in learning Outline major reasons to conduct a

program evaluation Why conduct an evaluation? General evaluation models. Who is Kirkpatrick? Kirkpatrick’s four level learning

evaluation model Write questions for each level to assess

and evaluate the developed course.

Topics

Evaluation is a systematic method for collecting, analyzing, and using information to answer basic questions about a program.

Evaluation is valuable for strengthen the quality of the program and improve outcomes for the learners.

Evaluation answers basic questions about a program’s effectiveness, and evaluation data can be used to improve program services.

What is an evaluation?

Evaluation In Learning

- Evaluation means assessing the effectiveness and possible improvement of a course.

- Evaluation is part of any instructional design model.

- Evaluation provides review checkpoints for each phase of ADDIE that allow the instructor to evaluate the work that has been produced.

- Without completing this portion of ADDIE model, the e-course is incomplete because the course cannot be redesigned and improved without EVALUATION is implemented.

Evaluation finds out “what works” and “what does not work.”

Evaluation showcases the effectiveness of a program to the community.

Evaluation improves staff’s practice with participants.

Evaluation increases transfer of learning to behavior/performance in order to maximize program results.

Evaluation increases a program’s capacity to conduct a critical self assessment and plan for the future.

Evaluation builds knowledge for the out-of-campus time field.

MAJOR REASONS TO CONDUCT AN EVALUATION

The ADDIE model stresses the concept that good training programs require planning, review, and revision.

The evaluation phase focuses on gabs to mend them.

Increase transfer of learning in order to maximize learning / e-content results.

The evaluation phase measures the course's efficiency and locates opportunities to improve learners' performance.

Why conduct an evaluation?

Benefits of Evaluation

Effectiveness: It shows the progress made toward program goals and objectives.

Best Practices: It provides the ability to determine what program approaches are most effective.

Improvement: It provides ongoing assessment of program design and implementation to identify areas of improvement.

Impact: It demonstrates economic or human impact.

Accountability: It provides the base for interpreting an organization or programs’ worth to its stakeholders.

Promotion & Advocacy: It informs policymakers about programs successes.

Appraisal & Coordination: It gives managers the performance information to make better operational decisions.

General Evaluation Models

There are a lot of models clarifying the evaluation process and determining the information needs of the intended audiences of the evaluation, these models such as:

Tyler’s early conception of assessing attainment of program objectives.

Decision-Making Evaluation Approaches.

Naturalistic Evaluation Approaches

Kirkpatrick’s Four Levels for evaluating program effectiveness; which suggests the most appropriate evaluation methodology to be used.

Donald Kirkpartick is Professor of the University of Wisconsin in the USA and a past president of the American Society for Training and Development.

He has developed a very popular evaluation model that has been used by the training/learning community.

He focused on measuring four kinds of outcomes that should result from a highly effective training program.

Who is Kirkpatrick?

Kirkpatrick’s four levels of evaluating learning

REACTIONLEARNING

BEHAVIOR

RESULTS

Kirkpatrick’s model includes four levels or steps of outcome evaluation:

Level One: Evaluate Learner ReactionHow well did the learners like the learning

process?

The goal is find out the reaction of the learners to the instructor, course and learning environment.

The purpose is not to measure what the learners has learned, but whether the delivery method was effective and appreciated.

This type of evaluation can be incorporated at the end of the instruction and can be delivered online.

Level one - Evaluate Learner Reaction

General questions can include the following:• Did the instructor attend the sessions on time? • Did the instructor respond to learners’ comments and

questions? • Did the instructor deliver the information clearly and

smoothly? • Were there distraction?• Did the learners feel comfortable in the surroundings?

E-content questions can include the following:• Level of appeal of the instruction. • The ease of navigation and use of tools. • The ability of the course to motivate and retain interest. • The amount of interactive exercises. • The relevance of the objectives. • Quality and relevance of multimedia. • The ease of navigation and use of tools.

Level one - Evaluate Learner Reaction

Level two: Evaluate LearningWhat did learners learn? (the extent to which they gain knowledge, skills and attitude)

The goal is measuring learning results. In other words: did the learners

learn what intended to be taught?

Level two - Evaluate Learning

Measurement methods of level two include: Formal and informal testing.

Self assessments at the beginning (pre-test) and end (post-test) for learners.

http://www.reap.ac.uk/reap/public/papers//DN_SHE_Final.pdf

Interviews, observation and feedback.

Product Assessment. Creation of a project as an authentic assessment to showcase the knowledge and talents.

This form of assessment evaluates whether the learner can apply the learned skills or concepts in a concrete fashion.

Level two - Evaluate Learning

Level three: Evaluate PerformanceWhat changes in performance resulted from the learning

process?

To what degree participants apply what they learned during training when they are back on the job.

The goal is measuring “what happens when learners leave the classroom and return to their daily lives/job.

Did the learners put their learning into effect when back on the job/live?

Would the learners be able to transfer their learning to another person?

How much transfer of learning occurs?

It is the most obvious sign to evaluate the training program’s effectiveness.

Level three - Evaluate Performance

Measurement methods of level two include:

Allow time for a change in behavior to take place Testing Observation, interview and survey. The learner interactions on the job. Evaluate both before and after the program if that is practical

Level three - Evaluate Performance

Level four: Evaluate ResultsWhat are the tangible results of the learning process in terms improved quality, increased production, efficiency, etc.?

To what degree targeted outcomes occur as a result of learning event(s) and subsequent reinforcement

The goal is to find out if the training program led to final results, especially business or environment results

Level four outcomes include the major results that contribute to the well functioning of an organization.

Level four outcomes Improve quality of work, increase productivity and profits.

Across an entire organization it becomes more challenging and affect organizational and business performance, which cloud the true cause of good or poor results.

Level four - Evaluate Results

Measurement methods of level four include:

Methods of Measuring Learning Outcomes GridHow colleges and universities can measure and report on the knowledge and abilities their students have acquired during their college years is an issue of growing interest.

http://web.mit.edu/tll/assessment-evaluation/methods-of-measuring-learning-outcomes-grid.doc

http://www.elcamino.edu/academics/slo/docs/SLOFocusOnResults.doc

Course-embedded assessment/Assurance of Learning Standards.

http://www2.cortland.edu/dotAsset/f2e9f1ee-0100-40ab-b78d-48ead9c047d3.pdf

Level four - Evaluate Results

To Sum Up

Checklist for effective questions

Do Not Do

Use complex phrases Give clear instructions

Frame questions in the negative Keep questions structure simple

Use abbreviations, contractions or symbols Ask one question at a time

Mix different words for the same concept Maintain a parallel structure for all questions

Use “loaded” words or phrases Define terms before asking the question

Combine multiple response dimensions in the same question

Be explicit about the period of time being referenced by the question

Give the impression that you are expecting a certain response

Provide a list of acceptable responses to closed questions

Bounce around between topics or time periods Ensure that response categories are both exhaustive and mutually exclusive

Insert unnecessary graphics or mix many font styles and sizes

Label response categories with words rather than numbers

Forget to provide instructions for returning the completed survey

Ask for number of occurrences, rather than providing response

Save personal and demographic questions for the end of the survey

Activity

Based on your study of Kirkpatrick's model of learning / training programs evaluation, apply this model to evaluate your course using the 4 levels of Kirkpatrick's model.

o Write questions that explores the items within each level and describe how you will collect information from participants

http://www.kirkpatrickpartners.com/Portals/0/Training-Events/Program%20Prework/Kirkpatrick%20Four%20Levels(tm)%20Evaluation%20Online%20Certificate%20Manual.pdf

ONLINE DETAILED COURSE MANUAL

Thanks for Attending