olf1206 301 dolbert and heinz handouts finalppt[1]

Guiding Adult Learning with Assessments

Theresa Dolbert, Compuware & William Heinz, Openskylearning.

Produced by

December 7 & 8, 2006

301

Designing & Developing Online Assessments & Evaluations

Designing & Developing Online Assessments and Evaluations


Page 1Session 301 – Guiding Adult Learning with Assessments, Theresa Dolbert, Compuware & William Heinz , Openskylearning

Guiding Adult Learning with Assessments

“Testing is the most important intervention in learning”

Theresa Dolbert

William Heinz

Important: While you wait, please go to:

http://www.openskylearning.com/Game/GuidingLearning.html

Send us a chat (if you like) with your score on the game. The highest score gets a copy of the authoring tool for creating these games.

Agenda

Compuware Case StudyThe Value of AssessmentsJob Task AnalysisStoryboarding AssessmentsTools Tracking




How We Came To Guided Learning

We were charged with creating a Certification Program.We have a background in integrated testing and training and we knew the value of testing first.We had an audience with diverse backgrounds and needs and with the ability for learning without much assistance.We did not have the resources to create any unnecessary training.These factors led us to draw from external resources and use assessments to guide and track learning.

What are the chief limitations of Computer Based Training as compared with Instructor Led?

Answer with Chat




Why can’t people just go learn on their own?

Most people are not good at evaluating their own skill level, and are therefore poor at deciding whether or not training is required.Without help, most individuals that need training have no way of knowing what the requirements are for success.Most people are unable to learn without a structure that continually supports their motivation.Most people are busy, and it can be a bit frightening to take time away from today's problems in order to possibly improve tomorrow.

The Argument for Guidance

One can learn to tile a kitchen, program in Flash, play the recorder, or make sushi from the web.SkillSoft and other vendors offer thousands of courses.Information is available with 2 or 3 searches from Google on virtually any subject.

The future for Learning Professionals lies less in creating training and more in guiding learners through the right learning path.




Certification as Guidance

The Compuware Certification Program is one way to guide learning•It adheres to adult learning principles•It accommodates learning styles•It uses employee training time efficiently•It promises to produce a continuous learning program for 5000 employees.

Compuware Certification Goals

Provide a learning compass for Compuware Professional Staff:

• Indicate which skills are most marketable.• Provide a means to assess the minimum

requirements associated with those skills.Help to ensure that Professional Staff are qualified for assignment.Educate all professional staff about Compuware’s products so that even if they are not using those products they can become a passive sales force for the products.




The Compuware Certification Components

External Certifications (e.g. MCSE or BrainBench)Web Based Training (SkillSoft)Web Based AssessmentsInstructor Led TrainingVirtual Performance Based Exercises

Testing Can be Funhttp://www.openskylearning.com/Game/GuidingLearning.html




The Value of Testing

A study from Black and William on formative testing has shown that testing can increase learning more than any other single educational intervention, and formative testing, when applied correctly, increased the learning rate by 70%. [1]Putting the Task First doubles learning speed (Instant Context and Motivation).

•[1]Assessment in Education, Vol. 5, No. 1,March 1998

The 3 Stages To Guided Learning

Determining what people should learn•Job task analysis•Assessments to see what is already known

Providing learning opportunities and incentives Tracking to make sure that the learning occurred




What Are The Costs Of Training With The Wrong Material?

Three types of wrong material•Already known•Not related to success•Aimed at incorrect Mastery Level.

The cost is a workforce that does not want to learn from your training – not to mention the wasted resources invested in developing irrelevant materials.The cost to you is a devaluation of your training department.A Job Task Analysis can help to align your material with Job success.

How To Conduct A Job Task Analysis

Put together a potential skills listGet acceptance from SMEsProduce a survey that gathers data for each skill for•Importance•Frequency•Difficulty•Mastery level required

Analyze data for mean and standard error for each skill.Define a decision criteria for which skills you will test and train and which ones may be omitted.

www.asptrain.com




Creating the SurveyJTAAS lets you import from Excel.

•What obstacles prevent Learning Professionals from performing a Job Task Analysis.

Answer in chat.




Performance Based Testinghttp://www.openskylearning.com/Production/word2003/

Considerations For Good Assessments

Performance-based testing• Simulate the activity performed on the job

Bloom’s taxonomy• Knowledge: Terms• Comprehension: Concepts• Application: apply • Analysis: discriminate • Synthesis:• Evaluation:

The assessment storyboard spreadsheetEvidence centered design




Entering Data in the Test Storyboard• Follow ECD

•Adhere to blooms Taxonomy

•Create 2 equivalent forms simultaneously

•Make it possible for SMEs to design the tests

The Assessment Storyboard Spreadsheet




Terminology Questions•SMEs list the terms

•Rank for Importance, Frequency and Difficulty

•Sort by importance and keep top 20.

•Sort by difficulty

•Odd items are form A; Even are Form B

Analysis Questions




Testing Tools Used By Compuware

•QA Assessment•http://www.openskylearning.com/QA%20Test%20v2/start.htm

•Captivate•Right click

•http://www.adobe.com/devnet/captivate/articles/rightclick.html

•(New version for Captivate 2)•Autoscroll

•Hatsize (the virtual sandbox)

Low Stakes Test Analysis

Expert Review – Allow comments on each question.Pass Rates - Make sure that pass rates are reasonableP-Values (Probability of a right answer on each question)Point-Biserial




Point-Biserial Analysis

Provides a correlation for how individuals do on each item relative to how they do on the whole test.Helps to see items that:•Have more than one factor

•Clever but not practiced•Practiced but not clever

•Are too easy or too hard•Are not good indicators of success (bad items)

How to do a Point-Biserial with Excel•http://www.eddata.com/resources/publications/EDS_Point_Biserial.pdf

Compuware’s Approach to Tracking

OnTrack Online•Classroom Training•External Certifications•Internal Assessments•SkillSoft CBT•Employee and Management View




Adult Learning Principles

Need to have a reason for learning something Learn best when the topic is of immediate value Value autonomy

Need to learn experientially Approach learning as problem solving Have a variety of backgrounds Value self-esteem

Knowles, M. The Adult Learner: A Neglected Species

Learner OrientationsDr. Maggie Martinez

TransformingPerformingConformingResistant

Transforming Learners: Use personal strengths, self-motivation, strong emotions, persistence, and abstract or holistic thinking abilities to self-manage their learning.




Compuware's Guided Learning Approach

Most Compuware employees are transforming learners.A certification program presents a set of appropriate problems.The program will direct and compel a continuous learning environment.This will put new demands on training.The result will be a more knowledgeable work force.

Which great man made these statements?•“You cannot help men permanently by doing for them what they could and should do for themselves”.

•“You cannot build character and courage by taking away a man's initiative and independence.”




Our OpportunityThe web makes it easy to get information.It is easier now to create interactive learning.Our job is to create guidance and motivation for learning.Tracking enables motivation. Tracking cannot occur without some form of testing.Testing needs to be focused on job success and the best tests are the ones where the tasks are similar to what is required on the job. With regard to learning, formative Testing is more effective and less threatening than summative Testing.

•From the perspective of Guided Learning…•What might we have done better?•What did we forget to do?

Answer in chat.




Links

Materials from Presentation (and more) •http://www.OpenSkyLearning.com•William Heinz (530) 265-3213•[email protected]

Compuware –•www.Compuware.com

Job Task Analysis (JTAAS)–•http://www.asptrain.com

olf1206 301 dolbert and heinz handouts finalppt[1]

Documents