online survey and assessment techniques dr. curtis j. bonk associate professor, indiana university...
TRANSCRIPT
Online Survey and Assessment Techniques
Dr. Curtis J. Bonk Associate Professor, Indiana
UniversityPresident, CourseShare.comhttp://php.indiana.edu/~cjbonk,
Pause…What tools do you use
or know about?
Collecting Evaluation Data
& Online Evaluation Tools
Types of Data Collection
• Direct Observation in Work Setting– By supervisor, co-workers, subordinates,
clients
• Collect Data By Surveys, Interviews, Focus Groups– Supervisors, Co-workers, Subordinates,
Clients
• Self-Report by learners or teams• Email and Chat• Online Tests• E-Portfolios or Learning Logs
I. Online Testing Tools
(see: http://www.indiana.edu/~best/)
Choice: Select companies that
specialize in online assessment.
Or: Use what the courseware package gives ya…
Some things in life are free…!
Test Selection Criteria
(Hezel, 1999)• Easy to Configure Items and Test• Handle Symbols• Scheduling of Feedback
(immediate?)• Provides Clear Input of Exam
Dates• Easy to Pick Items for Randomizing• Randomize Answers Within a
Question• Weighting of Answer Options
More Test Selection Criteria
• Recording of Multiple Submissions
• Timed Tests• Comprehensive Statistics• Summarize in Portfolio and/or
Gradebook• Confirmation of Test
Submission
More Test Selection Criteria
(Perry & Colon, 2001)• Supports multiple items types—multiple
choice, true-false, essay, keyword• Can easily modify or delete items• Incorporate graphic or audio elements?• Control over number of times students
can submit an activity or test• Provides feedback for each response
• Flexible scoring—score first, last, or average submission
• Flexible reporting—by individual or by item and cross tabulations.
• Outputs data for further analysis• Provides item analysis statistics
(e.g., Test Item Frequency Distributions).
More Test Selection Criteria
(Perry & Colon, 2001)
Web Resource: http://www.indiana.edu/~best/
II. Electronic Gradebooks (Vockell & Fiore, 1993)
• Calculate scores, store info• Weight scores• Flag students with certain
characteristics• Print reports by individual or
group• Provide prompt feedback• But inflexible, impersonal, &
can be incorrect
III. Online Survey Tools for
Assessment
Perhaps you have heard of my survey
techniques?
Survey Tool Features
• Maintain email lists and email invitations• Conduct polls• Adaptive branching and cross tabulations• Modifiable templates• Maintain library of past surveys• Publish reports• Technical support, chat advice• Different types of accounts—hosted,
corporate, professional, etc.
Still More Online Survey Features
• Online help, FAQs, demos, tutorials, white papers, glossaries, site maps, tips, overviews
• Website evaluation tools or options• Supports different types of items
(Likert, multiple choice, forced ranking, paired comparisons, etc.)
• Customizable audience lists• Launch and close surveys
More Online Survey Features
• Online viewing of assessment results and question analysis (including various graphical displays)
• Easy to author surveys• Instant feedback to participants• Custom reporting, editable images
and background, personalized greetings
• Storage of results in Access or SQL
Still More Features
• Ability to start and stop a survey• Reports on subgroups• Randomize q’s to reduce bias• Easy email reminders• Personalized banner or logo• Expanded question options• Ability to copy surveys
Web-Based Survey Advantages
• Faster collection of data• Standardized collection format• Computer controlled branching
and skip sections• Graphics can reduce fatigue• Easy to answer clicking• Wider distribution of
respondents
Web-Based Survey Problems: Why Lower
Response Rates?• Low response rate• Lack of time• Unclear instructions• Too lengthy• Too many steps• Can’t find URL• Perceived as aggressive
Web-Based Survey Solutions: Some Tips…
• Send second request• Make URL link prominent• Offer incentives near top of request• Shorten survey, make attractive,
easy to read• Credible sponsorship—e.g.,
university• Disclose purpose, use, and privacy• E-mail cover letters• Prenotify of intent to survey
Tips on Authentification
• Check e-mail access against list• Use password access• Provide keycode, PIN, or ID #• (Futuristic Other: Palm Print,
fingerprint, voice recognition, iris scanning, facial scanning, handwriting recognition, picture ID)
Sample Survey Tools• Zoomerang
(http://www.zoomerang.com)• IOTA Solutions
(http://www.iotasolutions.com)• QuestionMark
(http://www.questionmark.com/home.html)
• Survey Solutions from Perseus (http://www.perseusdevelopment.com/fromsurv.htm)
• Infopoll (http://www.infopoll.com)
Sample Survey Tools
• Active Feedback • (http://www.activefeedback.com/af)• Majon International • (http://www.coolsurveys.com)• EZSurvey from Raosoft • (http://www.raosoft.com/)• SurveyShare
(http://SurveyShare.com; from Courseshare.com)
Zoomerang Steps
1. Register: Sign Up for Basic Membership (SKIP FOR NOW!)
2. Click on new survey.
3. Click on survey template (e.g., “Education” and Professor)
4. Delete, modify, or add 1-2 items.
5. Submit survey.
Zoomerang Steps
6. Modify Web greeting and submit.
7. Pick option #1 for submission.
8. Type Survey invitation & email message
9. Enter email addresses for potential survey respondents
10. Decide on how share results
Zoomerang Steps
11. Launch survey
12. Click on “My Surveys” and browse
13. Click on “Email List” and browse
14. Click on “Account Info”
15. Other explorations…
SurveyShare Steps
1. Register: Sign Up (perhaps take Quick Tour, Flash version)
2. Login3. Click on Create Survey4. Check out existing templates5. Select one of the templates6. Add one item, delete one item, modify
one item
SurveyShare Steps
7. Edit Survey Introduction and Title
8. Edit Closing and Submit
9. Select Survey Invitation Via Email
10. Type emails to complete the survey
11. Click add (to move emails to right)
12. Give email subject and message
SurveyShare Steps
13. Enter email subject line and text
14. Preview survey
15. Activate survey
16. Go to MySurveys
17. Click on Survey Info
18. Go to mailing lists
SurveyShare Steps
19. Add a person
20. Add list
21. Check out resources (books)
22. Check out help, quick tour, and sitemap
23. Perhaps create a survey from scratch
24. Perhaps create a survey from “My Previous Surveys”
Any Reactions to Online
Surveys???
Fitting Surveys into Evaluations
Plans
Bonk’s Evaluation Plan…
Considerations in Evaluation Plan
1. Student
2. Instructor
3. Training
4. Task5. Tech Tool
6. Course
7. Program
8. University or
Organization
What to Evaluate?
1.Learner—attitudes, learning, use, performance.2.Instructor—popularity, course enrollments.3.Training—internal and external components.4.Task--relevance, interactivity, collaborative.5.Tool--usable, learner-centered, friendly,
supportive.6.Course—interactivity, participation, completion.7.Program—growth, long-range plans.8.Organization—cost-benefit, policies, vision.
The 4 Levels
•Reaction•Learning•Behavior•Results
Figure 26. How Respondent Organizations Measure Success of Web-Based Learning
0102030405060708090
Learner satisfaction Change inknowledge, skill,
atttitude
Job performance ROI
Kirkpatrick's Evaluation Level
Pe
rce
nt
of
Re
spo
nd
en
ts
Ziegler, April 2002, e-Learning
“…the key is not to measure every possible angle, but rather to focus on metrics that are pragmatic and relevant to both human and business performance at the same time.”
Some Final Advice…
Or Maybe Some Questions???