using the qualtrics research suite as a training lms

Using the Qualtrics® Research Suite as a Training LMSSIDLIT 2017 | LMS PRECONFERENCE

COLLEAGUE 2 COLLEAGUE

AUG. 2 , 2017

PresentationWhile learning management systems (LMSes) are a particular technology class, plenty of other technologies have been harnessed and retrofitted for learning purposes. Alternate LMSesinclude social media platforms like Twitter and survey systems. This presentation summarizes the uses of an online research (survey) suite, Qualtrics, for large-scale compliance-based trainings at Kansas State University. Some affordances of Qualtrics as a training LMS include the following:

a multimedia approach (including mobile),

a file upload approach,

logic functions,

branching logic for versioning a training (based on profiles or behaviors),

2

Presentation (cont.)

a scoring feature for grading performance,

logic tools to set standards for pass/fail,

integration with Google Translate,

security features,

built-in data analytics tools,

a dashboard for versioning different reports for different data clients,

an API for recording training completions and performances in information systems,

easy data export for external analytics,

and unique “skins”.

3

Presentation Order Basic Needs for Automated Trainings on a Campus

Qualtrics Research Suite Features

How Qualtrics is a Training LMS

Additional Desired Features

4

Basic Needs for Automated Trainings on a Campus

5

Automated TrainingsAn institution of higher education requires…

◦ a number of policy and safety compliance trainings for the entire campus as well as specialized trainings for specific populations (ability to roll out trainings to tens of thousands annually)…with a dynamic workforce and dynamic student population

◦ the ability to verify that the individual took the training and passed or failed (sufficient to the level of legal standards) (ability to respond to courts in lawsuits and to regulatory and oversight agencies)

◦ the ability to quickly update contents to a training to ensure that it is as accurate as possible for the particular topic (ability to ensure accurate training…to the minute, such as on issues of safety and security for lab workers, etc.)

◦ the ability to deliver trainings in an accessible way given the requirements from Section 508 and other federal accessibility requirements

6

Why Automated? Trainings have to be delivered to scale (to the full campus)

◦ Macro-level scale

◦ Micro-level scale

◦ All points in-between

Trainings have to be delivered to recurring high quality for pedagogical and training purposes

There have to be accurate records of the contents of the trainings, to ensure that they were designed to documented requirements

The trainings may be delivered synchronously or asynchronously, face-to-face or at a distance, and in various other permutations… The most convenient approach is to have online contents that may be versioned in different ways for different uses.

7

Why Automated? (cont.)

Trainings have to be delivered so that records are fully accurate and stand up to legal and regulatory agency scrutiny

◦ Legal requirements include protection of intellectual property, privacy protections, accessibility, and others

Regulatory agencies require three main things: ◦ Delivery of the proper information and skills, without inaccuracies

◦ Delivery of the proper trainings to the proper individuals at the proper times

◦ Clear record-keeping of all trainings taken and the performance of the individuals (in a way that stands up to courts)

The privacy of all involved has to be protected

8

About Qualtrics, Inc. and the QualtricsResearch SuiteABOUT THE COMPANY

Based in Provo, Seattle, Dallas, and Washington, D.C.; Dublin; London; Munich; Melbourne, Canberra, and Sydney

“Powers over 8,000 of the world’s leading brands and 99 of the top 100 business schools”

Features a range of tools, with Qualtrics Research Suite as one

◦ Many be instantiated with a range of features (some of which are not in the K-State instance and some of which will not be included in this slideshow)

ABOUT QUALTRICS RESEARCH SUITE

Offers a cloud-based research suite (software as a service / SAAS)

Survey tool typically used for qualitative, quantitative, and mixed methods research

9

Qualtrics Research Suite Features

10

Feature: Multimedia Integration Variety of question types (including visual ones like hotspot and heat map)

Ability to integrate audio, video, and other forms of multimedia

Ability to integrate external apps and websites through inline frames (iframes)

Ability to integrate external application programming interfaces (APIs) and applications with Qualtrics API

Ability to use scripts and codes to access external APIs (like Google Maps)

Strengths for Automated Training:

Ability to create a seamless multimedia-rich experience

Ability to enrich ways of interacting within the training

11

Feature: A File Upload Approach Ability to request learners / trainees to take a photo and upload it…or create a digital file (of various types) and upload those


Broadens the range of possible assessment types

Enables the use of smartphone and mobile device functionalities such as built-in cameras

12

Feature: Logic Functions Ability to set up notifications when particular events occur (email triggers)

Ability to populate various Contact Lists based on particular preconditions (contact list triggers)

Ability to set quotas for responses or particular occurrences (quotas)

Ability to set up math logic for variable questions (like for story problems) and with the use of Embedded Data (to capture the dynamic data)

Ability to use prior responses of the learner in a future customized question (piped text)

13

Feature: Logic Functions (cont.)


Ability for trainers to be aware of particular event occurrences

Ability to collect information based on particular preconditions

Ability to set ceilings on quotas

Ability to customize the training experience

Ability to add some non-deterministic elements (such as through math logic)

14

Feature: Branching Logic Ability to create branching logic based on particular conditions (like performance, like selection of a particular answer to a question, like identity-selection, or other feature)

Ability to customize training based on performance (or learner profile)

Ability to create different training experiences for people based on their level of understanding


Ability to make trainings unique to the individuals (based on their profiles, their performance, their selections, and other mixes of features)

Ability to build one training for a wide range of learners with differing learner needs

15

Feature: Scoring Ability to set values for particular questions

Ability to sum scores for a particular training (or segment of training)

Ability to change scoring levels after data collection for accurate processing ◦ For example, if a Likert scale is applied, and the valuing is opposite of what is desired, the scoring may

be reset to the correct values


Ability to add point values to each question…and to set granular conditionals for the earning of particular points within that question

16

Feature: Standards for Pass/Fail Ability to set a score threshold for pass / fail

Ability to set scoring tiers for grouping or classification

Ability to set defined granular conditions for passing (and / or)


Ability to set a threshold for “passing” and then sending learners off in different directions (based on performance)

17

Feature: Integration with Google Translate Ability to machine-translate an entire training into a number of different languages using Google Translate (with dozens of languages available and representable using UTF-8)

◦ But need vetting by native speakers to ensure the accuracy of the translations!

Easy ability to update the translations manually


Ability to tailor the language of the training to trainee’s first or preferred languages

18

Feature: Security Tools Access protections:

◦ CAPTCHA (Completely Automated Public Turing test to tell Computers an Humans Apart) requirements (to prove respondent’s humanity against ‘bot)

◦ Password protections

◦ Going to a training site from a given URL (uniform resource locator) but disallowing other accesses

◦ Making a training not findable by web browser spiders and crawlers

◦ No ballot box stuffing based on Internet Protocol (IP) address (so a person may not continuously take a training by returning to a particular link from a particular computer on the network)

◦ Setting time limits on any training

◦ Enabling access to a training based on an Authenticator feature or email invite (with unique links)

19

https://en.wikipedia.org/wiki/CAPTCHA

Feature: Security Tools (cont.)

Strengths for Automated Training:◦ Striving to ensure that whomever is signed up to take the training is the one actually doing so

◦ Protecting training contents against unintended downstream other uses

20

Feature: Built-in Data Analytics Data visualizations from online trainings

In-tool data analytics ◦ Cross-tabulation analysis (contingency table)

◦ Text analysis (rudimentary tool)

Ability to study results data (such as pass/fail) as a dummy variable


Ability to learn about learner performances

21

Feature: Dashboard for Reportage Dashboard for reporting summary data about trainings to different data clients

Dashboard enabling access to raw granular data (including single learner-level data, depending on privacy settings—but should never unmask learners)

Ability to conduct item analysis on respective questions in a survey


Can enable others to access the summary performance data (but will need to turn off “View Personal Data” feature at the global level, so personal identifiers and data are not accidentally leaked

22

Feature: Dashboard for Reportage (cont.)

Strengths for Automated Training (cont.):

Ability to capture an overall sense of learner performance◦ Can run a decision tree learning algorithm from the data to see if there are indicators of whether a

learner will “pass” or “fail”

Ability to capture the training efficacy

Ability to study question effectiveness through an item analysis ◦ Can know when learners stop a training and drop out and at which part of the training

23

Feature: API for Grades RecordingAbility to conduct a closed invite for a training…and then auto-capture learner performance for recording in external information systems

◦ Requires developer work through the Qualtrics API


Ability to handle grade recording to various information systems for college- or university-wide trainings and performance without the need for a human intermediary (human intermediaries may advertently or inadvertently introduce inaccuracies into the grade records)

24

Feature: Data Export Ease of data export for external analysis (such as in Excel and / or SPSS, or others)

An integration with NVivo 11 Plus (integration)


Ability to leverage the automated online training data to greater effect

Ability to create more data visualizations from the automated online training data

25

Feature: Unique “Skins” A variety of design looks-and-feels for laptops, desktops, and mobile devices


Branding trainings to particular learner groups based on look-and-feel

Branding trainings to particular “use cases”

26

About FeaturesA tool’s functions and features should not be considered in isolation but as part of a system.

As with all technology systems, there are tradeoffs with different parameter settings. People who would design to complex systems would do well to

◦ Understand the full implications of going with one setting over another (even the hidden implications)

◦ Test and retest all programming of the training ◦ Test and retest all programming of the training whenever certain changes are made because a cloud-based service may

occasionally have coding errors

◦ Keep open channels so that respondents can let a trainer know if something is not looking right

27

How Qualtrics is a Training LMS

28

Required Features of a Training LMS: Basics Learner enrollment

◦ Light learner tracking

◦ Light learner profiling

Learner and user identity protections

Delivery of digital contents ◦ Easy updatability

◦ Easy customizability

◦ Ability to deliver broadly

◦ Ability to target learners specifically

Data protection and storage

29

Required Features of a Training LMS: Basics (cont.)

Question creation and assessment ◦ Ability to provide real-time feedback

Accurate scoring per question and “grade” totaling ◦ Setting thresholds for passing

◦ Setting paths for training re-study and retakes

Accessibility features ◦ Accommodations for vision (visual acuity, color blindness, contrast, and others), hearing, symbolic

processing (dyslexia, and others), mobility, and other challenges

Mobile friendliness to enable broad device uses

Learner aids: cognitive scaffolding, effective feedback, access to review, access to downloadable learning objects

30

Required Features of a Training LMS: Basics (cont.)

Analysis of training efficacy / inefficacy (with downloadable data tables and effective data visualizations)

Data analysis

System robustness (to legal standards and legal defensibility)

Automated and accurate data populating to university information systems (for recordkeeping)

Seamless linking to other learning resources, through inline frames (iframes), links, uploads, and other integrations

31

Required Features of a Training LMS: AdvancedBranching logic

Customizations for learners ◦ Based on learner profiles

◦ Based on learner performance (prior or current)

Select email contacts (through triggers)…to enable further human support or interventions

Select contact list populating (through triggers)

Built-in data analytics ◦ Cross-tabulation analysis

◦ Text analysis

32

Takeaways Technology tools may be designed for particular target purposes, but there are other applications that may be applied. It helps to decouple a tool from its original stated purposes and to look at functionality alone.

A tool has to be tested from beginning-to-end to make sure everything functions as required.

It helps to have developers look at API (application programming interface) functions, too, to add capabilities to a particular tool instance.

33

Additional Desired Features

34

Additional Desired Features in Qualtricsas a Training LMSSocial profiles of learners

◦ Possible to build using {a} piped text feature

Heightened learner awarenesses of each other (for learning communities) ◦ Possible with third-party tool integrations

Ways to extract indicators that are correlated with learning ◦ Possible with decision tree learning in an external machine learning tool

More item analysis and learner performance features

Suggested ways to improve trainings

Easier integration of off-campus individuals to collaborate on the design of trainings

35

Additional Desired Features in Qualtricsas a Training LMS (cont.)

Ways to create learning communities

Some third-party content integration for learning

User-created content sharing (for training)

Ability to create course sequences that are linked and auto-sequenced

36

Contact and Conclusion Dr. Shalin Hai-Jew

◦ iTAC

◦ 212 Hale / Farrell Library

◦ Kansas State University

◦ [email protected]

◦ 785-532-5262

The presenter has no formal tie to Qualtrics, Inc.

37

mailto:[email protected]