collecting user preferences for web-based medline training: a

Mid-Atlantic Chapter of the Medical Library Association2002 Annual Meeting

Collecting User Preferences for Web-Based MEDLINE Training:

A Capitol Idea

Kelly Near

Andrea Horne

Karen Grandage

Claude Moore Health Sciences LibraryClaude Moore Health Sciences Library

Overview

• Proliferation of electronic resources

• Busy health professionals and students

• Need for continual end-user training and support, available any time & any place

Tutorial Study Objectives:

Compare user preferences

Gain experience with new technologies for delivering online training

Evaluate the educational effectiveness of each tutorial format

Evaluate the cost/benefit ratio for each tutorial format

Background

• Advancements in technologyWeb frames

Streaming audio & video

Flash

Java Script Applets

Background –MEDLINE Training

• Still our “bread and butter” resource

• Medical and Nursing School curriculum training sessions

• Monthly classes

• Individual consultations

Planning

• Always allow more time than you think you’ll need!

• We began with the assumption that online tutorials were an important part of educational delivery

Planning

• Tutorial formats were chosen based on - learning style theory

our skill & comfort level with the necessary technology

Tutorial A

• Interactive

• Created in Web frames

• Search instructions in the left frame

• “Live” version of the MEDLINE database active in the right frame

Tutorial A

Tutorial B

• Short animated screen capture illustrations of commonly performed database tasks

• Audio narrative

Tutorial B

Tutorial C

• Classroom format

• Streaming video and audio

• Clips of an instructor interspersed with screen shots of the database

Tutorial C

Study Design• Wanted our study participants to be a

cross-section of our usual MEDLINE class students

• 10 participants were randomly assigned to each tutorial

• We did not determine their MEDLINE searching abilities prior to the testing

• None of the tutorials were meant to be as detailed as an actual class

• Debate: - should they answer specific search questions, such as “what is the symbol used to truncate in Ovid MEDLINE”

or- should they conduct a complete search on a given topic, and have it measured against a “gold standard”?

Search Question

• Use MEDLINE

1996 – present

Perform a comprehensive search on the question:

Is there a connection between head injuries and depression in children?

Six Steps to a Good Search

• Rated each search on a 6 point scale1 point for selecting the MeSH term

“craniocerebral trauma”

1 point for exploding it

1 point for selecting the MeSH term “depression”

1 point for NOT exploding it!

1 point for combining the terms with AND

1 point for applying the age limit

Gold Standard

Explode !

Developing Tutorial A

Key: =development time $=money spent

• Key features: HTML frames Ovid Jumpstarts

• Products used: HTML Assistant Notepad

• Cost = 0• Challenges:

Condensing a full class down to key learning points and brief descriptions

Developing Tutorial B

• Key features: Short video &

audio clips with text

• Products used: SnagIt! Microphone and

headset

• Cost = $39.95 for single

user copy

$Key: =development time $=money spent

Developing Tutorial B

• Challenges:Time required to write a concise script

Time for the multiple takes needed to do a clean audio track …

especially using non-union help!

Take 5!

Starting a Subject Search This example will demonstrate a search for articles about oral cancers in teenage boys who use chewing tobacco.Begin by separating your search topic into individual concepts.Type your first search concept into the box labeled Enter Keyword or Phrase. For our search, we begin by typing in the keywords "oral cancer".Be sure that the checkmark is in the box by Map Term to Subject Heading

Next TopicReturn to Tutorial Home

Developing Tutorial C

• Key features: Used our classroom as

the “stage”

Hired a “professional actress” so fewer retakes were necessary

• Products used: Adobe Premiere

• Cost = $399

$$$$

Key: =development time $=money spent

Developing Tutorial C

• Challenges:

iMovie and Final Cut Pro didn’t allow us to intersperse SnagIt video clips with instructor video

Next tried videotaping the screen in the classroom as the MEDLINE search was being done

Adobe Premiere allowed us to combine the video and the SnagIt clips into one product

Recruitment

• Sent email to medical students in all 4 years inviting them to participate in the study

• Posted recruitment flyers around the Library, Medical School and neighboring Research Buildings

• Word-of-mouth seemed to help

• So did the FREE lunch!

Pre-Testing

• Pre-tested the process with 3 library staff

• Identified problems with inconsistent instructions

• Refined test questions and tutorials based on this input

Testing

• Randomly assigned 10 participants to each tutorial

• The tutorials were taken in a library classroom

• After completing the tutorial, they were each given the same search to complete in Ovid MEDLINE

Interviews

• Participants were asked12 questionsExamples

Have you ever taken a MEDLINE class before?

Did you have any technical problems with the tutorial?

Did you refer back to the tutorial to complete the steps for the test search?

Free Lunch

• Testing over the middle of the day worked very well as the lure of a

FREE LUNCH

was irresistible to all!

Test Scores

0

1

2

3

4

5

6

A (4.5) B (5.1) C (4.9)

A (4.5)B (5.1)C (4.9)

Number of questions

Number correct

ANOVA Test Results

Sum of

Squares df

Mean Square F Sig.

Between Groups

Within Groups

Total

1.867

30.300

32.167

2

27

29

.933

1.122

.832 .446

P value needs to be below 5% - this is 45%

Explode

0

2

4

6

8

10

Tutorial A(6/10)

Tutorial B(10/10)

Tutorial C(7/10)

Tutorial A (6/10)Tutorial B (10/10)Tutorial C (7/10)

Number

of

Participants

Number Exploding

Chi Square Analysis

• Chi Square analysis included 3 separate tests

• Only 1 showed a significant difference

• No significant statistical differences were found overall

Tutorial Preferences

• Overall, interview results showed positive acceptance for all three tutorial types

A B C

Reported Learning Preferences

Online TutorialOnline TutorialClassClass

PrintedPrinted

22

4

3

Lessons Learned …

• Only 8 users had not used MEDLINE before

• We looked at this data to determine if there was a difference in the scores between new and experienced MEDLINE users

• Did not find any difference between the 2 groups

Lessons Learned …

• May need a larger sample size

• AVI files (from SnagIt!) are too large

• Try Snapz Pro (Macintosh only) for video captures

One Caveat …

• Ten days before we were to start testing - we found we needed approval from the University’s Human Investigation!

Conclusions

• Scores = no difference

• Next, we looked at production costs and time and found …

MAJOR DIFFERENCES

…and the Winner is -- Tutorial A!

Future Directions

Tutorial A = the one to trust!

Questions ?

Collecting User Preferences for Web-Based MEDLINE Training:

A Capitol Idea

Kelly Near, Andrea Horne, Karen Grandage