The Impact of Conversational Navigational Guides on the

Learning, Use, and Perceptions of Users of a Web Site

Art Graesser, Tanner Jackson, Matthew Ventura, James Mueller, Xiangen Hu, and Natalie Person

This research was directly supported by contracts from ONR and IDA, and was partially supported from grants by NSF and the DoD.

Outline (The next 20 minutes)

• HURAA

• Experiment

• Results

• Conclusions

What is HURAA?

• HURAA (Human Use Regulatory Affairs Advisor ) is a web-based facility that provides help and training on the ethical use of human subjects in research.

• HURAA is based on documents and regulations from United States Federal agencies.

HURAA (Features)

• Hypertext• Multimedia• Animated navigational guide• Lessons with case-based and explanation-based

reasoning• Help modules• Context-sensitive FAQs (Point & Query)• Glossaries• Archives• Natural language queries

HURAA (Modules)

• Introduction

• Historical Overview

• Lessons

• Explore Issues

• Explore Cases

• Decision Consequences

• Query IRB Documents

Outline• HURAA

• Experiment

• Results

• Conclusions

Experiment• 155 students

• Between subjects:– Navigational Guides

• Full

• Voice

• Text

• None

• Within subjects:– Two Phases

• Acquisition phase

• Test phase

Experiment (Guides)

• Full guide (agent and voice)

Experiment (Guides)

• Full guide (agent and voice)

• Voice guide (synthesized speech only)

Experiment (Guides)

• Full guide (agent and voice)

• Voice guide (synthesized speech only)

• Print guide (text message only)

Experiment (Guides)

• Full guide (agent and voice)

• Voice guide (synthesized speech only)

• Print guide (text message only)

• No guide (no navigational guidance)

Experiment (Predictions)

• If guidance is important:– Full, Print, Voice > None

• Given guidance:– If speech medium is most effective

• Full, Voice > Print

– If text medium is most effective• Print > Full, Voice

– If the agent/persona effect is most effective• Full > Voice

– If the agent/persona effect is distracting• Voice > Full

Experiment (Acquisition phase)

• Introduction - Flash intro used to hook the user and acquaint them with the system.

• Lessons – 4 example cases used to teach/test the seven critical issues.

Lessons 1

Lessons 2

Lessons 3

Experiment (Acquisition phase)

• Introduction – Flash intro used to hook the user and acquaint them with the system.

• Lessons – 4 example cases used to teach/test the seven critical issues.

• Search Task – Users answer 4 specific questions, designed to use different modules in the system (specifically the query documents module).

Search IRB Documents

Experiment (Test phase)

• Memory – Free and cued recall tested core ideas from the Introduction and Lesson material. A Cloze procedure was used to test memory for key words.

• Issue Comprehension – As a transfer test from the Lesson module, users read two sample cases and provided respective ratings of problematic issues.

• Perception Ratings – Users provided ratings on the system (e.g. “You learned a lot about human subjects protections.”).

Outline• HURAA

• Experiment

• Results

• Conclusions

Results (Core ideas)

0

0.2

0.4

0.6

0.8

Free recall Cued recall Cloze recall

FullVoicePrintNone

Table 1. Proportion correct

Results (Problematic issues)

0

0.2

0.4

0.6

0.8

Hits False alarms d’ score

FullVoicePrintNone

Table 2. Proportion correct

Results (Search task)

0

0.2

0.4

0.6

0.8

Full Voice Print None

Table 3. Proportion correct

Table 4. Completion Time (min)

20

22

24

26

28

30

Full Voice Print None

Results (Time on task)

0

5

10

15

20

25

30

Introduction Issues Search

FullVoicePrintNone

Table 5. Mean minutes on task

Results (Perception ratings)

0

1

2

3

4

5

6

Amountlearned

Interest Enjoyment Ease oflearning

FullVoicePrintNone

Table 6. Mean ratings (1 to 6, higher is better)

Outline• HURAA

• Experiment

• Results

• Conclusions

Conclusions

• No significant results from any of the dependent measures.

• This null result is incompatible with any of our previously stated predictions.

• A practical implication of the result is that the animated conversational agent did not facilitate learning, usage, or perceptions of the interface.

Conclusions

• Perhaps, due to the well structured nature of the site, a navigational guide was superfluous.

• An agent could possibly provide help in a more complex environment.

• The value of a navigational guide may increase as a function of the complexity, ambiguity, and perplexity of the system.

Conclusions

• Animated conversational agents have proven to be effective when they deliver information and learning material in monologues and tutorial dialogues.

• Perhaps there are special conditions when a navigational guide of some form will be helpful, whether it be print, voice, or a talking head.

• However, these precise conditions have yet to be discovered and precisely specified in the literature.

The Impact of Conversational Navigational Guides on the

Learning, Use, and Perceptions of Users of a Web Site

Art Graesser, Tanner Jackson, Matthew Ventura, James Mueller, Xiangen Hu, and Natalie Person

This research was directly supported by contracts from ONR and IDA, and was partially supported from grants by NSF and the DoD.