Interactive Documentaries: First Usability Studies Marianna Nardon, Fabio Pianesi, Massimo Zancanaro

ITC-irst Panté di Povo, Trento Italy

{nardon,pianesi,zancana}@itc.it

Introduction

Today, research on i-TV focuses primarily on added services, such as interactive advertising [Lekakos et al., 2001] and alternative program suggestion [Ardissono et al., 2001]. Yet new technological improvements on video-on-demand and the success of streaming technology for video delivery on the web will also make feasible, in the near future, video content adaptation.

The present work attempts to set the basis for further studies on interactive video productions, such as interactive documentaries. We propose and test a model in which video and audio material can be organized in such a way that the user can choose his/her path through the documentary.

While there are plenty of usability studies on standard hypermedia, which are mainly text-based, there are no studies on the kind of hypermedia where text plays a minor role with respect to time-based media, such as video and audio (one exception is [Merlino and Maybury, 1999]). In this work we focus on (the issue of) providing the user with the optimal follow-up, at any given moment in an interactive document.

Four different ways of presenting follow-ups have been tested in a controlled experiment with 58 users in a between-subject design. The i-TV scenario was simulated using a RealAudio player embedded in a standard web browser (Netscape Navigator).

The results, although preliminary, show that the best way of organizing information is to provide the user with a clear path through the documentary (and therefore limiting his possibility of freely exploring the content) and allowing follow-ups to be selected even when not contextually appropriate.

A Model for Interactive Documentaries

We used a standard model of hypermedia, through the web, which dictates an organization of the content around the notion of a page. In this model, text plays the main role and follow-ups are suggested by embedding a link to a related page in a portion of the text (called an anchor for the link). In this way, all anchors are accessible as long as the page is accessible. This model can be easily extended to other static media, such as images like in standard HTML, which allows the creation of image maps where portions of a static image can function as anchors.

This model of hypermedia, however, is not adequate for truly dynamic hypermedia; although dynamic content can be played on a web page by embedding a plug in, there is no standard way or reference models for dealing with links or for what to use as anchors.

Historically, another model of hypermedia was proposed: the Amsterdam model [Hardman et al., 1994], based on the notion of objects evolving on a timeline. This model is very similar to those employed today by Flash and Director and it is compatible with the SMIL standard.

We define a model for interactive documentaries as a sequence of scenes where the

content of each scene is in the form of a time-based media (such as video, audio, animation or a combination of those). There are two kinds of scenes: main and additional scenes. Main scenes are collected in a sequence that forms a complete documentary. Each main scene can have one or more additional scenes that provide further details.

Figure 1 illustrates a portion of the graph of interactive documentary scenes we used for the experiment (squares represent main scenes, ovals additional scenes and an arc represents a link between two scenes)

Fig. 1. A portion of the graph of scenes used in the experiment

Follow-up links are shown on the interface as clickable items using the scene title as anchor. In all the interfaces we have designed (see below), we clearly separate the two different kinds of links: (i) from one main scene to another one and (ii) from a main scene to an additional scene (or between additional scenes).

This model has been implemented using RealAudio embedded in a Netscape Browser and it would be possible to implement it using Flash or Director as well. As can be see, our model is in fact a simplified version of the Amsterdam model, yet it can be easily employed for DVD productions and for video-on-demand services.

An interactive documentary with 13 scenes was prepared in the domain of the animal kingdom.

We designed 4 different interfaces, all similar in that the content of each scene is played in the central frame . They differed, on the other hand, in the way follow-ups were presented, and whether the user had to go through the main scenes sequentially or could exploit a strategy of her choice.

Interface 1: Table of Content + contextual follow-ups

In this interface, the table of content (links to other main scenes) is displayed in the left frame, and the anchors correspond to the main scene titles. The user is therefore free to access the main scenes in whichever order he/she chooses.

Follow-up links are displayed in the bottom frame, and only when relevant to the current scene.

Fig. 2. Snapshot of interface 1

Interface 2: Table of Content + follow-ups always-present

As in the previous interface, the table of content is displayed in the left frame as clickable scene titles. All follow-up links are displayed in the bottom frame, but they are always present, regardless of their relevance to the scene currently being played.

Fig. 3. Snapshot of interface 2

Interface 3: Sequential scenes + contextual follow-ups

In this interface, the table of content is not displayed. The scenes can only be played sequentially. In order to retain the possibility of comparisons across conditions, we implemented the sequential mode through explicit “next scene” buttons. As in interface 1, only the relevant follow-up links are displayed in the bottom frame.

Fig. 4. Snapshot of interface 3

Interface 4: Sequential scenes + follow-ups always-present

As in the previous interface, the table of content is not displayed and the scenes can only be played sequentially. As in interface 2, all of the follow-up links are displayed in the bottom frame, regardless of whether they are relevant for the scene currently being played.

Fig. 5. Snapshot of interface 4

The Experiment

The goal of the experiment was to determine which, among the previously described ways of organizing the material, supported a better and more complete

exploration of the hypermedia. To this end, we used two indices: the number of scenes seen1, and the overall time spent in exploring the hypermedia.

Design

The experimental design was a 2x2 between-groups factorial one. Subjects were randomly assigned to the 4 conditions consisting of the four interface settings.

Fifty-eight subjects participated in the experiment. The average age was 26,24 years, from which 41.38% were between the ages of 11 and 18 years, 25.86% between 19 and 29 and 32.76% between 30 and 51.

All participants had enough computer and Internet surfing skills to successfully complete the task, as tested by means of a questionnaire (see below). It was our hypothesis that beyond an established (and very low) threshold of computer literacy task completion would not be significantly affected. The independent variables were (a) the organization of information: table of content (T) vs. sequential presentation (S) and (b) the organization of follow-up links: always present (AP) vs. only relevant links that disappeared after the scene was finished (D). The 2x2 design corresponds to the 4 interfaces introduced above.

The dependent variables were (a) the time spent in navigation, and (b) the number of the links visited.

Procedure

Participants were tested individually. Each session lasted approximately 40 minutes. Each subject was asked to first compile a questionnaire on the Internet and multimedia systems expertise, then to navigate on the assigned interface, and finally to complete the usability questionnaire.

Questionnaire on Internet and multimedia systems expertise

The questionnaire includes 6 items (Likert scale) measuring familiarity with the PC, Internet and multimedia systems, 6 items assessing the amount of time spent (in terms of number of hours) using PC, Internet and multimedia systems, and 6 items addressing personal data.

Usability questionnaire

The usability aspects investigated were the following: “pleasant/not pleasant to use”, “easy/difficult to learn/understand”, “good/bad organization of information”.

1We simply recorded the number of scenes delivered to the user as we did not have the

possibility of checking whether the user actually watched a scene completely or not.

The questionnaire encompassed 40 items based on a Likert scale (agreement/disagreement degree from +2 to –2), 2 yes/no items, and 4 open-answer items that served to collect users’ comments (see Appendix for the complete questionnaire).

In realizing this questionnaire, we took into account the heuristics proposed by Nielsen [Nielsen, 1993] and some of the WAMMI features (Web Usability Questionnaire) [Kirakowsky, 1998]. The questionnaire was evaluated in a pilot study with 22 subjects.

Results

Table 1 reports the average number of the total links navigated in each of the 4 experimental groups, together with their standard deviations. An F-test (F= 5.56, df = 3, P = 0.006) reveals a significant difference between the 4 groups. A post-hoc analysis (Sheffe Test, p = 0.05) shows that this is mostly due to the differences between interface 2 (T+AP) and interface 4 (S+AP).

Mean % Std. Deviation

Interface 1 (T+D) 73.33 18.69

Interface 2 (T+AP) 66.67* 23.80

Interface 3 (S+D) 79.44 17.21

Interface 4 (S+AP) 89.88* 16.40

*Reliable differences to Sheffe Test, p = 0.05

Table 1. Total number of links navigated

Table 2 presents the number of links among main scenes that were navigated, whereas table 3 focuses on follow-up links.

Mean % Std. Deviation Interface 1 (T+D) 93.33 15.14 Interface 2 (T+AP) 82.65* 19.55 Interface 3 (S+D) 100* 0.00 Interface 4 (S+AP) 96.94* 8.27

*Reliable differences to Sheffe Test, p = 0.05

Table 2. Number of links between main scenes navigated

Mean % Std. Deviation Interface 1 (T+D) 45.33 38.15 Interface 2 (T+AP) 40.00* 36.79 Interface 3 (S+D) 50.67 41.31 Interface 4 (S+AP) 80.00* 36.79

*Reliable differences to Sheffe Test, p = 0.05

Table 3. Number of follow-up link navigated

By observing links among main scenes, it becomes clear that Interfaces 1, 3 and 4 do support a good level of exploration and do not differ significantly. It can also be noted that the performance of Interface 3 is remarkable while Interface 2 is significantly weaker (F =7.18, df = 3, P = 0.002). When follow-up links are considered, Interface 4 offers the best support, with all others more or less at the same level, even though the effect is significant only between Interface 2 and Interface 4 (F=4.47, df = 3, P = 0.016). Putting these considerations together, and comparing them with the results at the general level in Table 1, we can conclude that Interface 4 offers the best support for navigation. The worst combination is seen in Interface 2, where the table of content mode for main scenes and the always-present mode for follow-up links are combined. It seems that the presence of the entire set of links throughout the entire interaction hampers navigation, perhaps by disorienting users. However no independent confirmation for this hypothesis, has yet been provided.

With respect to the total time spent on exploring the documentary, no significant differences were found across the four different conditions.

Usability Questionnaire

Most of the items of the usability questionnaire did not show any significant difference between the four experimental conditions. Particularly interesting, nonetheless, was the trend in the item “I felt free to choose the information that seems more interesting to me”. There is a strong tendency towards agreement by users in condition T and a weaker agreement by users in condition S. An Anova test proved a significant difference between the four groups (F=4.61, df = 3, P = 0.006). The Sheffe test showed that T+D and S+D (respectively the strongest agreement and the strongest disagreement) were significantly different (p=0.05).

This result contradicts the quantitative observations on actual users’ behavior. One possible explanation is that the users’ perception of having less freedom in the S condition was due to the presence of the “next scene” button, even if the sequencing was imposed on main scenes only. The T condition, on the other hand, while providing a greater set of choices might have rendered the entire structure of the hypermedia more difficult to understand and consequently creating a greater disorientation: this may explain the lower navigation rate observed empirically.

Conclusions

The present work attempts to set the basis for further studies on interactive video productions, such as interactive documentaries.

We have defined a model of an interactive documentary as a sequence of scenes where the content of each scene is in the form of a time-based media (such as video, audio, animation or a combination of these). There are two kinds of scenes: main and additional scenes. Main scenes are collected in a sequence that forms a complete documentary. Each main scene can have one or more additional scenes that provide additional details.

We think that this model can be exploited in conjunction with the new technological improvements on video-on-demand and the streaming technology for video delivery on the web to implement new forms of adaptation on video content.

The present work introduces a first usability study on interactive documentaries. Four different interfaces have been designed to study the best way to present to the user the links between the scenes. Thus far, the results show that the configuration that leads to the best navigation (in terms of link traversed) is the one that constrained the main scenes in a sequence while allowing an unconstrained possibility of choosing among additional scenes.

This work is at a preliminary level and in particular the model has to be tested with a larger sample of scenes for documentaries.

Acknowledgment

We would like to thanks Elena Not for the help that she gave us in the initial phases of the experiments and for the many invaluable discussions.

This work has been partly supported by the project PEACH.

References

Ardissono, L., Portis, F., Torasso, P., Bellifemine, F., Chiaretto, A., Difino, A.: Architecture of a System for the Generation of Personalized Electronic Program Guides. In proceedings of Workshop on Personalization in Future TV. Sonthofen, (July 13th and 14th, 2001)

Hardman, L., Bultermann, D.C.A., van Rossum, G.: The Amsterdam Hypermedia Model: Adding Time and Context to the Dexter Model. In Communication of the ACM, Vol. 37, n. 2. (February 1999) 50-62

Kirakowsky, J., Claridge, N., Whitehand, R.: Human Centred Measures of Success in Web Site Design. 4th Conference on Human Factors and the Web, Basking Ridge, NJ. AT&T (1998)

Lekakos, G., Papakyriakopoulos, D., Chorianopoulos, K.: An Integrated Approach to Interactive and Personalized TV Advertising System. In proceedings of Workshop on Personalization in Future TV. Sonthofen (July 13th and 14th, 2001)

Merlino, A., Maybury, M.: An Empirical Study of the Optimal Presentation of Multimedia Summaries of Broadcast News. In I. Mani and M. Maybury (eds.) Automated Text Summarization. MIT Press (1999)

Nielsen, J.: Usability Engineering. Academic Press (1993)

Appendix: Usability Questionnaire

With the following questionnaire we intend to evaluate the usability of the system you have just used. We ask you to read the following statements and to express your personal opinion about them (from complete agreement=+2 to complete disagreement=-2).

Thank you for you precious contribution!

1. I found difficult to understand the use of the icons and the buttons present on the screen.

Complete Complete agreement disagreement

+2 +1 0 -1 -2 2. I found familiar the interface with witch I have interacted. 3. The contents presented were well articulated. 4. I performed an action obtaining an effect/behavior by the system completely

unexpected to me. 5. This interface looks like other interfaces I have used in the past. 6. I immediately understood how the system worked. 7. The pen was difficult to use. 8. The artificial voice was annoying. 9. I felt free to choose the information that seemed more interesting to me. 10. I felt obliged to listen to information that was not interesting to me. 11. I felt free to stop and replay the audio files I was listening to trough the headphones. 12. I liked the graphics. 13. It was difficult to understand how the system worked. 14. I felt bored during the interaction. 15. I paid attention to the images’ succession. 16. There was coherence between text and images. 17. The monitor was difficult to use. 18. The system tied down my choices 19. I heard information that I had never heard before. 20. I heard information that I found interesting. 21. I enjoyed using the system. 22. The voice was quite natural. 23. I enjoyed using the pen. 24. The texts were understandable. 25. The length of the texts was appropriate. 26. I enjoyed using the monitor. 27. The system let me make the choices that I found more useful or interesting. 28. The quality of the images was good. 29. The audio-video synchronization was adequate. 30. The proposed follow-ups made me curious. 31. I would have preferred to use the mouse. 32. I was able to access all the available information. 33. The follow-ups were useful. 34. This system was quite annoying. 35. It is difficult to find what I really want in this system. 36. Using this system is a waste of time. 37. This system could use a preliminary explanation. 38. I’m not sure I heard all the available information. 39. This system is too slow. 40. Did you revisited some pages?

yes no 41. Did you listen to the audio files until the end?

yes no

Personal comments

42. Do you think that this system could be useful as: (it is possible to check more than one option)

electronic guide in a natural science museum electronic guide in a natural park didactic tool for a primary school didactic tool for the secondary school didactic tool for a high school didactic tool for the university guide for personal use (for example on a C-D rom) it is not useful at all

43. Which aspect of the system did you like most?