Copyright © 2010 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail permissions@acm.org. ETRA 2010, Austin, TX, March 22 – 24, 2010. © 2010 ACM 978-1-60558-994-7/10/0003 $10.00

*e-mail: y.kammerer@iwm-kmrc.de †e-mail: wolfgang.beinhauer@iao.fraunhofer.de

Gaze-based Web Search: The Impact of Interface Design on Search Result Selection

Yvonne Kammerer* Knowledge Media Research Center,

Tuebingen, Germany

Wolfgang Beinhauer† Fraunhofer Insitute for Industrial Engineering,

Stuttgart, Germany

Abstract

This paper presents a study which examined the selection of Web search results with a gaze-based input device. A standard list interface was compared to a grid and a tabular layout with regard to task performance and subjective ratings. Furthermore, the gaze-based input device was compared to conventional mouse interaction. Test persons had to accomplish a series of search tasks by selecting search results. The study revealed that mouse users accomplished more tasks correctly than users of the gaze-based input device. However, no differences were found between input devices regarding the number of search results taken into account to accomplish a task. Regarding task completion time and ease of search result selection only in the list interface gaze-based interaction was inferior to mouse interaction. Moreover, with a gaze-based input device search tasks were accomplished faster in tabular presentation than in a standard list interface, suggesting a tabular interface as best suited for gaze-based interaction.

CR Categories: H.5.2 [Information Interfaces and Presentation]: User Interfaces - Evaluation/methodology; Screen design; Input devices and strategies

Keywords: gaze-based interaction, search result selection, input devices, search results interfaces, Web search

1 Motivation Gaze-based interaction has become a promising means of accessing computers when the user’s hands are occupied or cannot be used for some other reason. Even more, for some people with physical disabilities such as Amyotrophic lateral Sclerosis (ALS), the use of gaze controlled interfaces often is the only possibility to interact with computers and thus to communicate with their environment. A survey conducted among gaze control users with ALS listed internet access, e-mailing and social communication among the most used applications [Donegan et al. 2005]. Enabling these favored tasks of maintaining social interaction and educating oneself via internet is therefore a primary task of applied research in gaze control. Both tasks strongly depend on efficient user interfaces.

From a user’s point of view, internet search is a two-step process, consisting of query formulation and the processing of result sets. While eye-typing is of particular importance for the query composition, efficient information retrieval strongly depends on the presentation of result sets.

In this paper, we concentrate on the task of discovering and accessing information on the Web that usually starts by using a general search engine. The aim of this study is to identify alternative designs for search results interfaces that overcome the problems induced by densely printed result lists, which are most common with popular search engines.

2 Related Work

While eye tracking has been widely used for the examination of Web search patterns, the optimization of search result presentation for gaze controlled applications is rather new. Kumar et al. [2007] investigated a combination of eye gaze and keyboard input that was used for navigating through a series of Web pages. The study showed that gaze-based interaction resulted in longer click times and higher click errors than mouse interaction.

Other ways of overcoming the problem of too densely arranged search items are the magnification of parts of the screen, such as a gaze-contingent fish-eye lens [Ashmore et al. 2005], or the approach of graphical, multi-scaled information spaces [Mollenbach et al. 2008]. However, the initial case of easing conventional Web search by means of search engines has not been tackled. The experiment described in the sequel presents a direct approach towards optimizing search engine result pages (SERPs) for use by gaze control.

3 Improving Gaze-based Search Result Selection Due to the immanent inaccuracy of gaze control and its detrimental properties such as the Midas Touch Problem [Jacob 1990], densely arranged result lists - like linear pull-down menus - seem to be poorly suited for gaze-based search result selection. The poor performance of linear pull-down menus [Kammerer et al. 2008] points towards the necessity of a new design approach that places all interactive elements (i.e., the hyperlinks of the search results) sufficiently apart from each other (design guideline 1). Furthermore, as the effect of inaccuracy due to calibration errors tends to increase towards the screen periphery [Beinhauer 2006], interaction elements should be placed more towards the center of the screen (design guideline 2). Finally, in order to avoid Midas Touch, interactive elements should be separated from the actual content (design guideline 3). Based on these considerations, in this study two alternative layouts of search results interfaces were chosen: a grid interface and a tabular interface.

In a grid interface, which has lately been used in some novel search engines, search results are presented in multiple rows and columns. Therefore, in line with guidelines 1 and 2, search results can be placed more in the center of the screen and with larger space in between the search results. In a tabular interface, which was used in a Web search study by Rele and Duchowski [2005], search results are listed from top to bottom while grouping the individual elements (title, summary, URLs) in columns. Additional information comprised in the summary and the URL of a search result is separated from the link (i.e., the title), in compliance with guideline 3.

191

In order to test the suitability of the different search results interfaces for gaze-based search result selection, an experiment was conducted, in which participants successively had to use the three different search results interfaces. Additionally, the gaze-based input device was compared to conventional mouse interaction.

We expected the mouse to be superior to the gaze-based input device for search result selection, resulting in higher task performance and more positive subjective ratings. For instance, gaze-based search result selection should take longer and evoke more accidental selections. For gaze-based interaction, the standard list interface was expected to be the least suitable of the three search results interfaces, as the to-be-selected search results are vertically aligned next to each other in the screen periphery (to the left of the screen). In contrast, we hypothesized that the tabular interface would be most apt for a gaze-based input device: As the summary and URL of a search result can be read without the risk of accidental selections, the Midas Touch Problem should be reduced. Therefore, the tabular interface was expected to lead to higher task performance and more positive subjective ratings than the list interface. The suitability of the grid interface was supposed to be in between the two other search results interfaces because the likelihood of calibration errors should be reduced, but not the Midas Touch Problem.

4 Method 4.1 Experimental Setup Thirty-six able-bodied university students (7 male; mean age: 23.33 years) participated in this experiment. All participants had normal or corrected to normal vision. Participants reported to have intermediate or advanced computer- and Web search skills without any differences between the two experimental conditions (i.e., gaze-based input device vs. mouse). None of the participants had experience with gaze-based computer input.

The eye gaze data was collected with a Tobii 1750 remote eye tracker built into a 17” monitor set to a resolution of 1280 x 1024 pixels. Participants were seated on a height-adjustable seat with backrest. The viewing distance was approx. 65cm, and recorded gaze data was smoothed by a filter algorithm.

4.2 Tasks and Material In order to investigate a rather natural Web search situation, participants were requested to find the answers to specific questions by selecting search results presented by a search engine. Twenty-seven search tasks and associated result lists were created, covering a broad range of topics including sports, movies, travel, news, computers, literature, and automotive. Example tasks included: “When did Apollo 13 take off?” or “Who was the youngest world champion in chess?” Each search task started with a control page containing one of the 27 questions and a brief task description. By pressing the space bar, a Google SERP with pre-defined query terms (e.g., “take off Apollo 13”) and nine search results appeared. The search results were manipulated such that for each task there was exactly one search result, which lead to the correct answer. The eight other results were distracters. Note that the correct search result did not contain the answer, but clearly indicated that the answer could be found on the corresponding Web page. The correct search result was displayed in one of the nine positions, allowing three tasks for each position. The search results were not linked to real Web pages, but to control pages that a) denoted that the correct answer could not be found on this page or b) presented the correct answer. In case of a) participants

could navigate back to the SERP by clicking on a hyperlink “back to the Google page” placed in the center of the screen.

Apart from the experimental manipulation of the search results interfaces (see section 4.3) the SERPs were displayed in Google style because of people’s familiarity with this search engine. However, ads and the hyperlinks “in cache” and “similar pages” were not included on the SERPs. In the experiment the SERPs were presented in full screen mode such that there was no browser task bar displayed. In each of the interfaces the nine search results fit on the screen, thus obviating the need for scrolling.

4.3 Experimental Design The experiment was a 3 (within-subjects) x 2 (between-subjects) mixed-model factorial design.

As a first factor, the search results interface was varied within subjects by presenting search results in a list interface, a grid interface, or a tabular interface (see Figure 1). In the list interface the nine search results were listed from top to bottom to the left of the screen. In the grid interface, search results were arranged in three rows and three columns, towards the center of the screen. In the tabular interface every search result element was presented in a separate column. The titles (i.e., the hyperlinks) were presented in the left column, the summaries in the middle column, and the URLs in the right column. The nine search results were listed from top to bottom, with the hyperlinks being presented to the left of the screen.

As a second factor the input device was manipulated between subjects, who either used a computer mouse or a gaze-based input device for operating the search results interfaces. In the gaze-based input device, a dwell-time based selection mechanism was used. Because of the complexity of result selection, involving visual scan and decision processes, the dwell time was set to 750 ms. The hyperlinks indicated their interactivity by inverting their color when hovering over them in either interaction technique. Participants were randomly assigned to one of the two conditions, with 19 participants using the gaze-based input device and 17 the mouse.

Figure 1. SERP types, from back to front: List interface, grid interface, tabular interface.

4.4 Procedure

Participants were tested in individual sessions of approximately one hour. Before starting with the experiment participants were asked to provide some demographic and personal data, received

192

some general instructions and were calibrated on the eye tracking system using a nine-point calibration. Subsequently, the first experimental run started with a training task to get acquainted to gaze control and the interaction with the search results interface. Then, participants performed 9 tasks (with the correct search result being located once at each of the nine positions). Participants were asked to accomplish each task as fast and with the least number of clicks as possible. They were informed that for each task only one of the nine search results presented on a SERP lead to the correct answer. A search task was regarded as successfully accomplished if the correct search result was selected within a time limit of 90 seconds. Participants received a feedback on their task accomplishment after each task and were then provided with the next task. After having processed all tasks, a questionnaire addressing participants’ subjective ratings regarding the interface was administered. Afterwards, the eye tracker was recalibrated and the second experimental run started.

All participants performed the same 27 search tasks. The order in which participants used the three interfaces was counterbalanced across participants as well as the order of the search tasks and the position of the correct search results.

4.5 Dependent Measures To test the suitability of the three search results interfaces for accomplishing the fact-finding tasks, we examined participants’ task performance and subjective ratings with either the mouse or the gaze-based input device.

Task performance. Task performance was determined by three dependent measures. First, the number of correctly accomplished tasks (with a minimum of 0 and a maximum of 9 tasks) was counted. Second, task completion time was recorded (in ms) from the start of the task with the pressing of the space bar until the selection of the correct search result. However, the time spent on wrong pages (i.e., the time from the moment of having selected a wrong search result until the return to the SERP) was not included in the time measurement. Only correctly accomplished tasks were included in the calculations. For gaze-based interaction the dwell time (750 ms/click) was included in the analysis of task completion time. Third, the number of search results selected per task was counted both for correctly accomplished tasks and for all tasks (i.e., also including failed tasks). The number of search results selected in correctly accomplished tasks comprised the number of false search results selected plus the selection of the correct search result per task (resulting in a minimum of 1).

Subjective ratings. Subjective ratings included the following measures: First, participants were asked to rate their mental demand during task processing on a scale ranging from 0=very low to 100=very high. Second, participants were presented three statements, which they were asked to rate on a five-point scale (5=highly agree). The statements addressed 1) how much they liked the layout of the interface, 2) how easy they found the search result selection from an interface, and 3) how satisfied they were with the interface.

5 Results

Tables 1 and 2 show the mean values of the seven dependent measures as a function of the two factors input device and interface. For statistical analyses, first, comparisons between mouse interaction and gaze-based interaction were made. Second, the suitability of the three different interfaces for gaze-based search results selection was analyzed. Because of space limitations, statistical values are only reported for significant results.

Table 1. Means and standard deviations of task performance.

Mouse interaction Gaze-based interaction

List Grid Tab. List Grid Tab.

# correct tasks

8.88(0.33)

8.88(0.33)

8.88 (0.33)

8.05 (1.13)

8.05(1.31)

8.53(0.70)

time (in s)

22.99(6.03)

22.46(5.77)

23.47 (5.72)

29.67 (9.42)

26.03(9.34)

24.26(6.41)

# clicks /corr. task

1.76(0.60)

1.75(0.70)

1.65 (0.52)

1.82 (0.68)

1.64(0.62)

1.47(0.57)

# clicks /all task

1.78(0.60)

1.79(0.72)

1.67 (0.51)

1.84 (0.63)

1.67(0.65)

1.51(0.57)

Table 2. Means and standard deviations of subjective ratings.

Mouse interaction Gaze-based interaction

List Grid Tab. List Grid Tab.

mental demand

43.53(23.8)

51.76(21.0)

46.76 (17.7)

34.74 (18.0)

40.53(22.0)

32.37(19.7)

layout 3.24(0.97)

2.82(0.95)

2.82 (1.43)

3.32 (0.89)

3.53(1.07)

3.58(1.12)

ease of selection

3.88(0.78)

3.29(1.11)

3.59 (1.06)

2.79 (0.92)

3.00(1.20)

3.47(1.17)

satisfac- tion

3.47(0.87)

3.06(0.90)

3.41 (1.28)

3.42 (0.90)

3.37(1.12)

3.63(1.27)

5.1 Comparisons between Mouse- and Gaze- based Interaction To compare task performance and subjective ratings between mouse-based and gaze-based interaction we conducted MANOVAs with input device as between-subjects factor.

Task performance. The MANOVA showed a significant effect of the input device on the number of correctly accomplished tasks (F(3, 32)=4.71, p=.01). Univariate analyses revealed that in all three result interfaces mouse users accomplished more tasks correctly than users using the gaze-based input (list interface: F(1, 34)=8.50, p=.01; grid interface: F(1, 34)=6.42, p=.02; tabular interface: F(1, 34)=3.68, p=.06). The greatest differences between the two input devices appeared in the list interface and the least in the tabular interface. With regard to the task completion time, the MANOVA showed a marginally significant effect of the input device (F(3, 32)=2.50, p=.08). Univariate analyses revealed that this effect could be traced back to the list interface (F(1, 34)=6.25, p=.02). In the list interface tasks were accomplished significantly faster with the mouse than with the gaze-based device. Though, for the grid interface and the tabular interface there were no differences between input devices. Contrary to our expectations, the number of search results selected per task did not differ between input devices, irrespective of whether only correctly accomplished tasks were included in the analyses or all tasks.

Subjective ratings. The MANOVA showed no significant differences between input devices on participants’ perceived mental demand. Although not reaching statistical significance, the MANOVA showed a statistical trend of input device on participants’ ratings regarding the layout of the interfaces (F(3, 32)=2.30, p=.10). Univariate analyses revealed that this effect

193

could be traced back to the grid interface (F(1, 34)=4.28, p=.05) and marginally to the tabular interface (F(1, 34) =3.16, p=.08). Users of the gaze-based input device tended to like the layout of these alternative interfaces more than mouse users. With regard to participants’ ratings about the ease of selection of search results from a SERP, the MANOVA showed a significant effect of input device F(3, 32)=4.90, p=.01). Again, univariate analyses revealed that this effect could be traced back to the list interface (F(1, 34)=14.62, p=.001). Users of the gaze-based device rated search result selection in the list interface less easy (i.e., more strenuous) than mouse users, whereas for the grid and the tabular interface ratings did not differ between input devices. Finally, the MANOVA showed no differences between input devices with regard to users’ overall satisfaction.

5.2 Suitability of Search Results Interfaces for Gaze-based Interaction To compare task performance and subjective ratings between the three search results interfaces, repeated-measures ANOVAs with interface as within-subjects factor were conducted.

Task performance. The repeated-measures ANOVA showed no significant differences between the three interfaces on the number of correctly accomplished tasks. However, with regard to task completion time, the ANOVA showed a marginally significant effect of interface (F(2, 36)=2.55, p=.09). Bonferroni-adjusted post hoc tests showed that in the tabular interface tasks were accomplished faster than in the list interface (p=.08). With regard to the number of search results selected per task, the ANOVA also showed a significant effect of interface (F(2, 36)=3.30, p=.05). Again, post hoc tests revealed that in the tabular interface participants selected less search results to accomplish the task than in the list interface (p=.03). When analyzing the number of clicks for all tasks, this effect becomes even stronger (p=.01). Furthermore, for both variables (time and clicks), values for the grid interface were in between, neither differing from the list interface nor from the tabular interface.

Subjective ratings. Although not reaching statistical significance, the ANOVA showed a statistical trend of interface on participants’ perceived mental demand (F(2, 36)=2.51, p=.10). Post hoc tests revealed that in the grid interface participants tended to perceive a higher mental demand than in the tabular interface (p=.09). Besides this, no differences were found between interfaces on users’ ratings regarding the layout of the interfaces, the ease of search result selection, and their overall satisfaction with the interfaces. In case of mouse operation, no significant differences were registered between the interfaces with regard to task performance and subjective ratings.

6 Conclusions As expected, the study showed that mouse users accomplished more tasks correctly than users of the gaze-based input device. Of note, however, irrespective of the input device almost all tasks were accomplished correctly. No differences were found between input devices regarding the number of search results selected to accomplish a task, with very few wrong search results being selected. Thus, contrary to our expectations, gaze-based search result selection in general did not evoke more accidental selections. Furthermore, with regard to task completion time and ease of search result selection only in the list interface gaze-based interaction was inferior to mouse interaction, but not in the two alternative interfaces. The layouts of the two alternative interfaces were also liked better when operated with the gaze-based input device than when operated by mouse. Moreover, when search results were presented in a tabular interface with a gaze-based

input device, in line with our expectations, search tasks were accomplished faster and with fewer search results selected than in a standard list interface. One drawback of the grid interface might be that it is perceived more mentally demanding than the tabular interface, which might be due to its unclear arrangement of the search results.

To conclude, even though not all of the experimental results reached statistical significance, the study quite clearly speaks against using conventional list interfaces for gaze-based search result selection. Rather, this study provides first indications that the tabular interface is best suited for gaze-based interaction among the given alternatives. Its suitability for a series of consecutive searches in case that the desired information could not be found among the presented results or for browsing or navigational tasks has yet to be shown. Furthermore, one can assume that by including a separate activation link instead of using the title as link the advantage of a tabular layout might be further increased. Nonetheless, without further manipulation of search engines, the tabular interface as it was used in the current study presents a first step towards more efficient Web searching for situations, when the user’s hands cannot be used, for instance, due to motor impairment.

References ASHMORE, M., DUCHOWSKI, A.T., and SHOEMAKER, G. 2005.

Efficient eye pointing with a fisheye lens. In Proceedings of Graphics interface. GI ‘05. ACM International Conference Proceeding Series, vol. 112. Canadian Human-Computer Communications Society, 203-210.

BEINHAUER, W. 2006. A widget library for gaze-based interaction elements. In Proceedings of the 2006 Symposium on Eye Tracking Research & Applications. ETRA ‘06. ACM, New York, NY, 53-53.

DONEGAN, M. et al. 2005. User requirements report with observations of difficulties users are experiencing. COGAIN, IST-2003-511598: Deliverable 3.1.

JACOB, R. J. K. 1990. What you look at is what you get: eye movement-based interaction techniques. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering. J.C. Chew and J. Whiteside, Eds. CHI ‘90. ACM, New York, NY, 11-18.

KAMMERER, Y., BEINHAUER, W., and SCHEITER, K. 2008. Looking my way through the menu: The impact of menu design and multimodal input on gaze-based menu selection. In Proceedings of the 2008 Symposium on Eye Tracking Research & Applications. ETRA ‘08. ACM, New York, NY, 213-220.

KUMAR, M., PAEPCKE, A., and WINOGRAD, T. 2007. EyePoint: Practical pointing and selection using gaze and keyboard. In Proceedings of ACM Conference on Human Factors in Computing Systems. CHI ‘07. ACM, New York, NY, 421-430.

MOLLENBACH, E., STEFANSSON, T., and HANSEN, J.P. 2008. All eyes on the monitor: gaze based interaction in zoomable, multi-scaled information-spaces. In Proceedings of the 13th international Conference on Intelligent User Interfaces. IUI ‘08. ACM, New York, NY, 373-376.

RELE, R S., and DUCHOWSKI, A.T., 2005. Using Eye Tracking to Evaluate Alternative Search Results Interfaces. In Proceedings of the Human Factors and Ergonomics Society, Sep. 26-30, 2005, Orlando, FL.

194