3 2007 · based email application (figure 1.1a). its main page has roughly a hundred clickable...
TRANSCRIPT
Improving Target Acquisition in Web
Applications with Link Prediction
by
Roger B. Hanna
S.B., Massachusetts Institute of Technology (2005)
Submitted to the Department of Electric Engineering and Computer Science
in Partial Fulfillment of the Requirements for the Degree of
Master of Engineering in Electric Engineering and Computer Science
at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY
March 2007
@2007 Roger B. Hanna. MMVII. All rights reserved.
The author hereby grants to MIT permission to reproduce and
distribute publicly paper and electronic copies of this thesis document
in whole and in
A uthor ..............................................
Department of ElectricalkEngineering and Computer bcience
March 12, 2007
Certified by ................................
Robert C. Miller
- "£ssor
,isor
Accepted by .......................
Arthur C. SmithMASSACHUSETTS INSTITUTE
OF TECHNOLOGY Chairman, Department Committee on Graduate Theses
OCT 0 3 2007BARKER
LIBRARIES
2
Improving Target Acquisition in Web
Applications with Link Prediction
by
Roger B. Hanna
Submitted to the Department of Electrical Engineering and Computer Science
on March 12, 2007, in partial fulfillment of the
requirements for the degree of
Master of Engineering in Computer Science and Engineering
Abstract
Web application users spend considerable time clicking on hyperlinks and buttons to complete
frequent tasks. Individual application developers can optimize their interfaces to improve typical
usage; however, no single task model will accurately reflect the needs of a wide audience of users.
This thesis describes EasyLink, an automated optimization to the view of Web applications.
EasyLink facilitates the common activities of individual users without explicit customization by
each user. Using a record of the user's actions, EasyLink adapts the view of the page on later
visits. The new view reduces unused elements by decreasing their contrast and emphasizes the
most used elements by enlarging their size and ease of pointing. An evaluation of EasyLink on
Gmail shows that it accurately models 64% of user behaviour, significantly reduces the time to
complete simple tasks, and is preferred by users over the default view of Gmail.
Thesis Supervisor: Robert C. Miller
Title: Associate Professor
3
4
Acknowledgements
I'd like to thank all the people who have given me their support and patience throughout my
thesis: Professor Miller, Val, Kurt, Al, mom, and dad. I don't think I could have possibly divided
my time further. I'd also like to thank the other students in the User Interface Group especially
Greg, Kevin, Darris, Vikki, everyone who participated in my user studies, and everyone else who
gave me feedback about my project. You rock.
5
6
Contents
1 Introduction 15
2 Related Work 21
2.1 Target Acquisition Index of Difficulty . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.1.1 Area Cursor .................................... 22
2.1.2 Expanding Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.3 Jumping to Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.4 Dynamic Control Display Gain . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2 Modifying Web Page Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.1 Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.2 Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3 Link Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.1 Assisted Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.2 Reducing Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.3 Design Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3 Modeling User Tasks 29
3.1 Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Element Pruning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7
3.3 Feature Selection . . . . .
3.4 Recording . . . . . . . . .
3.4.1 Listeners . . . . .
3.4.2 Content Selection .
3.4.3 Scalability . . . . .
3.5 Distribution Space . . . .
4 Gmail Task Study
4.1 Method . . . . . . .
4.2 Results . . . . . . . .
4.3 Discussion . . . . . .
5 Browser User Interface
5.1 Visual Changes . . .
5.1.1 Font Size . .
5.1.2 Contrast . . .
5.2 Mouse Controller
6 Look and Feel Study
6.1 Method . . . . . . .
6.2 Results. . . . . . . .
6.3 Discussion . . . . . .
7 Conclusion
7.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2 Further Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.1 Full Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
32
37
37
37
38
39
41
42
43
43
47
48
48
49
51
53
53
56
59
63
63
64
64
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.2 Additional Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
7.2.3 Additional Web Application Evaluations . . . . . . . . . . . . . . . . . . . . 66
A Task Model Interface 69
B Enhancement Module Interface 71
C UI Study Pages 73
9
List of Figures
1.1 Simplifying Gmail ..........
1.2 Evolution of Hotmail.com Login.
1.3 Modified Hotmail.com Login
1.4 EasyLink Design Diagram . . . .
2.1 Bubble Cursor . . . . . . . . . .
2.2 Semantic Pointing . . . . . . . .
Visualizing Element Weights . . . .
Weights with Element Pruning .
Weights with Unread Class.....
Weights with Ordering . . . . . . . .
Weights without Text Content . . .
Training Model Evaluation.....
Click Sequence Minimum Threshold
Click Sequence Maximum Threshold
User Differences . . . . . . . . . . . .
. . . . . . . . 30
. . . . . . . . 31
. . . . . . . . 33
. . . . . . . . 34
. . . . . . . . 35
. . . . . . . . . . . . . . . . . . . . . 4 3
. . . . . . . . . . . . . . 4 4
. . . . . . . . . . . . . . . . . . 4 4
. . . . . . . . . . . . . . . . . . . . . 4 5
495.1 Size M odule View . . . . . . . . . . . . . . . . . . . . . . . .
10
16
17
18
20
. . 22
24
3.1
3.2
3.3
3.4
3.5
4.1
4.2
4.3
4.4
5.2 Contrast Module View ..
6.1
6.2
6.3
6.4
6.5
6.6
6.7
Look and Feel Study:
Look and Feel Study:
Look and Feel Study:
Spam Task Results
Reply Task Results
Label Task Results
Qualitative Results
Introduction . . . . .
Entire Login Screen .
Task Exit Questions .
7.1 Google Calendar Data . . . .
C.1
C.2
C.3
UI Study Spam Description
UI Study Reply Description
UI Study Label Description
C.4 Study Spam Page 1 .
C.5 Study Spam Page 2.
C.6 Study Spam Page 3.
C.7 Study Reply Page 1
C.8 Study Reply Page 2
C.9 Study Reply Page 3
C.10 Study Reply Page 4
C.11 Study Reply Page 5
C.12 Study Label Page 1.
C.13 Study Label Page 2 .
C.14 Study Label Page 3.
11
54
56
57
58
58
59
59
67
73
73
74
74
75
75
76
76
77
77
78
78
79
79
.. ..... 50
C.15 Study Label Page 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
12
List of Tables
1.1 Hotmail Complexity Time Line . . . . . . . . . . . . . . . . . .
2.1 Chickenfoot Trigger to Automate Hotmail Login . . . . . . . .
3.1 Recorded XPath . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1 U ser D ata Size . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1 Mouse Controller Pseudocode . . . . . . . . . . . . . . . . . . .
6.1 Task and Variant Data Size . . . . . . . . . . . . . . . . . . . .
6.2 Task and Variant Error Rates . . . . . . . . . . . . . . . . . . .
13
. . 17
. . 26
. . 36
43
52
. . 56
. . 57
14
Chapter 1
Introduction
Web pages are cluttered with content, navigational toolbars, and advertisements. Web browser
improvements have enabled developers to provide Web-based services such as Webmail, eCom-
merce, blogs, and discussion forums. However, not all parts of a Web page benefit individual
users. Excess content hinders users from finding and clicking on frequently used parts of the
page.
This thesis introduces EasyLink, a Firefox extension designed to simplify common tasks in
Web applications. After identifying which elements the user interacts with, EasyLink improves the
usability for frequent tasks by sacrificing usability for infrequent tasks. Consider Gmail, a Web-
based email application (Figure 1.1a). Its main page has roughly a hundred clickable elements,
but a typical user only wants to read new messages. EasyLink takes this knowledge and makes
that specific task easier and faster (Figure 1.1b). Unread messages are visually enlarged and
the mouse pointer is slowed when it is over these messages; other messages are faded into the
background to simplify the overall view of the Web page.
Simplifying a Web page's view is critical as page complexity continues to increase over time.
Using the Wayback Machine [26], consider the progression of Hotmail's login page. What once
15
[email protected] ISetn I 1102I |I gno1
searh t web -= -
02go Gvnai Gal ..d.- Ptop PM a " redsheets Gro " 0 fy y
Compost Ma How tQ, Get Into C041eoe - wWuceadirectoryoGJ
Sentmall
AH MailSos. (24)Trash
contactsQuick Contacts
ISOa add cr my v0 Easyink Project
5' StO s nere 'r
L abEfls
fan,4lYreado-hal (4i
Edit labels
F
F
Sarnwtt R.Bart slat- selt d a practical 7 raseni ea d triaa 10:29 am
Mai Mclain Saying up to 70% on he medicines Is reality now- cmrein t Feb 2
Vera SayZew Call for Daper Felnows-CORRECTED VERSION - Subject Ca P Feb 2
V.,. a-w.a FW- Cal for ECS Ranaarch Assistants at Draoar Laboratort8Feb 2
use tme bprtox or sarc to Ind m90011ee qulody
You e currently a 1 MN (4%) of Your 2813 Ms.Goralvow standard with chat I standardwithout chat tas iHTML Learnmoe
C=207 togoe - Tor a of Us- Privacy - Porasm POW - Goo
(a)
rat9 F Barnett R.Bart slat House It needn't be a practical present like 41 o2amSenLMaI #~_ _Setr Mai setain 6aig pt 76% on ihe Medici nis eSailty noVgf eb 2
All Mail r- Vera saymw Call Ior Draper FefcwsOCRREC TED VERSION -Sub e-t Ca 6?Feb 2All Mail - _ _ - - - - - -
31 Vera SayZ@w FW: Call for EECS Reseach Assistants at Drap Ffab 2
Trah Eugene Donaldson Throug a singe erfaca, users can manwve Wwnd# Jon 31
Edward Fauwkner Re:[as-irrotadons -o Wec Jan . 2 46 M *Jan3l
Mamis GonzissArden. Cab for_ * SecoN Annual 'Anorm Inten ExhibI Jan 30Ch m Eledge To mte tpiMUS mobe 7 - TNit;,etcf ot t t. made mA tauf Jan 29
Pjearch. add. or in, nta- - - li ~ t- e ,d t ~ 5 2carch d n amanda JMend needS hWpno Joke Hey, di Lruaflly forward messago Jan 28
Eal e Ed &Susan Hfm ranna Nr.ahose w" hays sons- we iighed a w cive. tem soJan27
Swnnain Viola The LA Friday N it S#ave m- aso be havkng teir Haloween f Jan 27
caail-rqlated S
"rando-hall (4) Send messages from your oher emai addrestes using your Gmal account. Learimore
reuse (M You are currenly using 104 MB (4%) of your 2813 M.EditLaIgS GmaE vmow standard with chat I standard withou chat bI HTML ean more
207 Goo* - Terms otJse - Pr-vacy Pokcy - PrM PoLes - Goodm
(b)
Figure 1.1: Views of Gmail: (a) default view, (b) simplified view presented by EasyLink, empha-
sizing unread messages
16
easylnkprject@qrnaNcoM I Sel Heo I Sign
M Searc ft Web ej -
GQoRlI G n d hs Dmls mvSrsaht Grom nl syico
GeaSe
Compms Mal NYTTPVE -4 isy t 0.
was a sparse login page is now filled with much more than a form with username and password
(Figure 1.2). As shown in Table 1.1, both the size of the page and the number of clickable
elements have grown over the years, making the most common task of logging into an account
increasingly difficult. An HTML-savvy user may notice this and invest the time to set up an
individualized entrance page with just the necessary fields. However, the individualized approach
requires delving into the page source to extract form data and may fail when Hotmail updates
its login page again.
Hotmair
World'smEE
Figure 1.2: The evolution of Hotmail.com's Login Page [16], [26]
Year Minimum Page Size Clickable Elements
1999 640x480 492002 800x600 632006 1024x768 93
Table 1.1: Hotmail Complexity Time Line
17
EasyLink handles the Hotmail login intelligently, without necessitating expert knowledge or
even user awareness. Using the information that a user only interacts with the login fields,
EasyLink improves usability by enlarging important features and minimizing the clutter (Figure
1.3). The username and password fields have increased font size, while the rest of the page is
muted.
Figure 1.3: Modified Hotmail.com After Personal Thaining.
The benefit produced by EasyLink is not limited to login pages. EasyLink assists with all
common Web application activities by bringing simplicity back to Web pages. EasyLink ad-
justs what users see by emphasizing elements they interact with and reducing advertisements,
extraneous features, or any other elements they do not click on.
The design of EasyLink is based on three principles: Automation, Invisibility, and Agency.
Automation
EasyLink handles customization of Web pages autonomously, giving end users the strength
of the system without the cost of understanding it. Thus it can micromanage Web applica-
tions to optimize many more targets than a user would care to customize by hand.
18
Invisibility
Extensive visible changes to a Web page are distracting. If EasyLink creates too much
distraction, it will discourage people from using it. Ideally all EasyLink's changes would be
subtle. Even if they are noticed, changes should not irritate the user.
Agency
The most important aspect of EasyLink is that all options remain available to a user.
Completely removing an element from its Web page or jumping the mouse pointer over any
target is not acceptable. Because users' tasks are likely to change, removing any option
should be avoided at all cost.
EasyLink's design can be divided into two main parts: (1) the user task model which records
input events and suggests a likelihood of interaction for an element; (2) an automated look and
feel extension which uses model predictions to reduce the contrast of unlikely elements or both
increase font size of likely elements and slow the mouse pointer over these elements. An overview
of EasyLink is diagrammed in Figure 1.4. When a user visits a Web page, the user task model
loads prior knowledge about the use of that page and passes the probability distribution for the
elements to the look and feel modules that alter the user's view.
EasyLink was developed in Firefox because of the strong division provided between the internal
representation and the rendered view of a Web page. Gmail was the primary Web application
considered because the complexity of its design exceeds most other Web applications. These
techniques could be applied to any application or user interface toolkit.
The rest of this thesis is divided into six chapters. Chapter 2 discusses prior work on mouse
usability, Web accessibility and customization, and machine learning in Web browsing. Chapter 3
discusses the design and implementation of the user task model. Chapter 4 evaluates the user
task model. Chapter 5 discusses the design and implementation of the look and feel modules.
19
document change events
Fircfox input events
input training
scale elementsmode loadng D ocurnent Inter actor
U-ser T ask Modcl elmn wihs L o ok and Feel
Mouse Controller
Font Size
Contrast
Figure 1.4: EasyLink Design Diagram
Chapter 6 evaluates the look and feel modules. Chapter 7 discusses the contributions of this
thesis and the implications for future research and development.
20
Chapter 2
Related Work
Prior related work is focused on three different areas. They are: (1) improvements to time spent
moving to clickable targets; (2) browser customizations; or (3) web usage mining to assist with
searching, to reduce browsing latency, or to improve existing designs.
2.1 Target Acquisition Index of Difficulty
Target acquisition time has been modeled by Fitts' law (Equation 2.1) since the seminal studies
by Fitts [13] and MacKenzie [20]. Fitts' law states that movement time (MT) is constrained by
the amplitude of the movement (A) and the target's width (W). Other environmental conditions
affect the empirically determined constants, a and b. The logarithmic term of amplitude over
width is referred to as a target's index of difficulty (ID).
MT = a+ blog ( +1 (2.1)(W
Researchers have investigated several different ways to improve the index of difficulty for
pointing tasks. Proposed solutions are to increase the target width with an area cursor or ex-
panding targets, to decrease the amplitude by moving the pointer and object closer together, to
21
improve both amplitude and width by dynamically adjusting the control display gain.
2.1.1 Area Cursor
An area cursor increases the effective width of a target by using a region of activation rather
than a point of activation as in traditional pointing. Kabbash and Buxton [17] proposed the
original system and showed this system could be modeled by Fitts' law. However, increasing
the activation area of the cursor posed a problem: how should the area cursor resolve activation
conflicts when two or more targets are within the region? Worden et al [27] resolved the problem
by including a single point of activation when more than one target is in the area of activation.
This enhanced area cursor gave the benefits of the original system with the authoritativeness of
traditional pointing.
Grossman and Balakrishnan [14] proposed another improvement with their bubble cursor. Like
the previous area cursors, a bubble cursor activates a region, but the region dynamically resizes so
that exactly one target is within the activation region. Figure 2.1 divides the resulting clickable
regions with dashed grey lines for each circular target. Elements have an enlarged clickable area,
with no impact on the application's layout nor the cursor's movement. As a result, the bubble
cursor makes efficient use of the provided space with no effort by the application's developer.
0>
Figure 2.1: Bubble Cursor Activation Regions [14]
In a complex application that maximizes its clickable area (such as Gmail), the extra white
22
space that is required by the bubble cursor is not available. Thus the bubble cursor degenerates
to traditional pointing. For example, suppose the user wants to select a character within a
paragraph. Since every character of text is a potential target, the bubble cursor cannot improve
on the original index of difficulty.
2.1.2 Expanding Targets
Another means to increase the width of a target is to expand the target's actual size as the
mouse pointer approaches the target. McGuffin and Balakrishnan [21] showed that users benefit
from expanded targets after 90% of the distance had been traveled. Zhai et al [29] determined
that expanding targets had the expanded target's index of difficulty. While expanding targets
improve a single target's index of difficulty, it has not been shown to be effective for multiple
nearby targets, which includes the most well-known implementation, the Mac OS Dock.
2.1.3 Jumping to Targets
Many displays have sparse sets of targets. Rather than adjusting the width of the targets,
Baudish et al [5] bring the target to the cursor with the drag-and-pop technique. By examining
the direction of movement, drag-and-pop can identify hard to reach targets and provide alternate,
closer aliases. This system is designed for large displays where it would be too cumbersome to
move the entire distance.
Most computer systems don't have the physical distance problem drag-and-pop resolves. An-
other alternative is to bring the cursor to the target. Guiard et al [15] designed a system to skip
over void space so that the cursor is always pointing at an object. Both of these techniques are
a dramatic change from traditional pointing on a displayed screen image. Drag-and-pop requires
users to hit a previously non-existent target and object pointing has no cursor at all. These kinds
of changes are not feasible as incremental improvements to existing computer systems.
23
2.1.4 Dynamic Control Display Gain
The final way to improve movement time with respect to width and amplitude is to intelligently
change both values. By adjusting the control display (CD) gain (the ratio of control device
movement to displayed position movement), both Keyson [18] and Worden et al [27] improved
movement time. If the cursor is outside the expected target, an increase in CD gain reduces the
movement's amplitude. If the cursor is inside the expected target, a decrease in CD gain increases
the target's width. By combining these two ideas, targets intended to be frequently used can have
reduced movement times without requiring a change to the appearance of the page or a change
to the appearance of the cursor.
Blanch et al [7] proposed an extension to user interface toolkits to specify a widget's CD gain.
In this system, called Semantic Pointing, the mouse's speed changes relative to the element which
it is over. Semantic pointing allows specific targets to have a smaller movement time, relative
to other targets. In effect, a user interface designer controls the visual space independent of the
motor space. For example, while a user may see a consistently sized view of a menu (Figure 2.2a),
the mouse cursor will pass over unusable menu items more quickly (Figure 2.2b). Applied within
a browser, this system of weighting elements' motor space emphasizes targets that are expected
to be used frequently by making their effective width larger. Additionally, dynamically scaling
mouse speed is not obvious to the user, even though users benefit from it.
Undo ^Redo Cut ^X
Copy ^ACCut ^X Paste ^
Copy ^CPaste ^V
Figure 2.2: Semantic Pointing: (a) Visual Space, (b) Motor Space [7]
24
2.2 Modifying Web Page Content
Different users interact differently with Web pages; it is not advantageous to have all users view
the same page. Fortunately, Web pages only suggest what a user should see, and the user's
browsing environment determines the rendered view. Early work such as Barrett et al's WBI [4]
demonstrated that client-side modification of the view was an effective tool to personalize Web
pages. There are three components to Web page personalization: automating the Web browser,
improving accessibility, and simplifying the displayed content.
2.2.1 Automation
Initial work to automate web browsers, such as WBI [4] and Greasemonkey, required intimate
knowledge of a web page's source in order to customize the Web page. Since most users don't have
this knowledge, simpler interfaces were necessary. Anupam et al [1] enabled users to automate
tasks with their WebVCR. Users could record sequences of clicks and replay these macros at a
later time. Because users had to demonstrate each situation explicitly, WebVCR had limited its
scope to automating simple tasks.
To handle a broader set of tasks, parameters become a necessary tool. Bolin et al [8] provided
a simple interface to arbitrarily automate a Web page for end users. Their system, Chickenfoot,
provided its own set of understandable functions to operate on a Web page independently of
the underlying representation. For the Hotmail login page from Chapter 1, Chickenfoot can
automate the process of logging into Hotmail without requiring knowledge of HTML (Table 2.1).
End users can gain some of the performance benefits of EasyLink with Chickenfoot, but they
have to recognize the problem and spend the time to resolve it.
25
enter("e-mail", "[email protected]")enter( "password", "letmein")click("sign in")
Table 2.1: Chickenfoot Trigger to Automate Hotmail Login
2.2.2 Accessibility
Not all users can effectively use the typical view of the Web. Older adults have degraded vision
and movement precision. Disabilities also may hinder Web page usability. Focused work had been
done targeting individual handicaps, such as text-to-speech for blindness [3], [28] or increased vis-
ibility for low vision [9]. Most designs were limited to a group of users and to a subset of Web
pages. Richards and Hanson [23] approached Web browser enhancement as a means to improve
the overall accessibility of a Web page for all users. They provided a broad set of enhancements so
individual users could pick and choose which ones they preferred. These enhancements included
audibility improvements (text-to-speech for both text and image ALT tags), readability improve-
ments (enlarged font size, spacing, and contrast), and complexity reduction (hiding animations,
backgrounds and images). With this approach to the problem, Richards and Hanson were able
to improve Web page usability for a broad audience of users.
Approaching accessibility as a generalizable, robust solution is an important concept that
carries over to EasyLink. Since many Web applications have good color schemes, we don't want
to discard this information. To best preserve both Web application style and Web accessibility,
EasyLink only blends existing colors and increases the size of text.
2.3 Link Prediction
Link prediction and data usage mining are machine learning techniques used to identify what
a user interacts with in a web page. Use of these techniques enables developers to improve a
26
user's web browsing experience based on prior knowledge of the user. This can be done on the
user's computer (client-side) to reduce latency, on the server to improve caching, and off-line by
developers to improve the page design. Initial work in link prediction came out of the need to
facilitate searching on the Web.
2.3.1 Assisted Searching
Armstrong et al [2] designed an extension, WebWatcher, to help users locate information. It
asynchronously fetches pages with content similar to what the user requested, and alters its
recommendations based on explicit user feedback. WebWatcher helped inexperienced users find
content on their subject matter and it helped advanced users do so more quickly. However,
WebWatcher fails when people are unaware of precisely what they want.
Lieberman [19] addressed this problem with Letizia by looking for similarities in the pages
the user browsed. By suggest alternate pages with similar content, Letizia trained its user model
implicitly on whether users continued with the recommended pages or went back to a prior Web
page. Both WebWatcher and Letizia were interactive extensions for Mosaic with front-end changes
to the browser, but their primary purpose was to actively assist in user searches.
2.3.2 Reducing Latency
As well as providing suggestions to the user, link prediction can be used to improve Web browsing
performance. Bestavros [6] used trends in server HTTP requests to predictively cache and send
additional pages with the requested page. Padmanbhan and Mogul [22] applied similar techniques
on the client-side. Using knowledge of prior Web browsing sessions, they requested additional
Web pages the user was likely to click on. Since users had to wait for Web page downloads less
often, users browsed the Web faster.
This idea of invisibly facilitating expected links plays an important role in EasyLink, but
27
predictively caching additional web pages has limitations. It can not capture all generated content
and it increases the total bandwidth every time it incorrectly predicts a link. Instead, EasyLink
uses link prediction to improve a Web page's accessibility.
Another key difference between these systems and EasyLink's user task model is the type
of data being learned. Existing link prediction systems operate on visited URLs to predict
subsequent URLs the user will visit. Since Web applications don't necessitate URLs or even
server requests, EasyLink cannot assume URLs are the only relevant information on a Web page.
EasyLink operates on elements within the Document Object Model (DOM) to predict subsequent
elements the user will click on.
User task models can be developed independent of their application. Other researchers pro-
posed many subsequent improvements to the task model by using Markov Models [10], [12], [24],
or page content [11].
2.3.3 Design Evaluation
Link prediction is not limited to on-line performance. Spiliopoulou [25] analyzed server HTTP re-
quests off-line to identify common paths a user takes. This system enables developers to critically
evaluate existing Web site designs and justify changes to the user interface.
28
Chapter 3
Modeling User Tasks
Measuring the way a user interacts with a Web page is a key feature of EasyLink. A probability
distribution of the user's interaction with elements within a Web page is necessary to make
intelligent alterations. This chapter addresses the design of a model of the user's interactions
and noteworthy issues I encountered implementing it. The user task model is broken down into
four parts: (1) selecting which elements to operate on; (2) representing the set of elements; (3)
choosing when to train an element; and (4) providing a metric of user interaction.
All task model details are demonstrated as in Figure 3.1, which displays the frequency an
element is clicked on as shades of grey. These frequencies, or element weights, are relative to
others, so their average is 50% brightness. Darker values indicate elements that are used more
frequently than others.
3.1 Design Goals
Before describing the details of the task model, the overall design goals are presented.
Automation
The task model needs to automatically prepare itself in advance so that it can to respond
29
Figure 3.1: Visualizing Element Weights with Background Color. Darker elements are used more
frequently.
in real time to whatever queries the enhancement modules make. EasyLink provides the
task model a programming interface to enable this functionality (Appendix A).
Invisible
EasyLink should have invisible long term data storage. It should use a negligible amount of
disk space and should minimize the amount of stored confidential or personal information.
Agency
All of EasyLink's computation needs to be done in real time. Spending 200 milliseconds
may be an acceptable cost when a document is first loaded, but no noticeable delay is
acceptable in response to a mouse or keyboard event.
3.2 Element Pruning
EasyLink operates on a subset of elements in the DOM. Pruning the element set reduces model
complexity and provides a more consistent view to the user. For example, the simplest Gmail
30
inbox has roughly 800 HTML elements, creating 120 uniquely clickable objects. The other 680
elements provide stylistic layout to the page by emphasizing parts of the information such as
unread subjects, and deemphasizing less important parts like the message body. The difference
in user interaction models resulting from element pruning is demonstrated in Figure 3.2. Without
element pruning, the message label, subject, and body each have different weights. With element
pruning, they have a uniform weight because they are operated on as a group.
(a)
(b)
Figure 3.2: Weighted Elements (a) with Pruning, (b) without Pruning
EasyLink operates on one clickable element and its non-clickable descendants as a group.
Unfortunately, current Web browsers do not provide the necessary mechanism for EasyLink to
determine what elements actually have click event handlers attached to them. EasyLink solves
this problem by heuristics that derive semantic meaning from syntax. Syntactically clickable
elements have one of these properties making them relevant to the task model: (1) they are a
block element (an element that is rendered as a rectangle) which developers are likely to use
for grouping purposes, or (2) they are an inline element (an element that does not form a new
block) with a known native click event handler (such as <A> or <INPUT>). The block elements
are significant because they naturally group descendents together in the rendered view, making
them appear as a single object to the user. The inline elements are significant because they
always capture click events. The complete list of syntactically clickable elements is: A, APPLET,
BUTTON, DIV, INPUT, LABEL, OPTION, SELECT, TD, TEXTAREA, TH.
31
M
3.3 Feature Selection
EasyLink needs a means of identifying the same element across browsing sessions. Even with
the reduced set of clickable elements, a plethora of features remain to identify elements. I have
considered several feasible features to observe: tag name (the type of the element), class name (the
style of the element), previous target (the previously clicked element), child index (the element's
position in the DOM), node control name (the label assigned by developers), and the element's
text content. There may be other features influencing a user's decision, but there is a tradeoff
between information gain and training rate loss as more features are observed. Unless the existing
feature set is insufficient or less concise than other features, it is not necessary to consider other
features.
Tag Names indicate which type of element is involved. Different types of elements (such as
buttons, text fields, hyperlinks, and tables) look and behave differently, so a user would not
consider them to be the same.
Class Names indicate the decoration of a element. Cascading style sheets (CSS) describe the
style associated with each class name. Developers use CSS and class names to produce
a consistent view across multiple documents and elements without transferring excessive,
redundant information from the server to the client. Distinct class names suggest different
element states that may encourage or discourage a user from clicking on that element.
For example, Gmail changes the class name of table rows to indicate whether a message
is unread or not. The difference in what the user clicks on resulting from different class
names is demonstrated in Figure 3.3. Unread messages and read messages have different
class names. Since a user is more likely to click on an unread message than a read message,
unread messages are weighted higher.
32
M Uy skR Jan 10
Figure 3.3: Weighted Elements (a) with Unread Mail Class, (b) with Read Mail Class
Previous Target is the element the user previously clicked on. At times, users must perform a
sequence of clicks to accomplish a single task. For example, in Gmail you can tell when you
have unread messages in a folder. These messages can only be read after clicking on that
folder, followed by a click on the message. To distinguish between independent tasks, only
consecutive clicks within five seconds are considered. Five seconds should be enough time
to perform any planned movement. This cut off makes it possible to discard information
when the user pauses to read or think.
While the previous target is an important feature, a non-distracting static view of the page
is preferred for its invisibility than the potential reduction of the task's index of difficulty
resulting from dynamically rescaling elements. Therefore, the previous target is only used
for invisible changes to the Web page's look and feel: adjusting the mouse speed.
Examples of Gmail tasks with sequential clicks include:
* Filing multiple messages
One of Gmail's main filing operations is to first select a set of messages by clicking
on their checkboxes, and then to click on the appropriate filing button, be it archive,
report spam, delete, or one of the many other options in the drop-down menu. When
selecting several messages, people tend to select them in order, either from the top to
bottom or bottom to top. This creates a sequence of events for adjacent checkboxes
33
ending with a click on the filing buttons.
* Drop-down Menu
A drop-down menu is the best example of a required sequence of clicks, since you
always have to click on a menu before you click on a menu item. In a Web page,
the <SELECT> element requires a click before it is possible to pick an <OPTION>
element.
Child Indices indicate the structural position of the element in the Web page. When the user
is provided an ordered list of objects, the frequency of use of elements will often vary. In
Gmail, lower index messages are more frequently read since they arrive chronologically,
with the newest messages at the top. The difference in what the user clicks on resulting
from child indices is demonstrated in Figure 3.4. With child index information, messages
at the top of the inbox are more likely to be clicked on. Without child index information,
all messages have a uniform weighting.
(a)
(b)
Figure 3.4: Weighted Elements (a) with Child Index Ordering, (b) without Child Index Ordering
Node Control Names indicate an explicit label placed on the element by the developer to
uniquely identify the element. Anchor and frame elements have names to direct page loads
to them. Form elements and their components (e.g. buttons and text fields) have names
34
to identify their key-value pairs used for posting data. For example, a login form typically
has two fields named "username" and "password" so the login button can send the data
"username=easylink.project, password=letmein" to login to Gmail. Radio and checkbox
elements share these names, so their value is included to identify them. Named elements
are better identified by their node control names than by their structural position, since the
names are coupled with the underlying tasks.
Text Content was considered because it can identify some equivalences better than structural
data. Specifically, the text label of a link may more accurately describe it than its child
index.
For example, Gmail chat reorders its Quick Contacts based on availability, although a
contact's name is often more relevant. The erroneous difference in user interaction resulting
from repositioned content is demonstrated in Figure 3.5. This figure shows that the second
index contact is regularly used. Most of the time Valerie is the second contact, except
occasionally when the first contact is not logged in. Not using text content as a feature
results in slower training of the quick contacts, but sufficient data will enlarge both the first
and second index contacts as they are both clicked on. Child indices is able to model this
task, just not as well.
uik Contacts V, QIc Contacts
* Roger Hanna RogerHannaSet status here w Set status here y
* Rob Stott 4 Valerie
4 Holden Caulfleld JoeySJoey Rob Stott. Teddy Andrews e Teddy Andrews
Add contact I Add contact
(a) (b)
Figure 3.5: Weighted Elements (a) with correct weighting, (b) with erroneous text content
35
Another possibility is that the user is more interested in a substring of the text content than
the entire string. Links often contain more than one word, and the user may be looking for
specific information. For example, consider a user reading a news feed. A user interested
in Bosnia will open more articles containing the word Bosnia than a user not interested in
Bosnia. Without exhaustive training techniques such as a bag-of-words, which measures
the frequency of each individual word, the use of substrings as a feature is not worthwhile.
Finally, saving any text content across sessions introduces problems with privacy. Since
storing any confidential content such as email details is unacceptable, text content was not
used as a feature.
With all of the features reviewed above, each recorded mouse click generates a concise repre-
sentation of the event. With this representation of events, it is possible to identify equivalent
elements. Examples of recorded mouse clicks are shown in Table 3.1. Each recorded click is saved
as a unique XPath identifying the features of the element that was clicked.
Login Username:/HTML/BODY/TABLE[3]/TBODY/TR/TD[2]/TABLE[1] [©class= "form-noindent"]/TBODY/TR/TD/DIV
/DIV [class= "body"]/FORM/TABLE/TBODY/TR[3]/TD[2]/INPUT[(name="Email"] [@class= "gaia le val"]
Compose Message:/HTML/BODY[@class="sb bn"]/DIV[2]/DIV[1]/DIV[1] [@class="nl"]
Sign Out:/HTML/BODY[©class="sb bn"]/TABLE/TBODY/TR/TD[3]/TABLE/TBODY/TR/TD[@class="trb"]/A[2] [class="sol"]
First Unread Message's Subject and Body:/HTML/BODY[Lclass="sb bn"]/DIV[21/DIV[2]/DIV[2] [©class= "fs"I/DIV/FORM [name= "af"]/TABLE[Oclass= "tlc"]
/TBODY/TR[1][@class="ur"]/TD[5]
Table 3.1: Examples of recorded XPaths from Gmail
36
3.4 Recording
EasyLink works over multiple Firefox sessions by saving to disk when a page is no longer in use
and reloading page data when it is later accessed. Page data is stored on disk as a chronological
log of recorded events (such as the XPaths in Table 3.1). When a page is first accessed, the
user task model expands this chronological log into a complete representation of the user's tasks,
including: a table with a click count of each element, and pairing information for sequential clicks.
Additionally the complete representation contains the last trained event from the current session,
which is used to determine when to record new sequential clicks.
3.4.1 Listeners
All model data comes directly from user input. The mouse down and key down events are recorded
in the Web browser before any Web page scripts are notified, ensuring that the event's target has
not been altered by page content.
For more complex input such as drag-and-drop and drop-down menus, use of mouse up and
key up events are also necessary. If the target of these events is not on the same document, the
event is discarded. This prevents page changes on mouse down (such as Gmail messages) from
training on whichever element is created at the location of the mouse. Some legitimate sequences
may be lost, but recording all sequences of clicks across all Web pages does not scale well. Thus
events like dragging a picture from one Web page to another would be discarded.
3.4.2 Content Selection
Every time a training listener fires, the event may provide unnecessarily redundant or misleading
information that can result in overtraining and mistraining. To minimize this problem, EasyLink
selects which events it will train on.
37
Repetitive tasks may involve clicking on the same element multiple times, but the amount of
information EasyLink gains from subsequent clicks is negligible. While repeatedly clicking, the
user is already pointing at the element and does not need to move to click on it. Only the first in
a set of redundant events is recorded. The last event time used for sequences of clicks is refreshed
on every repeat click.
User error can often lead to mistakenly clicking on an element with no real effect. In order to
best prevent training on these errors, clicks within 500 milliseconds are removed from the model.
Only the last click in a rapid sequence of clicks is recorded.
3.4.3 Scalability
EasyLink needs to perform all of its calculations in a timely manner, regardless of what the user
does in the browser. Therefore, it is worthwhile to evaluate the scalability of its model.
Task models are stored for individual URLs. There is potential for improvement to identify
common structures within multiple documents, e.g. a navigation bar shared across pages on one
domain. But a naive implementation of this would result in under-emphasis of page content,
which is more important than the shared navigation information.
Since Firefox supports many Web pages open at the same time, multiple logs are cached to
prevent context switching. As individual logs grow too large, only the more recently recorded
events are kept and the older events are discarded. This size constraint prevents overtraining the
user task model and maintains the freshness of the model data. If a developer changes their Web
site, a smaller model will adapt more quickly. As data are loaded off the disk they are compared
against the Web page to determine the element weights. (Recall that element weights reflect the
frequency of use). This is the slowest operation in EasyLink. Large Web pages like Gmail take
150 milliseconds to build this data structure.
In order to identify the highest entropy distribution - a uniform distribution across all elements
38
in the page - EasyLink needs at least twice as many recorded click events as the number of clickable
elements. Otherwise, low frequencies will not be observed. To keep data as fresh as possible, twice
the number of clickable elements is a reasonable number of recorded click events.
With respect to the number of clickable elements, persistent data on disk grows linearly and
task model operations grow O(n log n).
3.5 Distribution Space
Making trusted changes to the view of a user's page requires that the user task model have a
well specified, understandable interface for its weights. All weights reported by the task model
are normalized to the range [0,1] such that a the average weight of all elements in the page is
0.5. Elements with weight less than 0.5 are unlikely to be interacted with, and elements with
weight greater than 0.5 are likely to be interacted with. As the variance in element frequencies
increase with log size, the task model grows more confident in its reported values. So a weight of
0.7 has higher confidence in its likelihood of being interacted with than a weight of 0.6. Weights
are generated by Equation 3.1. Element size is included because smaller objects are in more need
of enhancement. Weights are returned in a log scale because slow growth of emphasized elements
is desired to gradually change the user's view of a Web page.
Later sections of this thesis refer to important and unimportant elements. Important elements
have an above average weight (greater than 0.5). Unimportant elements have a below average
weight (less than 0.5).
log clicks(element) +1weight (min(element width, height)(31log = (.))weight 2 log (median clicks/px + 1)(31
39
40
Chapter 4
Gmail Task Study
The purpose of the long term task analysis is to build a corpus of real data with which we can
evaluate user tasks outside of a laboratory setting. Obtaining significant results from a laboratory
environment would not mean the model will accurately predict users completing their own tasks
outside a laboratory. In order to guarantee external validity, the user task model needs to be
able to predict real user activity. Equally importantly, we can use this data to resolve several
questions about the task model:
- Is it able to generate a consistent frequency distribution for a given user?
- How much data is sufficient to model a user's tasks?
- How does the model compare within subjects when the data size is varied?
- How does the model compare between subjects?
- What is the maximum duration of time necessary to capture sequences of clicks?
- What is the maximum duration of time necessary to discard misclicks?
41
4.1 Method
I recruited 6 volunteer subjects who use Gmail daily. Each subject ran a compatible version of
Firefox with an extension installed to record data from input events without user involvement.
The extension intentionally recorded more information than necessary to reduce the chance that
an important feature was missed. Every mouse down, mouse up, and click event on a document
in the mail.google.com domain was recorded. The event name (e.g. mouse up, mouse down),
time stamp, and XPath (such as the XPaths in Table 3.1) were recorded. Also, the first two
characters of text content were recorded to compare labels and child indices without recording
confidential information. When the recorded data grew large or the browser closed, the extension
flushed these data into a log file in the user's home directory.
Each user was recorded for at least one month and the resulting data logs were used to train
individual user models offline. Different size models were compared to find the smallest model
which accurately predicted the user's behaviour. Models of various sizes from 50 to 700 clicks
were trained and tested against the next sequential set of data. Each test element the model
did not correctly predict as either likely or unlikely were considered test errors. The percent test
error was used to evaluate how accurately it predicted those data. The results of these tests are
shown in Figure 4.1.
A second test was run to identify the timing of sequential clicks and misclicks. The user
task model was trained only with sequential clicks with different time thresholds. The minimum
threshold was determined by evaluating the task model's accuracy with different time thresholds.
The maximum threshold was determined by evaluating what time threshold captured the bulk of
sequential clicks.
42
4.2 Results
Users were consistently modelable. On average, based on user data with a training size of 700
clicks, the model had 36% test error in its trained weights. As the training data size increased,
the model grew more accurate.
Subject Log Size Unique Clicks1 18037 79542 9044 35703 11598 49654 8949 38245 7977 28196 15328 4544
Table 4.1: User Data Size
60
75 - .- . - - - - - -- ---
70 - - - - - - -
5 5 .. ... .. .-- --. ...--. .- -. -. .-..-- - - -- - - -
5 0 -.. .. --. .. . -- ..-- .. - ... -- --.. .. .-. .- - - - - - - - - - -
45 -
3 5 -... . .. .--- -- -- ... .. -. . -- -. . ..... . . .. --- -- -- - -.--- -.-- - - -
3010 100 200 300 400 500 600 700
Training Set Size (Number of ClicKs)
Figure 4.1: Training Model Evaluation
4.3 Discussion
From the data, we can conclude that the model was more accurate than random guessing beyond
250 clicks. Achieving beyond 50% accuracy is only possible because of the non-uniform nature
of the user's behaviour. Users may be more or less predictable depending on the variety of tasks
they choose to complete.
As shown in Figure 4.2, the sequences of clicks within 500 milliseconds provided no information
43
70
60
E50-
40-
30 --
20
10
0 200 400 600 600 1000 1200 1400 1600 1600 2000 2200Time Threshold (milliseconds) for Training Click Sequences
Figure 4.2: Click Sequence Minimum Threshold Evaluation
90
6070
60
50
40
30-
20
10
-L 10 15 20 25 30 35 40 45 50Time Threshold (seconds) for Training Click Sequences
Figure 4.3: Click Sequence Maximum Threshold Evaluation
44
gain. This suggests that clicks made in a shorter period of time should be discarded because they
are more likely noise than directions from the user. Leaving them in would result in a less
accurate, slower model of the user's tasks.
In order to capture the bulk of sequential clicks, we need the upper time threshold to be above
the longest duration a user spends moving the mouse between a sequence of targets. Inspecting
Figure 4.3, five seconds is a sufficient upper time limit on sequences of clicks. Extending the
maximum click sequence threshold beyond that yields minimal gain of captured click sequences
and is prone to include clicks that are not sequences.
Different users had similar, but distinct frequency distributions. Even when they were com-
pleting the same task, they did so in different ways. Some causes for these differences include how
many messages they receive or how much automation they have set up (e.g. the automatic filing
of messages into folders). Other factors were what they click on and how they label or delete
messages. Examples of different behaviours are shown in Figure 4.4. Some users preferred to
click on the sender name for emails; others clicked on the message subject. This may mean users
choose whether to open a message dependent on the sender or subject, respectively. Another
interpretation is that users keep their mouse closer to the left or right half of the screen.
(a) 4.4: Ue
(b)'7i an1
Figure 4.4: User Data that Clicks on (a) Message Sender Names, (b) Message Subjects
45
46
Chapter 5
Browser User Interface
With an accurate model of what the user is likely to interact with, we can dynamically alter
visited pages to cater to the user's needs. This chapter discusses the different ways EasyLink
changes a Web page's look and feel. These changes are referred to as enhancement modules.
The enhancement modules in EasyLink consist of two visual enhancements (size and contrast)
to reduce search time and one motor enhancement (mouse controller) to reduce pointing time. The
size module scales up the size of important elements. The contrast module reduces the contrast
of unimportant elements. The mouse controller module slows the mouse speed over important
elements. Each individual enhancement should benefit either the user's visual search, motor
movement, or both. The programming interface that enhancement modules must implement is
described in Appendix B.
Every time a Web page changes, all enhancement modules asynchronously process the page.
Clickable elements are processed in the order they are likely to be interacted with, starting with
the most frequently used. This provides instantaneous feedback about the elements most likely
to be interacted with and, due to font resizing, finishes page layout changes as quickly as possible.
In order to maintain user agency, not all elements may be processed immediately. Any remaining
47
elements are rescheduled in the JavaScript event queue.
5.1 Visual Changes
Visual changes have the greatest impact on a user's opinion of EasyLink. Whether or not a user
wants to use EasyLink will likely be determined solely from these changes. These changes are
simple and easily understandable so any viewer will understand why they happened: important
elements have their font size increased to improve readability and clicking ease; unimportant
elements have their contrast decreased to reduce distraction.
5.1.1 Font Size
The size module enlarges element font size based on their probability of interaction. Because
page developers often build dependencies into the size of images or tables, changes to the layout
should be avoided. I experimented with shrinking unimportant elements and padding the size to
retain the original layout but any font size change appears to trigger the layout manager. For
this reason, operating on every targetable element is not feasible. Instead, important elements
(element weight > 0.5) have their font size increased in a reasonable amount of time, since the
number of important elements is small. Additionally, text elements generally have slack to their
layout so an increased size doesn't typically disrupt the entire page layout. A scaled view of
Gmail using the size module is shown in Figure 5.1. Unread, recent messages and frequently used
buttons have their font size enlarged. The size module operates on element font-size (in pixels)
as described in Equation 5.1.
fontSize' = fontSize(1.0 + fontScalar(weight - 0.5)) (5.1)
48
Goc ls Gman Cnd Rxgos DocaSnt readthmt Groy3 at.mdje uess essynk.project@gmaLcom1Settings Hel|Sign out
(.a~t jI fSearh Mail I Search the WbShow sftaro sylg1____Web__I Cfeat a file
Compose Mal NYT Travel - Check In. Check Out: New York City: Hotel 17 2 ;:
Archive Report spam I Delete R sMore actions.. J R 1-5 of 31 Olderarred ___ __ _TjACA Seaat AsI None, Read, Unead, Starred. Unstarred
Seats Ma7l f Csail Event Calendar csa ratec TALK:Thursday 1-11-07 A 12:08 am
Drafts Rosaie scarcity - bungler We stopped by Amtc#Jan 10
Al Mail F Matty sulky - curiousl y Parker mad a par of Jan 10Spam (34) FMarsh Political perspectives in the East and %,*Jan 10
T- ACMBulletln ACM CareerNews Alert for Tuesday, Janusa*Jan 9Contacts Elsie charge primarily-rostbe dog ietn r! Jan a
W uck Contacm r Julia rve been banging on howprotectionism doe' Jan 7
-TTT 0 Tr r N R m Reuse) Huge pile of miscellany now a Jan6
4 EasyLink Project f~ Latham odam- vaccination - tripe Musind ce Jd eJan 4Sign into chat r technews iam ACM TechNews; Friday, December 1tJ 121506
Addenaca Zhow f~ Eddy Your competitorswMi think you're nice and i9 12115106
Labelt Archfve IReport Spam ! Delete I -Moreactions.. 250f31Older:csail-related (1) Select All, None, Read, Unread, Starred, UnstarredfamIly 11)random-hall (1)res hall (I Visit settins to save time with keyboard shortcutal
Edit lab els You are currently using 83 M (3%) of your 2804 MB.Gnil view: standard with chat stardard without chat I basic HTML Lean more
SInvite aend2006 Google - Terms of Use - Privacy Poky - Prowram Poices - Gooole Home
Figure 5.1: Size Module: Likely elements such as the top messages in the inbox have enlarged
font size
5.1.2 Contrast
The contrast module adjusts the color of elements to blend unlikely targets into the background.
These changes are less disruptive to the view of the page than those produced by the size module
because contrast changes blend existing element styles together. Text elements have their color
property interpolated between their foreground and background colors. Other elements (such as
images, buttons, and drop-down menus) are handled by setting their opacity since blending their
foreground and background colors won't produce a faded effect in the rendered view.
The contrast module operates on each RGB channel of an element's color as described in
Equation 5.2. The percent opacity was operated on as described in Equation 5.3. A scaled view
of Gmail using the contrast module is shown in Figure 5.2. Older, read messages have their
contrast reduced
49
color' = color + colorScalar(O.5 - weight) (backgroundColor - color)
opacity' = opacity(1.0 - opacityScalar(O.5 - weight))
(5.2)
(5.3)
Originally colors were interpolated as HSV, but often the hue of the foreground or background
was undefined. Additionally, interpolating as RGB results in an interpolated saturation and
value without requiring conversion to and from HSV. I considered using the contrast module to
increase the contrast of likely elements, but there is a limited amount of gain this can produce.
For example, the contrast can not be increased when the background is white and the text is
black.
Gk5 Gmi~tl Clendr PnLs Ds &S~ S sGrs m s ' easyink [email protected] I SetMnas I Help Sian out
Compose Ma NYT TmvP - Chpck In Check C New Yofk Citv, Hotel 17 - :
Drafts
Spam Q4Dha s
- Quick ContactA
I'm ar-h u -N rSEasytink Profect
Sign into chaAdd cna stowW Labelscsail-relate d ()famad 11random-hab (1)
fE1Unk t
P ivte a fiend.
usan rvemr ualenaarRosalie
MattyMarshACMBuietinElsie
JuliaN RLathamtechnewsEddy
Cleelatd ^L^; Iinuraudy I I-UW A nWW a' 1.a0 di
scarcity - bunglertWe Stop by Amrossa ie Jan 10
sulky - curnusly Parer made a par of three- Jan 10
Polifical perspectives in the East and West 19Jan 10
ACM CareerNews Alert for Tuesday, Janul Jan 9
charge prinarly - Iitbit PrOVdmm rnle .oW 9Jan 8
rve been banging on howprotectionism doe19Jan 7
[Reuse) Huge pile of miscellany now a Jan 6
. ra a vaccInation ; i o - a , Jan 4
ACM TechNews: Friday. December 150 12115106
Your competltorswvl think youre nice and i9 1211 SM6
I More actions 1- 25 of 31 . .de.
Visit setngs to save lime with keyboard shortcuts!
You are currenity using 83 MB (3%) of your 2804 MB.G ie: standard with chat standard wyvgt cha I bast HT Lewm moie
2 Gg - Trs _ - Prvacy Poky - Prorm Po;jqes -Goo HV c
Figure 5.2: Contrast Module: Unlikely elements such as bottom messages in the inbox have
muted colors
50
5.2 Mouse Controller
Unlike the other modules, the mouse controller has complex constraints. It runs in real time by
intercepting mouse move events and adjusting the mouse's speed proportional to the weight of
the element it is over.
The mouse speed is controlled to add motor space to an element. A highly emphasized element
may add up to 16 pixels to the effective width and height of the element. This produces a dramatic
improvement for elements with small targets and a negligible improvement for large, easy-to-hit
elements.
The mouse controller uses one of the findings from Chapter 4: clicks within a short time period
indicate a sequence of clicks to perform one task. No state of the mouse controller is directly
observable, so it may dynamically change weights without distracting the user. The result is that
complicated tasks don't create distracters when the user is having the mouse pass over to a target
on the other side.
To alter the mouse's speed, the mouse controller is implemented using the JavaScript event
queue and java.awt.Robot, which issues low-level movement commands to the mouse pointer.
Pseudocode of the implementation is provided in Table 5.1.
The mouse speed is derived by comparing the last known mouse location against the event
location. Focus events and mouse out of window events corrospond to mouse movements that
were not captured by our listener. Since the mouse location is unknown, the last known mouse
data is invalida. Additionally, computationally expensive processes (such as page loads) may slow
processing of the event queue. Since position adjustments should not be noticable, out-of-date
mouse events are discarded.
Mouse speed is calculated for the X and Y axes independently as a function of element
width for each axis and the element weight. New screen positions are calculated and set via
51
processMouseMove(event)
{if ( event queue is valid,
lastTimeStamp exists,
and event.timeStamp - lastTimeStamp < 5 ms )
{offsetPixels = min(maxOffset, element.width, element.height)
offsetPixels *= speedScalar * (element.weight-0.5)
mouseSpeedX = element.width/(element.width + offsetPixels)mouseSpeedY = element.height/(element.height + offsetPixels)
dx = (event.screenX - lastScreenX) * mouseSpeedXdy = (event.screenY - lastScreenY) * mouseSpeedYnewScreenX = lastScreenX + dxnewScreenY = lastScreenY + dy
invalidateEventQueue
robot.mouseMove(newScreenX, newScreenY)
queueEvent (validateEventQueue)
}lastScreenX = event.screenX
lastScreenY = event.screenYlastTimeStamp = event.timeStamp
}
Table 5.1: Mouse Controller Pseudocode
java.awt.Robot.mouseMove. Since there may be other movements in the JavaScript queue, those
also have to be discarded.
Originally element weight linearly controlled mouse speed, but results from the study suggest
that users do not use the entire width of large elements. Instead, they had to do more work at
a higher index of difficulty to move through the weighted element. As a result, mouse speed was
changed to increase effective width rather than effective speed.
52
Chapter 6
Look and Feel Study
The purpose of this study is to verify that the modifications made to the browser facilitated faster
execution of user's tasks. Typical tasks in Gmail were chosen and adjusted so that the user only
had to click on elements, not think or type.
Changes were evaluated both by subject completion time and by qualitative preferences pro-
vided by the subjects. Subjects completed tasks with no changes (the control), mouse movement
changes, and visual changes. At a minimum, the mouse pointing changes should improve the task
completion speed, and visual changes should be preferred by the user.
6.1 Method
Before beginning, each subject was asked background questions about Gmail usage since Ea-
syLink is intended for frequent Web application users (Figure 6.1). All responses from the user
were handled through the Web browser. Each subject completed three common tasks in Gmail:
reporting an email as spam, replying to an email, and labeling a message.
For each task, subjects completed a warmup followed by three randomly ordered test groups:
no changes (the control), visual changes (both size and contrast), and mouse movement changes.
53
Dear Sir / Madam,
I am a student from MIT collecting data for my master's thesis, titled"Improving Target Acquisition in Web Applications Using LinkPrediction". Your participation is voluntary, you may decline to answerany or all questions and at any point you can decline further participationwithout adverse consequences. You will not be asked for your name orother identifying information to assure the confidentiality and anonymity ofthis information.
In a fake Gmail environment, you'll go through the motions of three tasks:marking spam, replying, and filing messages. Each task always beginswith the warmup, followed by three randomly ordered variations. All tasksshould be completed only using the mouse. This study takes 20-40 minutesto complete, depending on how quickly you complete the tasks.
Thank you for participating in this study
How old are you?
Have you used Gmail before?r Yesr No
How many hours per week do you spend browsing the web?
I-
Thank you very much for your participation.
Let's Begin
Figure 6.1: Look and Feel Study: Introduction
54
All weights and visual changes were precomputed and stored in the elements on disk to eliminate
task model implementation details. Element weights were derived from a composite user task
model generated from a normalized union of the Gmail task model data gathered in Chapter 4.
Each warmup and test group ran for at least 100 seconds and either 200 seconds or a fixed number
of repetitions whichever came first. Spam, reply and label tasks had fixed repetitions of 32, 20,
and 23, respectively. These maximum repetition numbers were chosen so that each of the tasks
would have a consistent test duration if the subject completed them quickly.
All pages were displayed on a charging Macbook Pro running Mac OS X Tiger with its screen
resolution set to 1440x900. Subjects were instructed to use only the provided wired Dell infrared
mouse. All tasks took place in a maximized browser window with a centered inner frame document
fixed in size to 835x675 with scrolling disabled. An example of the browser's view is shown in
Figure 6.2. All normal affordances were visible to the user, but only a click on the next element
in the task resulted in a page change. All pages used the study are available in Appendix C.
Subject's mouse events (mouse down, mouse up, mouse move, click) were recorded during the
tasks and analyzed later. Analysis extracted all valid movements beginning and ending with a
mouse down or mouse up. Typically this measured the amount of time from mouse up to the
mouse down at the next target. The only exception to this occurred while labeling a message.
This task required the user to select an option in a drop-down menu, which could be completed
from mouse down on the drop-down menu to a mouse up over the option element. Targets which
took more than one click to successfully click on were discarded as errors.
After all test groups were completed for a task, subjects were asked qualitative questions
about that task (Figure 6.3). Subjects were asked for their favorite and least favorite of the four
variants in the order they completed the tasks (warmup and a permutation of control, mouse,
and visual changes). Subjects were also asked if they had completed similar tasks with Gmail
55
Figure 6.2: Look and Feel Study: Entire Login Screen
previously to evaluate the validity of the tasks.
6.2 Results
Seven subjects volunteered to complete the 45 minute lab study. Subject ages ranged from 19
to 50 with a median age of 23. The number of hours subjects reported having spent browsing
the Web ranged from 10 to 50 with a median of 15 hours. Six of the seven had used Gmail
previously. The total data size (number of completed repetitions) for each task and group is
listed in Table 6.2. The percent of data discarded due to misclicks is listed in Table 6.2.
Warmup Control Mouse Visual
Spam 180 201 213 208Reply 125 132 132 131
Label 144 151 148 152
Table 6.1: Task and Variant Data Size
56
- .---- -- ..- - - - -- - P E
Task Evaluation
In the order they were shown to you, what what was your favoritevariation:
Don't RememberNo Preference1st2nd3rd4th
In the order they were shown to you, what what was your least favoritevariation:
r Don't Rememberr No Preferencer1str 2ndr 3rdC 4th
Had you done this task previously in Gmnail?r Yesr No
Let's Continue
Figure 6.3: Look and Feel Study: Task Exit Questions
Warmup Control Mouse VisualSpam 3.9% 2.8% 2.8% 5.3%Reply 6.2% 6.5% 7.7% 6.9%Label 9.3% 12.3% 7.6% 8.0%
Table 6.2: Task and Variant Error Rates
The results of the spam tasks are shown in Figure 6.4. Each of the three targets (the unread
message, report spam button, and sign out button) were evaluated for each of the four test
groups. Subjects performed the motor variant of the spam task significantly faster than the
control (p < 0.05). Four of seven users claimed to have reported messages as spam previously.
The tests for effects of EasyLink on both the reply and label tasks were inconclusive, due to
less data. Also, users became faster and more error prone at their tasks over time, which further
reduced the difference between test groups. The results of the reply and label tasks are shown in
57
Spam Task Results
Figure 6.4: Spam Task Results
Figures 6.5 and 6.6. Two of seven users claimed to have replied to a message previously. Three
of seven users claimed to have labeled a message. These numbers seem unexpectedly low and will
be discussed in Section 6.3.
Figure 6.5: Reply Task Results
Each task was also evaluated based on the user satisfaction. Of all the test groups, the opinions
regarding visual changes were most contentious as they were both the most preferred and least
preferred changes. Reported preferences for tasks are shown in Figure 6.7.
58
1800
1600
1400
1200
1000
Boo600400
200
STaret 1 Target 2 Target 3
Reply Task Results2000
1800C
1600 Visual
600
400
200
Target 1 Target 2 Target 3 Target 4 Target 5
WarmupControl
NlmouseMvIsued
Label Task Results1500 W
STarget 1 Target 2 Target 3 Target 4 Target 5
Figure 6.6: Label Task Results
Qualltatve Preference
10
8
0Target Conrtr Mousret Visuage No Tret Fog5
6
4
cu62
0 - -
-2
-4
Warmup Control Mouse Visual No Pref Forgot
Figure 6.7: Qualitative Results
6.3 Discussion
The results from the spam task are most promising. The time to click on each target was sig-
nificantly improved by the mouse movement changes. Additionally, visual changes insignificantly
improved the time. The spam task was the first and by far the simplest task provided to the users.
This caused several benefits including more data points and less errors caused by the subjects.
Short of a long term study with different types of users, these findings are quite encouraging.
Most subjects did not report having replied to messages. Since email is normally a two-way
communication, replying to messages should have been the most common task and suggests some
aspect to this task artificial. Subjects may have interpreted the question to mean the entire task,
59
which included the use of Gmail's labels, or the manner in which the task was completed, which
required clicking on the text area rather than the reply button.
The third target and fourth targets of the reply task did not produce expected results. The
third target was a large text area of size 483x60, which is relatively large compared to other
targets whose heights were never greater than 24 pixels. The mouse variant caused users to
spend more effort within the text area than the control. This resulted in higher movement times
to this target. This problem has since been addressed by adjusting the design of the mouse system
to increase motor space by effective width rather than by mouse speed. The fourth target was
simply a poor choice for a pointing task. Since the third target surrounds the fourth target, the
movement time depended more on the cursor's start position than time to move to the fourth
target. This is the most likely cause of the high variance for this target.
The results for the label task were unfortunately biased due to a bug in the mouse controller
at the time. The drop-down menu sometimes caused the mouse cursor to jump to the upper left,
depending on the ordering of events in the JavaScript queue. This may have influenced users to
report the mouse variant of this task as the least preferred, even though they were instructed that
the bug was not intended as part of the study. Both mouse controller issues have been resolved
since the study was completed.
After the study finished, some users expressed their concern that the visual scaling felt artificial
for their task. In fact, since element weights were derived from actual data, weights were not
artificial, but also there were distracter elements that went unnoticed by the subjects. This
feedback may explain the large qualitative contention over the visual changes. Regardless, a user
should be able to configure their view of EasyLink, so users who do not like visual changes could
disable them.
The implications regarding the effect of EasyLink may depend on the type of task. The
60
benefits for various audiences doing more complex tasks should be explored with more data. It
may have been possible to evaluate this by randomly ordering the tasks and grouping all variant
work together, but this change certainly would have increased the variance in the spam task's
results with other confounding variables. Qualitative evaluations would also receive one third its
data if this were the case.
61
62
Chapter 7
Conclusion
This thesis has presented EasyLink, an extension to facilitate expected user interaction in Web
applications. EasyLink's design and performance oriented implementation enable it to support
user tasks in Web applications. The user task model is robust to the user's behaviour. Visual
changes enhance the view of the page as it loads. And the mouse movement makes it easier to
accomplish frequent tasks. This chapter concludes the thesis with an overview of the academic
contributions, and work that should be done to further EasyLink's validity and usability.
7.1 Contributions
There are several contributions this thesis has presented. Since subjects were able to adapt to the
dynamic changes in the mouse controller, procedurally generating motor space changes are not
too confusing to users. EasyLink automated the control display gain using task model weights.
Additionally, the X and Y axes of CD gain were set independently, so there could be a high gain
in one axis and low gain in another.
EasyLink demonstrates that applications are ready for more advanced alterations to the look
and feel of pages based on predictive behaviour. Study results suggest that improving target
63
acquisition benefits the user when key elements are emphasized even with distracter targets.
EasyLink records the user task model with the XPaths of elements clicked on, not the URLs
associated with them. This provides a more robust mechanism to identify what parts of a Web
page are used rather than what Web pages are used next.
EasyLink provides new techniques for transforming Web pages visual and motor space in
real time. Operating on the elements as groups and prioritizing modifications gave immediate
feedback to the user.
7.2 Further Work
EasyLink is not a complete, polished extension. There are several important aspects missing in
its current form, but combined results from both studies are encouraging. Both expected features
and further improvements are missing from its implementation and a breadth of testing on other
applications is needed to validate its usability and effectiveness.
7.2.1 Full Evaluation
The evaluation presented in this thesis is preliminary. Some results of the user study were
significant, but many of the tests were inconclusive or even suggested problems with EasyLink's
implementation. The problems have been addressed with design changes or bug fixes, but more
testing in a non-laboratory setting is recommended.
One way to gather field data is to test volunteers using the extension for a long period of time
on their own computer. Choosing a set of common sequences of clicks would enable us to test the
user by randomly assisting with those tasks and measuring the resulting time. Instances where
we do not assist the user would act as the control group, and instances where we assist the user
would act as the experimental group. Since we know users perform certain tasks in field settings,
64
we can evaluate these tasks without compromising the validity of the study.
7.2.2 Additional Features
EasyLink has been developed and tested with a month of data for a handful of users. For
EasyLink's usability to be sufficient for a larger user base over a longer period of time, EasyLink
needs additional features. EasyLink needs to provide a user interface to its settings, to handle
visual information, to address security issues, to discard unused Web page data, and to further
improve its performance.
Configuration User Interface
Currently there is neither a means to adjust the severity of alteration of documents nor a
means to disable some of the enhancement modules. This user interface does not need to
be complex, but users want high-level control of the extension. A simple preferences dialog
would suffice.
Visual Information
Some content on pages is not intended to be clicked on (such as images and paragraphs
of text). Text is handled by scaling up the weight as though it were one line of text, but
some unwanted text may be more than one line, such as Google advertisements. Since lower
contrast images did not seem as bothersome, images have not been addressed at all. Both
of these are minimal work on visual data and they should be further iterated on.
Keyed Security
In order to support sensitive information concerns, stronger security measures should be
incorporated, as well as a stronger evaluation of the overall security.
65
Long-Term Garbage Collection
A long-term implementation of EasyLink needs to address stale content in its records.
General Web browsing creates many cached files with negligible clicks. EasyLink should
evaluate the longevity of stored data with consideration for the amount of recorded data.
Asynchronous Cache Loading
The largest performance bottleneck in EasyLink is the fetching and preprocessing data from
disk into a usable form. Large data sets like Gmail may take up to 150 milliseconds to pre-
pare. EasyLink should load data asynchronously and block the enhancement modules until
the data is ready. In order to keep EasyLink responsive, it may need further development
of its stored data structure to permit likely targets to be processed immediately.
7.2.3 Additional Web Application Evaluations
Gmail has been the primary testbed for this project as my primary mail client, but a broader
spectrum of applications should be evaluated by people who regularly use them. For example,
EasyLink has given high weights to hard-to-click elements in Google Calendar (Figure 7.1). This
suggests EasyLink would improve the index of difficulty of adjusting scheduled event lengths.
No thorough testing has been done, so an evaluation of other Web applications such as Google
Calendar is recommended.
66
Figure 7.1: Cursory Google Calendar Data. The end interacters of scheduled events have high
weights.
67
68
Appendix A
Task Model Interface
Element[] modelGetElementChunk(node);
Returns the group of Elements to which the Node belongs with the following properties:
the returned array has non-zero length; parent Elements are always earlier in the array than
child Elements; if the Node is an Element, it will be within the array, otherwise its parent
Node will be within the array.
Requires Node != null.
Weight modelGetAffordanceWeight (node);
Returns the Weight information for the specified node, which contains the following infor-
mation:
nodeWeight - The node's current weight irrespective of the previously trained event.
pairedWeight - The node's current weight only considering unpaired data and the previ-
ous target's paired weight.
69
void modelDocument Changed (root ChangedElement);
Notifies the model of all changes to the document. Ensures the document's model data is
loaded and calculates global statistics on usage.
void modelTrainEvent (event);
Notifies the model of an event which it may train.
void modelCloseAllData(;
Notifies the model that it should flush all its data to disk. Used to close all logs on exit.
70
Appendix B
Enhancement Module Interface
void scaleElement(element, weight);
Provides an Element and its associated weight for the module to operate on. The module
should not operate on other elements.
void initializeModuleO;
Initialization hook for the module. Used to construct the mouse controller's Robot and set
up mouse move listeners not available through EasyLink's standard interface.
71
72
Appendix C
UI Study Pages
Spam Task Description
You've received spam, and want to get rid of it properly
Steps:
1. Sign in2. Open the unread message from Roger Hanna3. Report this message as spami4. Sign out
Start the Task
Figure C.1: UI Study Spam Description
Reply Task Description
You're responding to a message sent to reuse to claim some goods.
Steps:
1. Sign in2. Open the reuse label3. Open the unread message from Roger Hanna4. Click in the text area to begin a reply to this message5. Send6. Sign out
Start the Task
Figure C.2: UI Study Reply Description
73
Figure C.3: UI Study Label Description
a% USjf an &U~tQ nu &" 41"Wtan 0 a* amqe N., l I S tfly*
Sals I nwa I--
5"a Vv .. ecrrny.ie su
ainn
YEw " on m amdw uh igon" lrn
Ec'Qm Gogh-Imejn -f ln -
yw Eth* acaun fl
.iynanut Se.I bI t LWa
xww o PN3 -
Figure C.4: Study Spam Page 1: The user clicks on the first, unread message in their inbox.
74
Label Task Description
You've received a bcc'd message and you want to file it appropriately.
Steps:
1. Sign in2. Open up the unread message from Roger Hanna3. Click on the "More actions..." dropdown to get a list of labels4. Apply the label irandom-hall to this message5. Archive this message6. Sign out
Start the Task
w
Ci
p0
~ I
Id ,.
ii~ ~
iV
ii 4-D
*3 Am
~-i~
.I P
-4
E~
~
-e
iril.. 0
Ie U
~fl ~ ~-4-D
eN I-
b
I 41
Gol Erink GC46W DM G ll my ls eagmyb I fsea I I ane
ntMailSDiM
IaMbCImS
searci Semohij "MW
lraemn*,n& Heald Trbuna - Tmwam wifl chanla took of Pans andt S taim - apmm
No new maill Twor's alvveys Gooplt it youe looking Ioom lng ID read.
A Repr pam Dehtt fMor ons . JgoS$1a tH, No"s. Reed. UMa, trv, rsd
Get fti EzU*n Hadiflr ftr fth Mc. Learn mors
You wre arsftoiy g4 O (2%) of your 2791 MILEzU~k Vow ndiniot st I fordaIw NhOl basIc HT LemO
C20W Gooab - Term o - Atya b PL-V - PrMMa Pfoiles - G0nh
Figure C.7: Study Reply Page 1: The user clicks on the label with an unread message.
safmytmc"Moo $411111" ItMMV-6t
fthoo hsrorn
add lad
Sentuall r- R"" HOMMMean inbu"epd un__s__-_rginal 4: pm
AN Mail T 3 on L fto Ri R tsef eMie asirnt at 37311 clie d Cled 3;$4 V
M ML _
hflAA FOUNO $E** TypswM , ThNeis, 3.1pmhalb Jennft L&4hs FAKMJutlM"in UVealInhAII37411 -t+i RAUSOMIs i 2:-Wp
Contsenan ptrueesnt h W* WocW " , poeaffti ph"ou headas 1:47 pon
_T 9msmLutool 1117i* oarn
iiIW F u .......... t.ees .0 OWain0 .MW MomMKA" A W M s" PO"OT 806 M
F OWnl bma m1"aswiar samband, b ue a rn Of as- Undo L. ,J~e $4usC-MpUt C-1t(30aivrols) -nnoogling lid Ole a h 000 3
- Raidt KW WPAK" Def at Plexte Boaitmms MB) Severa desks. K Dec 2fousefill r- Davi L. M&AWn DoakUMUMo- nOOW r uthin EV3V.. 0. Do2
rdf - ------ ------- _ - -. 1-11 Co
- Steven frWs fe"tsuse) 5W(1 DOW) U&NsO010- CLAMED - On SA.Do Dec 2
F NWtw N WAlwW PU* 1 OW) wurewor A radia shack 4C1301 su Doc 2#e a tiend Trvi FrnkItle]AM 7Wco dlin sew I Do 2br om#4 h ) PtsrlW drk*kn o - h itas beas dam 'd ftrn you aM o I lo a D*c2
bFoigur pCSuse Rdrul P 2 hdndeiM - piure Aread a D iciDavi Jalm (2) duuslo~ cIts i onk one C uain Damid Teffolow n Doc 2Dan Vk"y Re: fteu* )150 kfhz pc - CLAMED - caam*d On Owg 1, 21 Dec 2Dave Robertson ptft*e 150 Wt pc - fn ttw batemwt of 35 a Acer PC I bel Doc 1
FHOWOn Ray PWeUW LARGE, EMPTY MEDICMNE BOTTLS -VARMSU 000r-Sony Chapman ftelme} idner cerrigM -Ii a" rmer cmddge b trmachw Doc I
r- N FR 01 laei"I Mr*W In A fte dinli* - ree i w a~ t hur nft m f 1
Figure C.8: Study Reply Page 2: The user clicks on the first, unread message in their inbox.
76
0~ bLInk ~ gM~ ~ [email protected] I~aItOSo It~a~ II~aftOMI
aewmsere s S Whuewe IUM oML
IKOUS9j vimnage wmnvA uumm a ip O3K noimerRoger Hana 'ardoGinILsdIP 403 pm (1 fhourago) Reply
oiginal W disks for lhe following wIngM programs
Cleansweep 95lomega DIO TOGIS
opga p a
Aso, hari pleatc holder for zip disks, roabty holds 3 disks
Email lo dulninclude cwrmus mail addrens
Rspik Reov z all ko w n* *v Roaer to Euk
New wnoPdint a
1-M75MOZip Drives US2.0/hn Slok. Ships N"xt Day, Save 1
2n100=20 Dafta R020naZIPI0MS250MO
res EifmW (O0033-M6OwvwAInewaorkswom
SGIMl ash Drlnnsr4GB Flesh Doives $40Sale Ends Wednesday!Realliquidationzcom
AbouttleselnMk
Figure C.9: Study Reply Page 3: The user clicks on the text area to begin a reply.
|yennraf t&* W W Ii l -
ixausej vimage winwo uimes a zip OWK nomer ma Neein
Roger Mom vrdoamAidj !Lw 4:03 pm (1 how rago) SPly PdInt a
ol 3. disls bre followng wlrginrgm
Olleweea Stan riaassW"wgDa Took 10-750MB p Dves,U SB2V
lonmea ZIP , In, In SIOCK Shis et ty, SeeAlso. hwa piuk holder fm zip dMka.probay hol3 d wwwoliIwf um
MLi2NIMSAMMEnma b aiam Fro* Esimai (800)23"-m4
St4M flubeiftInadenasrmnwt emde..s wanla
era!Wt, Reply a E Roerto .zSGB Rom DdveOSe
Sate SbtemSgSoMe .nla Dv
komm JazOSI Eint~l Mad) Alal Add eiaen2dlDs *
TpM Hennanem
Hey Roger,
rd like the herd plastrc holder for zip disks.
JIM at 7y Mesear'+usetta Avenue
Thanks.EssyUnk.Pojact
On 1214108, Roger Hanne CrdomiLeda> wrote IRelted Pages
Fes PC ddtrs doentoad. Updatne yourayiamnopwtmlywwwyPedverknm
Abutflenh
Figure C.10: Study Reply Page 4: The user clicks on the send button.
77
Comosela~t~msarred
chalssent M"a
ItMil
EtltiedaPgd-
muse
VYOe a 1t
I-m
slonedchMU
Edwn 0 tww s
A- 2m I-- -
ONOM Extink DMI.Swumm"m Gmm MftwIwYt" easintpojctgmllomI S"-ng ldflf tSin
Gbslrsuss SGawshll l Ssamtfqhls WIb *I IS n
R9use] vintage wtn9* Utilities 5 ZIp dISK nOldSr musRager Hanna 1ds-ts - 403 PM (
1 hour ag) Ripy
Owgintal 3.5-fMks for the fomoving W"n5 pograms
Claasesp 95Iomega DIND Tools
omegaP Zinsl al
Also, hard plaskhoklerfo ,zo laiO& probably holds 3 disks
Emal o daimIncude campus nal address
EasyLk* Prjet tO ROge 5:19pm(minulasago) Reuy
Hay Roger,
rd lke the hard plask holder or zip disks.rir at 77 Massachrsfts Avenue
Thanks,d,
Now VindowRIMt all
Collarse allFor ward 0
Spsored unks
100-l70MB Zip Drivs, USS 2D1In Sisk, Shlps Nte Day, Save NwwwwMPMegaMo
hIOM25atIOUF ESWMWl (SOD)33-3IW
400 Flash Dris a4l8GB NOs Ddme $99Sale Enda WsdnsesdaReaffiquidaontiom
Uwre abit
kMM JazV Drv
About i nks
Figure C.11: Study Reply Page 5: The user clicks on the sign out link.
Figure C.12: Study Label Page 1: The user clicks on the first, unread message in their inbox.
78
SturedChIIIIII21105Sent Mail
Daft
IrAID
Labelf
andgmeNreuse
swch MWI sm-c vis ~ &MM~da
Co mn a EsPm". - awmas ?bo"i Mw - Ww-b C* $9t garm 11 *5K5i* paaay'a-sm vhcmb 0I I4a
sawad
ANMallf~- ----
fen raplynsg or fowardlng, 0d01g Vie ubkecdwfi saatanewenversallo.
Yauwsuaaruri uag64WGmI2%)0ypu I M.museEzank ~ slandua Vah abd I sand dis biobl HUM. Lasm
Co2 Goo* -Tmh of EUssmr v I P - PRxMUaoki - HINNO
Nm a3124 a(1 our ao
easyftnk~roject@gmal~comI Settinnm nSion
search h searsc toWeb _="e
Caoon MaM ESPNtam - S-id1 5tack. whtei oaduaon 4vde at OCSc
kw 4 back A Didote | More aRoeso SJ l
hal Dance Troupe Shows Next Weekend mmRefr~. EdtRoger Hannsa ardooft"mftAdua o r-h-f
i af,Irs that time ot term for the obligatory span smal about Dance Trotpef If you didnt go lastsemester then you missed me wearing a clown noe for a dance, and if you don't go this term youmiss me waltzing as a wind-p dof in a Victorian dress. Of course it' not ad about me, there aremany coollprofound/fun dances tie term, so you should come check it outf We have shows at nightTfursday, Friday and Saturday, and in the atemoonr on Saturday and Sunday. See the tet below formore inomatton.
Cheers.
Don't miss&
Gyro~azard
Dance Troupe Concertcastions Contents my so 5ett
Lit& KresgeTheater
Thursday. 12/7 - 8pm $5
WickRswwA
We HaDotnowwW.N
DO* D4GreatcDoe D(Comp
UntquiViedditwwej
More aConce
Cleo P
Ceov PDenvewwwZ
SentmailM
AD Mal%MItnati
Contacts
rfndofwf
FPOLM nd
Figure C.13: Study Label Page 2: The user clicks on drop-down menu and selects the random-halllabel.
ySUL ranqmaorm In IMMIMuW
Ch_ Dance Troupe Shows Next Weekend mo woommhfttMiI RosW HOMNt <rdrtAMaimP ID r41t 3hIw d 12A9m(1fhregot Setfy 0Y
M t't that time of term for the obigetary spn smal about Daom Trospre If you idinl go feet ___k
Igo emester than you missed ma weing a clown nose for adan ,iand If you dort go th6 term youl thdck
miss me waltzing as a vrnd-np doll in a Victorian dress. Of course it's not al about me, there ar wksa.Contacts many coolprofourndfun dances td. term, so you ehould come check ft ut0 We he. shows at night wwwAdriday and Saturday, and In the eftemoon on Satunday and Sundey. See thetext below for
Dofto
RowgDO
se Ed O rseloe O
D Tome T CCOMCM
Op a"ndD n't mi1ss& w4NMIs
Gyro~azardcao
Dance TroupeConcertCatiou. C.esasea MSy s. Ves
Little Kmsg Theater
Thursday, 12/7 - 8pm $5
Figure C.14: Study Label Page 3: The user clicks on the archive button.
79
Cleo FCleo PDam
QgggEttnk Cal Od r Dos~ad29MP G at aw Emservko%
2QR* etu Dou a Sprolsts Graum alfr ovc 0&SY**nkPro tG2-" E I 182Mt~aali|I~inoSwcth MMi MW S1r1, bh e "" *"
Te OaWWr.W bn*enmcu b WC UMl
DmhtAll MailM M
Comacts
Ea*mAn* Prood
VT.*o al w
No new mlli Them's ahways Goo*l NWYOV 1fyu 61(iong fr sOWehN" ID rMW
NOO 8I m.', acOOnOtAon1ms
Selet M, 190e.Read. UrftW, StWW.Uft e
Add phone numbems, no*sand more lm.r t people In your CIaIh Lear mom
You ae cummniy vsig64 = P%) of yo 2791 MI.Ezk view: suindard wth chak I sbwndad wBI&hucAt I bair HTLwwr mom
COM5 Google - Terasn ofe . Ptwmw v - POManM POW"l - GCOfts o
Figure C.15: Study Label Page 4: The user clicks on the sign out link.
80
Bibliography
[1] V. Anupam, J. Freire, B. Kumar, and D. Lieuwen. Automating Web Navigation with the Web-
VCR. Computer Networks: The International Journal of Computer and Telecommunications
Networking (2000), 503-517.
[2] R. Armstrong, D. Freitag, T. Joachims, and T. Mitchell. WebWatcher: A Learning Appren-
tice for the World Wide Web. Proceedings of the AAAI Spring Symposium on Information
Gathering from Distributed Heterogeneous Environments (1995), 6-12.
[3] C. Asakawa and T. Itoh. User Interface of a Home Page Reader. Proceedings of the third
ACM conference on Assistive Technologies (1998), 149-156.
[4] R. Barrett, P. Maglio, and D. Kellem. How to Personalize the Web. Proceedings of the SIGCHI
conference on Human factors in computing systems (1997), 22-27.
[5] P. Baudish, E. Cutrell, D. Robbins, M. Czerwinski, P. Tandler, B. Bederson, and A. Zier-
linger. Drag-and-Pop and Drag-and-Pick: Techniques for Accessing Remote Screen Content
on Touch- and Pen-Operated Systems. Proceedings of Interact (2003), 57-64.
[6] A. Bestavros. Using Speculation to Reduce Server Load and Service Time on the WWW.
Proceedings of the fourth international conference on Information and knowledge management
(1995), 403-410.
81
[7] R. Blanch, Y. Guiard, and M. Beaudouin-Lafon. Semantic Pointing: Improving Target Ac-
quisition with Control-Display Ratio Adaptation. Proceedings of the SIGCHI conference on
Human factors in computing systems (2004), 519-526.
[8] M. Bolin, M. Webber, P. Rha, T. Wilson, and R. Miller. Automation and Customization of
Rendered Web Pages. Proceedings of UIST conference, (2005), 163-172.
[9] S. Brown and P. Robinson A World Wide Web Mediator for Users with Low Vision Proceedings
of the SIGCHI conference on Human factors in computing systems (2001), Workshop n.14.
[10] E. H. Chi, P. Pirolli, K. Chen, and J. Pitkow. Using Information Scent to Model User
Information Needs and Actions on the Web. Proceedings of the SIGCHI conference on Human
factors in computing systems (2001), 490-497.
[11] B. Davison. Predicting Web Actions from HTML Content. Proceedings of the thirteenth
ACM conference on Hypertext and Hypermedia (2002), 159-168.
[12] M. Deshpande and G. Karypis. Selective Markov Models for Predicting Web Page Accesses.
ACM Transactions on Internet Technology (2004), 163-184.
[13] P. Fitts. The Information Capacity of the Human Motor System in Controlling the Amplitude
of Movement. Journal of Experimental Psychology 47 (1954), 381-391.
[14] T. Grossman and R. Balakrishnan. The Bubble Cursor: Enhancing Target Acquisition by
Dynamic Resizing of the Cursor's Activation Area. Proceedings of SIGCHI conference on
Human factors in computing systems (2005), 281-290.
[15] Y. Guiard, R. Blanch, and M. Beaudouin-Lafon. Object Pointing: A Complement to Bitmap
Pointing in GUIs. Graphics Interfaces (2004), 9-16.
[16] Hotmail. http://www.hotmail.com. (2006)
82
[17] P. Kabbash and W. Buxton. The "Prince" Technique: Fitts' Law and Selection using Area
Cursors. ACM CHI Conference on Human Factors in Computing Systems (1995), 273-279.
[18] D. V. Keyson. Dynamic Cursor Gain and Tactual Feedback in the Capture of Cursor Move-
ments. Ergonomics 12 (1997) 1287-1298
[19] H. Lieberman. Letizia: An Agent that Assists Web Browsing. Proceedings of the International
Joint Conference on Artificial Intelligence, (1995).
[20] I. MacKenzie. Fitts' Law as a Research and Design Tool in Human-Computer Interaction.
Human-Computer Interaction 7 (1992), 91-139.
[21] M. McGuffin and R. Balakrishnan. Acquisition of Expanding Targets. Proceedings of the
SIGCHI conference on Human factors in computing systems (2002), 57-64.
[22] V. Padmanabhan and J. Mogul. Using Predictive Prefetching to Improve World Wide Web
Latency. ACM SIGCOMM Computer Communication Review v.26 n.3 (1996), 22-36.
[23] J. Richards and V. Hanson. Web Accessibility: A Broader View. Proceedings of the 13th
conference on World Wide Web (2004), 72-79.
[24] R. Sarukkai. Link Prediction and Path Analysis using Markov chains. Computer Networks:
The International Journal of Computer and Telecommunications Networking (2000), 377-386.
[25] M. Spiliopoulou. Web Usage Mining for Web Site Evaluation. Association for Computing
Machinery. Communications of the ACM; (2000) 127-134.
[26] Wayback Machine. http://www.archive.org/web/web.php. (2006)
[27] A. Worden, N. Walker, K. Bharat, and S. Hudson. Making Computers Easier for Older
Adults to Use: Area Cursors and Sticky Icons. Proceedings of the SIGCHI conference on
Human factors in computing systems (1997), 266-271.
83
[28] M. Zajicek, C. Powell, and C. Reeves. A Web Navigation Tool for the Blind. Proceedings of
the third ACM conference on Assistive Technologies (1998), 204-220.
[29] S. Zhai, S. Conversy, M. Beaudouin-Lafon, and Y. Guiard. Human On-line Response to
Target Expansion. Proceedings of the SIGCHI conference on Human factors in computing
systems (2003), 177-184.
84