consistency in web design from a user perspectivebachelor's thesis 15 credits consistency in...

Bachelor’s Thesis15 credits

Consistency in Web Design from a User Perspective

Konsekvent webbdesign utifran ett anvandarperspektiv

Anton [email protected]

Exam: Bachelor of Science 180 creditsSubject area: Computer ScienceProgramme: Information ArchitectureDate of final seminar: 2012-09-20

Examiner: Bengt NilssonSupervisor: Goran Hagert

Abstract

Within Human-Computer Interaction, it has long been speculated that in-consistency impedes the user experience. However, defining and categor-ising consistency has been shown to be a challenging task. Several studieson the subject have categorised consistency with mixed perspectives of thesystem, its developer, and its user. The present thesis considers only theuser perspective, and categorises consistency into Perceptual, Semantic,and Procedural consistency. 21 subjects, with moderate experience in us-ing the web, participated in an experiment designed to explore the effectinconsistency might have on usability. In order to test both main and in-teraction effects between the three proposed consistencies, the experimentwas based on a full 2× 2× 2 factorial design for repeated measures. Theparticipants’ task was to use eight partly different versions of a mock-upweb shop in which a subject selection drop-down menu was experimentallymanipulated. Multiple Analysis of Covariance revealed that Perceptualand Procedural inconsistency affected user performance negatively. It alsoindicated that inhibitory interaction effects occurred between some of the(in)consistencies. The results have important implications for web de-velopers in designing usable applications. By adapting a user perspective,they can aid users to avoid performing faulty actions.

Keywords: consistency, usability, web design, HCI, user perspective

Sammanfattning

Inom människa-datorinteraktion har det länge spekulerats huruvida in-konsekvent design påverkar användarupplevelsen. Att definiera och kate-gorisera olika typer av konsekvens har visat sig svårt. Flera studier påområdet har kategoriserat typer av inkonsekvens med blandade perspek-tiv av såväl systemet, dess utvecklare samt dess användare. Denna uppsatssätter användarens perspektiv i fokus och kategoriserar typer av inkon-sekvens i perceptuell, semantisk och procedurell konsekvens. 21 personer,med måttlig erfarenhet av att bruka nätet, deltog i ett experiment utfor-mat att utforska effekterna av inkonsekvent design på användbarhet. Föratt pröva såväl huvud- som interaktionseffekter baserades experimentetpå en fullständig 2 × 2 × 2 faktordesign för upprepade mätningar. Del-tagarnas uppgift var att använda åtta prototyper av en webbutik där endropdownmeny för ämnesval utsattes för experimentell manipulation. Entrevägs variansanalys med kovariat visade att perceptuellt och procedu-rellt inkonsekvent design påverkade användarupplevelsen negativt. Resul-taten pekade också på att hämmande interaktionseffekter uppstod mellanvissa av de tre inkonsekvenserna. Resultaten ger viktiga implikationer förwebbutvecklare när de skall utveckla användbara applikationer. Genomett användarperspektiv kan utvecklare hjälpa användare att undvika fel-aktiga handlingar.

Nyckelord: konsekvens, användbarhet, webbdesign, MDI, användarper-spektiv

Contents1 Introduction 1

1.1 Previous Research on Consistency . . . . . . . . . . . . . . . . . 11.2 Redifining Consistency within HCI . . . . . . . . . . . . . . . . . 41.3 Purpose and Expected Outcomes . . . . . . . . . . . . . . . . . . 6

2 Methods 92.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.5 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Results 153.1 Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Handling Covariance . . . . . . . . . . . . . . . . . . . . . . . . . 153.3 User Satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.4 User Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.5 User Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4 Discussion 214.1 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.3 Further Research . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

A Baseline Choice Selections 29

B Variations of Inconsistency in Subject System 30

C Task Compendium 33

D Consent Form 44

1 Introduction

It has long been debated whether or not consistency is an important factorof usability within Human-Computer Interaction (HCI). Shneiderman (1992)claimed consistency to be the primary concern when putting forth his heur-istic eight golden rules of dialogue design. Grudin (1989) argued antitheticallyagainst this by pointing out instances where consistency might counteract usab-ility. Reisner (1981) highlighted early on that there is very little agreement as towhat consistency actually is. Many have since tried to approach an explanationby singling out and categorising different types of consistencies (e.g., Grudin,1989; Kellogg, 1987; Tanaka, Eberts & Salvendy, 1991).

Findings from studies and experiments on the subject have since been broughtforward, disclosing mixed results (e.g., Adamson, 1996; AlTaboli & Abou-Zeid,2007; Kellogg, 1987; Mendel & Pak, 2009; Ozok & Salvendy, 2000; Satzinger,1998). Therefore, the answer as to what consistency within HCI should entailremains elusive.

The aim of the present study is to explore whether consistency affects usab-ility, by an experiment testing the main effects and interactions of Perceptual,Semantic, and Procedural consistency. In contrast to previous research, we ex-plicitly define consistency entirely from the user’s, rather than the developer’sor system’s, point of view.

1.1 Previous Research on Consistency

1.1.1 Difficulties in Defining Consistency

Attempts at capturing the essence of consistency were made already in the timesof terminal based operating systems and applications. The focus of research wasinitially aimed at consistency in command language (e.g., Barnard, Hammond,Morton, Long & Clark, 1981; Payne & Green, 1986; Reisner, 1981).

At the ACM CHI’88 conference in Washington, a workshop was held with15 usability experts aiming to agree on a joint definition of consistency for HCI(Nielsen, 1989). An agreement was never met and, therefore, the term still hasnot been given an adequate definition. Instead, it has been, and still is, usedimplicitly under its conventional meaning.

Generally, it is claimed of consistency within HCI, that it allows users toreason analogically and thereby predict actions within novel tasks (Blake, 1986;Mayhew, 1992). Others reason that consistency means that similar actions leadto similar results (Wolf, 1989; see also Shneiderman, 1992; Wiecha, Bennett,Boies & Gould, 1990).

Kellogg (1989) argued that consistency is meaningless on its own, thus ren-dering a guideline with consistency as an independent goal meaningless. Con-stantine and Lockwood (1999) made a similar claim saying that striving forconsistency for its own sake might lead to “consistently bad solutions” (p. 62).

Conversely, it has been claimed that there are instances where inconsisten-cies are desirable. Shneiderman (1992) mentions for example how passwordsshould not be echoed to users and that delete operations should result in a con-firmatory prompt, also pointing out that these types of inconsistencies shouldbe kept at a minimum.

With all these separate views on consistency it is no surprise that Smith, Irby,

1

Kimball, Verplank and Harslem (1982) deemed it the most difficult characteristicto attain within system development.

1.1.2 Frameworks for Consistency

Grudin (1989) requested a new way of viewing consistency within HCI. He putforth an antithetical view that consistency of a user interface is an unwork-able concept and said it to lead designers astray, disguising good design as aninterface property.

After the ACM CHI’88 workshop he came to the conclusion that work ondefining consistency should be restarted. As a way of starting anew, he proposedthree categories of consistency: (1) internal, (2) external, and (3) analogue. Theinternal category comprises consistencies within an application (or web site), theexternal category comprises consistencies between applications or platforms, andthe analogue category comprises consistencies with conceptual metaphors, suchas the desktop metaphor (Smith et al., 1982). He also hinted at a fourthcategory, (4) veridical, which would be an interface fully consistent with itssystem’s design (rather than the user’s needs).

Subsequent to the theoretical work of Grudin (1989), attempts at definingconsistency was abandoned and instead work was aimed at finding types andcategories of consistencies.

Building on Moran’s (1981) ideas of interface levels, Kellogg (1987) con-structed a framework consisting of (1) conceptual, (2) communicational, and(3) physical consistency, which were all either internal or external. Conceptualconsistency concerned correspondence to metaphor and mapping between userand system tasks. Communicational consistency concerned interaction betweenuser and system, and physical consistency was related to screen layout.

Tanaka et al. (1991) distinguished between two types of consistencies: (1)cognitive and (2) display. Cognitive consistency is characterised by consistencyin user knowledge, whilst display consistency is defined as consistency in screenlayout.

Somberg (2000) argued that too much focus within development of interfacestandards had been aimed at the look and feel of systems. A new approach calledfunctional user interface standards, based on object-oriented programming, wasproposed to deal with task performance, thereby facilitating what he calledprocedural consistency.

1.1.3 Measuring Usability

In order to measure the usability of a system, usability metrics are used. Twocategories of metrics are normally used in experiments on consistency withinHCI: (1) performance metrics and (2) self-reported metrics. Performance met-rics can be measured through the time it takes to complete a task (time-on-task),number of errors made, or the number of clicks made (click rate), whereas self-reported metrics are measures reported by users themselves, such as how theyrate an application or how difficult they found a task to perform (Tullis &Albert, 2008).

2

1.1.4 Preceding Experiments

Adamson (1996), Ozok and Salvendy (2000), and AlTaboli and Abou-Zeid(2007) found physical inconsistency to increase user errors. AlTaboli and Abou-Zeid (2007) also found evidence that it affects user satisfaction.

Adamson (1996) found indications that communicational inconsistencies af-fect user performance and satisfaction, which Ozok and Salvendy (2000) didnot.

Kellogg (1987) found that conceptual inconsistencies affected both user per-formance and satisfaction, whilst Ozok and Salvendy (2000) found no such evid-ence.

Mendel and Pak (2009) pointed out the fact that the mixed, and sometimeseven detrimental, results of all these experiments may lie in the manipulation oftask difficulty. Adamson (1996) used a combination of radio buttons and dropdown menus to create his inconsistencies. It is both interesting and surprisingthat he found evidence for effects on user performance and satisfaction for sucha simple experiment. One factor might be that nearly 30 percent of his subjectshad very little experience of using graphical user interfaces. Ozok and Salvendy(2000), on the other hand, had participants more experienced with user inter-faces, which could account for the diametrically opposite results of Adamson(1996) with regard to communicational inconsistency.

Another possible explanation for the mixed results of these experiments isthat experimenters might be tainted by their own interpretations of what isconsistent and what is not. Satzinger (1998) carried out a study on consistencyof conceptual models. He found no evidence that his manipulation affecteduser performance or satisfaction. He did, however, find indications that “moreaccurate mental models might be developed when conceptual models are incon-sistent” (p. 11, original emphasis). Participants in his experiment were asked tocarry out tasks in two separate systems. All participants used a system calledNUCLEUS as a first system. Half of the participants used a second system con-sistent in action grammar to NUCLEUS and the other half used a second systeminconsistent in action grammar to NUCLEUS. For example, in the inconsistentversion the action ‘Delete’ was called ‘Erase’. The question is whether the terms‘Delete’ and ‘Erase’ are really conceptually inconsistent from a user’s point ofview. Perceptually they look different, but semantically they mean the samething, and in the two systems they resulted in the same action. This wouldmake them conceptually consistent. If, on the other hand, the two terms hadbeen used within the same system with the same outcomes then this could beperceived as inconsistent, and would thereby possibly confuse users.

1.1.5 Consistency from the Developer’s Perspective

The current view on consistency seems to be founded in a developer’s perspect-ive. Satzinger’s (1998) study on conceptual consistency illustrates this, but themost prominent example is one of Grudin’s (1989) arguments of when incon-sistencies are desirable. Although acknowledging the fact that what a developer(or designer) might consider consistent might as well be inconsistent in the eyesof the user, he seems to overlook this in his example.

The example is one from early word processors, where a user selects a phrasein order to italicise it through a menu. The next time the user selects a phrase

3

and enters the same menu, the italic option is preselected. Thus, the system con-sistently preselects previously used options to ease word processing. However,when the user returns to the menu after copying a selection to the clipboard,this time the paste option is preselected. According to Grudin (1989), this isa facilitating inconsistency in the word processor. But, this example is onlyinconsistent when considered from the developers reasoning when building thisfeature. If we instead look at it from a user’s perspective, this feature is consist-ently facilitating the user with what action the user most probably would liketo perform next.

Thus, it is proposed that we shift our focus from a developer’s, and evena system’s, perspective to one which only considers the user’s perspective withregard to consistency in HCI.

1.2 Redifining Consistency within HCI

In recent years outspoken professional web developers have claimed consistencyto bear no importance, at least with regard to web usability (37signals, 2006;Hurst, 2004). Instead they advocate that user needs should be prioritised overinterface consistency. Hurst coined this way of reasoning intelligent inconsist-ency.

The shift of concern regarding consistency from Shneiderman (1992) sayingits vital, via Grudin (1989) calling it misleading, to Hurst (2004) deeming it isinsignificant, is possibly due to the fact that consistency has become easy totake for granted within application development. This because it has becomeeasier to achieve over the years thanks to concepts such as object-oriented pro-gramming and graphical user interface libraries. These ensure reuse of codeand graphical elements. The definitions of HTML standards do the same forweb site applications, as do the multitude of usability guidelines which manydevelopers and designers adhere to.

This is possibly why Hurst (2004) considers consistency a non-issue when itcomes to web site design. However, it does not alter the necessity of consistencyfrom a user’s point of view. It might be somewhat easier to achieve, thanksto a tremendous work of standardising the development of applications, butconsistency is still of fundamental importance. It is therefore vital to come toa conclusion as to what consistency really is and decide on a definition of whatit means within HCI.

1.2.1 Etymology of ‘Consistency’

The term ‘consistency’ is derived from the Latin word consistere whose literalmeaning is “to stand still”. Encyclopedia Britannica (2012) defines ‘consistent’as “not having or showing any apparent conflict”, whilst The Free Dictionary(2012) defines the term as “[r]eliability or uniformity of successive results orevents”. Thereby, consistency is purely relational and concerns perceived uni-formity between two or more occurrences. Related words include ‘coherence’,‘familiarity’, and ‘regularity’.

The opposite of consistency is inconsistency, which entails that irregularpatterns emerge when two or more occurrences are compared, between whichwe would expect regularities. This, as we shall see, can be crucial for decisionmaking.

4

1.2.2 Cognition and Consistency

In a fire’s seemingly haphazard behaviour there are regular patterns which anexperienced firefighter can interpret. In order to explain the inner workingsof intuition, Klein (1999) referred to a particular case that illustrated how acommander reacted on a gut feeling to flee when a fire did not act as anticip-ated, and thereby he saved the lives of himself and of a group of several otherfirefighters.

Tiny clues in the environment can be picked up by humans and other an-imals. Regular patterns are subconsciously registered upon which decisions arebased. All species alive today are the product of their ancestors’ decisions.By evolving in an environment with regularities, we have learned what to eatand what to avoid. Poor decisions means being left out of the gene pool. Thesurvivors have thereby added refined skills of pattern recognition to succeedinggenerations by making the right choices (Shermer, 2011).

We are dependent on our ability to recognise inconsistency for survival. Werecognise familiar patterns, or acknowledge disruption of patterns, and uponthis we base our decisions. Our innate ability to recognise inconsistency alertsus also when interacting with computers, and the conventional view is that itimpedes usability.

1.2.3 Definition and Categorisation

As previously acknowledged, through the dictionary definitions, consistency isa relational concept. It can be argued that it is therefore also a subjective,observational phenomenon. In order to have consistency, you need an observerwho apperceives. Thus, in HCI, this implies that consistency of a web site cannever exist independently from its user’s experience and expectations; they areintricately intertwined because consistency is founded in the user’s apperception.

With the view that consistency should be redefined from a user perspective,the need for a new definition of consistency within HCI also arises. The followingdefinition will be used for this study:

Consistency is the user’s apperception of regularities within a sys-tem, leading the user to actions in the task environment based onprevious experiences.

This leaves Grudin’s categories somewhat counterintuitive because they arebased on a claimed consistency’s origin, and therefore bears little relation tothe user. The veridical category leaves the user completely out of the picture,whilst the analogue category is unreliable because it is dependent on knowledgetransfer. Studies have shown that there are no guarantees that the knowledgetransfer will occur on its own, without connections being explicitly pointed outto the learner (or user; Gick & Holyoak, 1980 in Barnett & Ceci, 2002). Internaland external consistencies can be somewhat useful as a means of discussingwhether an argued consistency is derived from, for example, industry standards(external) or company guidelines (internal).

Based on previous work on categorising and developing frameworks for con-sistency (e.g., Grudin, 1989; Kellogg, 1987; Tanaka et al., 1991), we proposea new categorisation, redefined from a user’s perspective, taking cognition into

5

consideration. This is motivated by the subjective nature of consistency, as dis-cussed in Section 1.2.2. Previous categories mixed features of the system withthe user’s cognitive abilities. These must be kept apart. The three redefinedcategories proposed are Perceptual, Semantic, and Procedural consistencies.

Perceptual ConsistencyPerceptual consistency has to do with what we perceive when interacting with,for instance, a web site. Most commonly it will be through the visual perception,but can of course be auditory for visually impaired. This includes the use ofcolours and graphical elements, adhesion to the gestalt principles (e.g., Kanizsa,1979), and so forth. For example, the use of the same fonts and font sizes ina text in a paragraph facilitates reading. Perceptual consistency is similar toKellogg’s (1987) physical consistency, although with a shift in focus from screenlayout to the user’s perception of the system.

Semantic ConsistencySemantic consistency is derived from our semantic memory for facts and words.It has to do with consistent use of symbols and icons in correct context; if whathas been perceived is consistent in meaning so that the correct action can betaken in the next step. When a system is semantically consistent it facilitatesthe user’s conceptual model (Foley & van Dam, 1982). The term “semantic”is here used in its traditional meaning, but with the amendment of includingthe position of objects and symbols. In interaction design, the placement ofan object is as important as its appearance for inferring the object’s purposein the same way that words can have different meanings in different contexts.Examples of Semantic consistency include that all anchors should be colouredblue and underlined, or that submit buttons should appear in the same placein all forms throughout a web site.

Procedural ConsistencyProcedural consistency has to do with whether action types and sequences ne-cessary to achieve a goal are consistent or not. For example, all drop-downmenus need one click to reveal options, and another click to choose one of therevealed options. Procedural consistency is similar to Kellogg’s (1987) commu-nicational consistency with the alteration that it only takes the user’s actionsinto account when interacting with the system. In the present model, the sys-tem part of the communication is considered under Perceptual consistency, inhow the user perceives feedback from the system.

1.3 Purpose and Expected Outcomes

The area of consistency within HCI lacks in clear theories. As mentioned earlier,the conventional view is that inconsistency impedes usability, but (in)consistencyhas previously only been loosely defined. Our purpose is to explore whether ornot Perceptual, Semantic, and Procedural inconsistencies affect user perform-ance and satisfaction, and if there are any interaction effects when combiningthese inconsistencies. Empirical results on the relationships between differentkinds of inconsistency, based on the user’s point of view, is expected to contrib-ute to the development of theory of consistency within HCI. Adapting a user’s

6

perspective justifies the need for conducting a behavioural experiment.Drawing on previous research (Adamson, 1996; AlTaboli & Abou-Zeid,

2007; Kellogg, 1987; Ozok & Salvendy, 2000) it is expected to find maineffects on both user performance and user satisfaction from the three kindsof inconsistencies. Importantly, it is unusual to consider interaction effects instudies of inconsistency, but drawing on previous results, any interaction effectsbetween the three inconsistencies are expected to be cumulative, leading to anamplified decrease in user performance, as well as, satisfaction.

7

2 Methods

2.1 Participants

A convenience sample of 21 participants was recruited for the experiment (7women, 14 men; Mage = 31 years, SD = 7.04 years). The minimum computerexperience was 10 years. Of the 21 participants, 8 had either begun or completeda university degree in, or worked professionally within a computer related field.On average, the participants reported spending 3.5 hours each day actively onthe Internet. They also reported having, on average, moderate experience (3on a 4-point category scale) filling out forms on the web, as well as purchasingproducts over the web.

2.2 Material

Data was collected in a computer lab at the Malmö School of Technology. Eachparticipant was assigned a computer, running Windows 8, and the assignmentswere performed using eight Web site versions of a mock-up web shop runningin a Firefox (v14.0.1) browser. Each participant was also given a pen and anassignment compendium, consisting of a cover page, an introduction, and eightnumbered tasks (an example compendium is enclosed in Appendix C).

2.2.1 Mock-Up Web Shop

The web shop was built using PHP, HTML5 and JavaScript (Figure 2.1b). Theback-end was running on an Apache server and utilised a MySQL database. Alluser activity, key strokes and mouse clicks, was logged through JavaScript andsent in the background through AJAX requests to the server.

Throughout the web shop, sequences of drop-down menus were used to selectcategories and subcategories of options (see Figures in Appendix A & B).Initially, in the consistent, or baseline, version of the web shop, only one drop-down menu for subcategory selection was visible. Once an option was selected inthe visible drop-down menu, the next subcategory drop-down menu appeared.This system of drop-down menus was used to select newsletter options, expirydate for credit card payment, and subject categories in a contact form, in threeseparate parts of the web shop.

The subject drop-down menu system (henceforth, subject system) in theweb shop’s contact page was subjected to experimental manipulation, whereasthe newsletter and expiry date selections were kept constant in the experiment.The rationale behind this design was that inconsistency is relational, and maythus only occur in relation to a baseline. Hence, in the present experiment thedrop-down menu system for selecting newsletter options and expiry date optionsfor credit card payment was used as the baseline.

All combinations of presence/absence of the three inconsistencies (Percep-tual, Semantic, and Procedural) were used in a full 2 × 2 × 2 factorial designto alter the subject system on the contact page. This resulted in eight differentversions (W1-8) of the web shop, or rather, eight different versions of the subjectsystem on the contact page depicted in Figure 2.1c. The figures in AppendixB depicts the manipulations of the subject system.

9

(a) Introductory demographics form. (b) Web shop mock-up.

(c) Contact page. (d) Post task evaluation.

Figure 2.1: Web site setup.

Web site 1 (W1) included none of the inconsistencies (Figure B.1). Thereby,the subject system on the contact page was fully consistent with the baselinenewsletter and expiry date selection.

Web site 2 (W2) included Perceptual inconsistency in the subject system onthe contact page when compared to the baseline drop-down menus (Figure B.2).In this version the subject system was replaced by buttons and radio buttons,this in order to change its visual appearance compared to the baseline. A webdeveloper might argue that this would be a Semantic inconsistency, referringto the semantic web. However, the manipulation is carried out from the user’sperspective, in such a manner that it only affects how the subject system looks,keeping the interaction of subject selection constant.

Web site 3 (W3) included Semantic inconsistency in the subject systemon the contact page when compared to the baseline drop-down menus (FigureB.3). The drop-down menu system was kept. However, the drop-down elementwas given the visual appearance of the baseline element label. Conversely, theelement labels were given the visual appearance of the baseline drop-down ele-ment (see Figure B.1). In addition, the element label was placed to the rightof the drop-down element instead of above as in the baseline version. Thus, thebaseline semantics were interchanged; an orange frame and yellow backgroundwith black text represented the drop-down element in the baseline, but here rep-resented a label, whilst black text on white background represented the elementlabel in the baseline, but here represented the drop-down element.

Web site 4 (W4) included Procedural inconsistency in the subject system

10

on the contact page when compared to the baseline drop-down menus (FigureB.4). The drop-down system was kept. However, when choosing a subjectsubcategory in this version, the subsequent subcategory drop-down menu didnot appear automatically. Instead, the user had to click on an orange arrowto the right of the subcategory in order to reveal the subsequent subcategorydrop-down menu. This obliged the user to carry out additional actions (pointingand clicking) in the procedure of choosing the contact subject. This resultedin an inconsistent procedure when choosing contact subject compared to thebaseline’s automatically appearing drop-down menus.

Web site 5 (W5) combined Perceptual and Semantic inconsistencies (FigureB.5). Thereby, the subject selection drop-down menus on the contact page werereplaced by buttons and radio buttons, and also, an interchange of semanticswas carried out in a similar fashion of W3.

Web site 6 (W6) combined Semantic and Procedural inconsistencies (FigureB.6). The exact interchange of semantics applied in W3 was used in this version,along with the need to click the orange arrow in order to reveal the subsequentsubcategory drop-down menus.

Web site 7 (W7) combined Perceptual and Procedural inconsistencies (Fig-ure B.7). Buttons and radio buttons replaced the drop-down menus and theorange arrow had to be clicked to reveal the subsequent subcategory buttonsand radio buttons.

Web site 8 (W8) combined all three inconsistencies (Figure B.8). Thus, thecontact page used buttons and radio buttons, along with interchanged semanticsof the elements in line with W3, together with the need to click the orange arrowto reveal subsequent subcategory buttons and radio buttons.

2.2.2 Task Sheets

The participants’ task was to assist eight fictive customers in buying three differ-ent products from the web shop. For this purpose the participants were providedeight unique task sheets (T1-8; Appendix C) holding pretend customer inform-ation, including contact, as well as, credit card details. In addition, all tasksheets included (1) three unique products to be purchased in the web shop, (2)customer request to obtain a specific newsletter, and (3) a contact message fromthe customer. On all eight task sheets, the customer message was kept to anaverage length of 164 characters (Range: 160-168 characters). All eight taskswere designed to first introduce the participants to the consistent, baseline fea-tures of the web shop before they used the experimentally manipulated contactform.

2.3 Design

As indicated above, the experiment was conducted with a within-subject, 2 ×2×2 factorial design. The Perceptual, Semantic and Procedural inconsistencieswere the independent variables, where the two levels of the factors representedpresence or absence. One web site in combination with one task (e.g., W6:T2)was considered a treatment for each participant. All participants were giveneight treatments.

An irregular Latin Square design was adapted in order to counterbalancethe task, as well as, web site order. To minimise carryover effects, four criteria

11

were established for the design of a basic 8× 8 design matrix:

(1) Every one of the eight participants, in a basic design matrix, should performall eight tasks (T1-8), and evaluate all eight web sites (W1-8).

(2) Each web site (W1-8) should be evaluated using all eight tasks (T1-8).Thus, 8(Websites)× 8(Tasks) = 64 combinations were to be used.

(3) Every one of the eight participants, in a basic design matrix, should use allthe eight web sites (W1-8), combined with a task, in a unique order (i.e.,Latin Square).

(4) A basic design matrix should be organised in such an order that each of theeight web sites (W1-8), combined with a task, should be followed by anyother web site only once (i.e., irregular Latin Square).

The four criteria resulted in a design matrix for 8 participants. The pro-cedure was then repeated to create a different design matrix for another 8 par-ticipants. In all, it is possible to create 16 unique design matrices based onthese four criteria. The design means that at least 8 — and thereafter multiplesof 8 — participants are necessary in order to secure that an imbalance in thecombination of tasks and web sites will not influence the experimental results.

2.4 Procedure

Data was collected at three separate group sessions with 7 different participantsat each occasion. Each session started with the experimenter reading an intro-duction aloud for the participants. The participants were asked to read alongin the compendium, or on the computer monitor in front of them. The parti-cipants were explicitly told to finish each task as fast as they could but with asfew errors as possible.

Before the participants were allowed to start they were instructed to use theprovided pen to circle the treatment number once they had finished a treatment,and before continuing on to the next. Remember that a treatment consisted ofa web site (W1-8) combined with a task sheet (T1-8). It was stressed that it wasimportant that the participants finished the treatments in the order given, andthat circling the treatment number should help them to achieve this. The parti-cipants then entered an individual nine digit identification number on the startpage of the experiment web site, and were redirected to a demographics form(Figure 2.1a) asking for their year of birth, sex, and computer experience. Oncethis form was completed the participants started on their first treatment. Theunique treatment order for each participant was held in a pre-filled relationaldatabase with the individual identification number as a primary key.

Within each treatment, the participants conducted two subsequent tasks:(1) completing a purchase of three products, and (2) sending a customer con-tact message to the fictional company behind the web shop (Figure 2.1c). Allinformation the participants needed was provided on the eight task sheets. Apurchase was completed by placing the three specified products in the shop-ping basket, creating a new customer account with specified contact details anddesired newsletter, and completing the order with delivery and credit card de-tails. Sending the message meant finding the web shop’s contact page, entering

12

customer details and specifying the subject, as well as writing the specifiedcustomer message. Once the contact message was sent the participants wereredirected to an evaluation form (Figure 2.1d) where they rated their experi-ence with the web site (Table 2.1). When the evaluation was submitted, theparticipants were redirected to the next web site and they continued on theirnext task sheet. When participants had finished all eight treatments, they weregiven a consent form allowing for the use of their recorded data in analysis andthey were asked to read and sign it (Appendix D). Completing the experimenttook, on average, 53 minutes (SD = 9 minutes).

2.5 Measurements

There were four dependent variables measured in the experiment divided intothree categories: user efficiency, user error and user satisfaction.

2.5.1 User Efficiency

Efficiency was measured through (1) Time-On-Task for contact form comple-tion, and (2) Click Rate for selecting a subject on the contact page. Timestampswhere registered as checkpoints throughout the web sites so that activities car-ried out between these checkpoints could be monitored. Time-On-Task for com-pleting the contact form was measured between a start point when the userentered the contact page and an endpoint at contact form submission. ClickRate was measured by the number of clicks carried out between a start pointat first interaction with the first subcategory selection element and an endpointat the last interaction with the last subcategory selection element. The use ofkeyboard keys when selecting subject (such as tab, arrow, space bar and enterkeys), as well as mouse clicks, were included in Click Rate. Clicks on the orangetriangle needed to reveal subcategories in the procedurally inconsistent web siteswere intrinsic to the experimental manipulation and were consequently not in-cluded in Click Rate, because it would have increased the click rate beyond thatneeded for the other experimental manipulations. The minimum Click Rate toselect a subject was 6 clicks for all eight web sites (W1-8).

2.5.2 User Error

The number of errors a participant made between subject selection checkpointswere recorded. User Error was measured as the sum of three types of errors:

(1) Click error — One click error was recorded for each element the parti-cipant clicked between subject selection checkpoints that was not associ-ated with choosing the subcategories.

(2) Alert error — One alert error was recorded for each time the participantreceived an alert dialogue informing the participant that one of the sub-categories was missing.

(3) Selection error — One selection error was recorded, on contact form sub-mittal, for each faulty subcategory the participant had selected.

13

2.5.3 User Satisfaction

User satisfaction was measured as SUS Score by self reported metrics throughan adaptation of Brooke’s (1996) System Usability Scale (SUS). The SUS con-sists of 10 statements to which users score their degree of agreement on a 5-pointcategory scale (Disagree-Agree). The 10 original SUS statements were modifiedto fit the current study (see Table 2.1). The wordings of the statements arealtered between positive and negative. To calculate the SUS Score, each pos-itively worded statement is given a score of its scale value minus 1, and eachnegatively worded statement is given a score of 5 minus its scale value. Thesum of these scores is then multiplied by 2.5 to give a total percentage scorebetween 0 and 100, where 100 represents complete satisfaction.

It should be pointed out that users were asked to rate the whole web shopexperience, not just their interaction with the contact form (where the manipu-lation of the independent variables lies). The rationale behind this decision wastwofold: (1) not to hint to users that the contact form was different in any waybetween the separate versions of the web shop, thereby avoiding any primingeffects, and (2) to find out whether inconsistency in a small part of a web sitecould affect the whole user experience.

Table 2.1: The 10 SUS statements used for user satisfaction measurements. Adaptedfrom Brooke (1996).

Statement Wording1 I think that I would like to use this web site frequently Positive2 I found the web site unnecessarily complex Negative3 I thought the web site was easy to use Positive4 I think I would need the support of a technical person to

be able to use this web siteNegative

5 I found that the various functions in the web site were wellintegrated

Positive

6 I thought this web site was too inconsistent Negative7 I would imagine that most people would learn to use this

web site very quicklyPositive

8 I found the web site very cumbersome to use Negative9 I felt very confident carrying out the task using this web site Positive

10 I needed to learn a lot of things before I could get goingwith this web site

Negative

14

3 Results

3.1 SampleData obtained from 16 of the 21 participants were used in the analyses. Thiscorresponds to the first two complete basic design matrices (see Section 2.3).Two participants were excluded due to technical errors at data collection, andtheir unique treatments were reassigned to two new participants in order tomaintain the design matrices to which they belonged. Data obtained fromanother three participants were left out of the analyses because they belongedto a third, incomplete basic design matrix.

3.2 Handling CovarianceScreening of the data revealed statistically significant Pearson coefficient of cor-relations between User Error and Time-On-Task, as well as between User Errorand Click Rate (Table 3.1). The Pearson coefficient of correlations between SUSScore and the other dependent variables, as well as between Time-On-Task andClick Rate were not statistically different from zero. Because of this pattern ofcorrelations, four separate multivariate analyses of covariance (MANCOVA) forrepeated measures were conducted in a step-down analysis (cf. Roy-Bargmannstep-down analysis; Tabachnick & Fidell, 2007; see also Finch, 2007). All ofthese analyses were conducted using the MANOVA syntax command in SPSS20 for Mac OS X.

Because SUS Score was largely independent of the other three dependentvariables, there was no need to subject this variable to a step-down analysis.Nevertheless, in order to minimise the noise in the data, the first MANCOVAfor repeated measures reported below used SUS Score as dependent variableand User Error, Click Rate, and Time-On-Task as covariates.

The correlations between User Error and Click Rate on the one hand, andTime-On-Task on the other, indicate that the more errors the participants made,the more clicks and the more time they needed to complete the contact form.Thus, the question is whether or not inconsistency in the web page design hadany effect on Click Rate and Time-On-Task over and above the effect of UserError. To investigate this, a step-down analysis was employed, where User Error,Click Rate, and Time-On-Task were entered in this order, corresponding to theirtheoretical importance. In all these analyses SUS Score was included as covariateto minimise the noise in the data. Thus the second MANCOVA reported belowused User Error as dependent variable and SUS Score as covariate. The thirdMANCOVA used Click Rate as dependent variable and User Error and SUSScore as covariates. The forth MANCOVA used Time-On-Task as dependentvariable and Click Rate, User Error and SUS Score as covariates.

Because there were correlations between the dependent variables, and thatfour separate MANCOVA were conducted, it was motivated to apply a Bonfer-roni correction to minimise the risks of committing Type I errors (Tabachnick& Fidell, 2007; see also Perneger, 1998). This resulted in the need for analysisof statistical significance at the 0.01 level (Equation 3.1).

α = 1− (1− 0.01)4 ≈ 0.039 (3.1)

15

Table 3.1: Pearson’s coefficient of correlations between dependent variables: SUSScore, User Error, Click Rate, and Time-On-Task.

SUS Score User Error Click RateUser Error -.029Click Rate .079 .187∗Time-On-Task .016 .227∗ .026

∗p < 0.05 (two-tailed).

Table 3.2: F-statistics of 2 (Perceptual inconsistency) × 2 (Semantic inconsistency)× 2 (Procedural inconsistency) repeated measures MANCOVA for SUS Score, UserError, Click Rate, and Time-On-Task.

Sus Scorea User Errorb Click Ratec Time-On-Taskd

Factor F1,12 p F1,14 p F1,13 p F1,12 pPERceptual 4.51 0.055 0.11 0.744 0.88 0.364 10.28∗ 0.008SEMantic 0.57 0.466 0.22 0.648 7.70 0.016 0.16 0.694PROcedural 3.61 0.082 31.70∗ <0.001 2.20 0.162 1.27 0.282PER*SEM 0.01 0.931 2.50 0.136 1.53 0.238 0.50 0.493PER*PRO 1.40 0.260 0.21 0.651 6.59 0.023 0.04 0.843SEM*PRO 5.28 0.040 0.03 0.875 0.05 0.829 0.96 0.347PER*SEM*PRO 0.84 0.377 4.90 0.044 1.35 0.265 0.68 0.424

∗p < 0.01.aSUS Score with User Error, Click Rate and Time-On-Task as covariates.bUser Error with SUS Score as covariate.cClick Rate with User Error and SUS Score as covariates.dTime-On-Task with Click Rate, User Error and SUS Score as covariates.

3.3 User SatisfactionFigure 3.1 presents a factor plot for SUS Score, using the observed mean values,divided on the three independent variables: Perceptual, Semantic and Proced-ural inconsistency.

The plot indicates that SUS Score decreases when any of the three incon-sistencies are introduced alone. However, there is a tendency that Semanticinconsistency inhibits Procedural inconsistency, which creates an interaction ef-fect.

Table 3.2 presents F-statistics for the four individual MANCOVA analyses.The results for SUS Score confirm most of the tendencies indicated in the factorplot in Figure 3.1, particularly the interaction between Semantic and Proceduralinconsistency. However, none of the effects are statistically significant at the 0.01level.

Absent Present

75

80

85

90

Absent Present Absent PresentSemantic inconsistency

Mea

n va

lues

of S

US

Scor

e

Procedural inconsistencyAbsentPresent

Perceptual inconsistency

Figure 3.1: Observed mean values of SUS Score divided on the three factors Percep-tual, Semantic, and Procedural inconsistency. Each factor has two levels: absence orpresence. Error bars represents standard errors of the mean values (± 1 SE).

17

3.4 User ErrorFigure 3.2 presents a factor plot for User Error, using the observed meanfrequencies, divided on the three independent variables: Perceptual, Semanticand Procedural inconsistency.

The plot indicates that User Error increases when Procedural inconsistencyis introduced. The MANCOVA confirms this main effect, which is statisticallysignificant at the 0.01 level (Table 3.2).

Perceptual and Semantic inconsistency shows little to no effect on User Er-ror when introduced alone. However, there is a tendency towards a complexthree-way interaction effect between the three inconsistencies. Both Perceptualand Semantic inconsistency tends to inhibit the negative effect of Proceduralinconsistency on User Error. On the other hand, Perceptual and Semantic incon-sistency also tends to cancel each other out. Thus, when all three inconsistenciesare introduced together, Procedural inconsistency is as influential as when in-troduced alone. However, the three-way-interaction effect is not statisticallysignificant at the 0.01 level (Table 3.2).

Absent Present

0.0

0.5

1.0

1.5

2.0

2.5

3.0


Mea

n fre

quen

cies

of U

ser E

rror



Figure 3.2: Observed mean frequencies of User Error divided on the three factors Per-ceptual, Semantic, and Procedural inconsistency. Each factor has two levels: absenceor presence. Error bars represents standard errors of the mean frequencies (± 1 SE).

18

3.5 User Efficiency3.5.1 Click Rate

Figure 3.3 presents a factor plot for Click Rate, using the observed mean fre-quencies, divided on the three independent variables: Perceptual, Semantic andProcedural inconsistency.

The plot indicates that Click Rate increases when Semantic inconsistencyis introduced. This tendency, although not statistically significant at the 0.01level, is apparent in Table 3.2.

Neither Perceptual nor Procedural inconsistency has any effect on Click Ratealone but both separately tends to inhibit the negative effect of Semantic in-consistency on Click Rate. When combined they boost the effect of Semanticinconsistency, resulting in a peak value of Click Rate when all inconsistencies arepresent. However, both these effects disappear once User Error and SUS Scoreare controlled for as shown in Table 3.2. A tendency towards a two-way in-teraction effect between Perceptual and Procedural inconsistency also becomesevident once these factors are controlled for, which is not as clear in the observedmeans in the plot.

Absent Present

7

8

9

10


Mea

n fre

quen

cies

of C

lick

Rat

e



Figure 3.3: Observed mean frequencies of Click Rate divided on the three factors Per-ceptual, Semantic, and Procedural inconsistency. Each factor has two levels: absenceor presence. Error bars represents standard errors of the mean frequencies (± 1 SE).

19

3.5.2 Time-On-Task

Figure 3.4 presents a factor plot for Time-On-Task, using the observed meandurations (s), divided on the three independent variables: Perceptual, Semanticand Procedural inconsistency.

The plot indicates that each of the three inconsistencies alone has an effect,resulting in an increase in Time-On-Task.

Semantic inconsistency tends to inhibit the effect of Procedural inconsist-ency, whereas Perceptual inconsistency only has a small effect on the other twoinconsistencies. However, when all three inconsistencies are combined, the effectis cumulative and results in the longest completion time. Table 3.2 shows thatafter controlling for the effect of Click Rate, User Error and SUS Score onlythe main effect of Perceptual inconsistency reaches statistical significance at the0.01 level.

Absent Present

90

100

110

120

130


Mea

n du

ratio

ns o

f Tim

e−O

n−Ta

sk (s

)



Figure 3.4: Observed mean durations of Time-On-Task divided on the three factorsPerceptual, Semantic, and Procedural inconsistency. Each factor has two levels: ab-sence or presence. Error bars represents standard errors of the mean durations (± 1SE).

20

4 Discussion

The present experiment resulted in statistically significant main effects (p <0.01) of both Perceptual and Procedural inconsistencies on user performance(i.e., User Error, Click Rate, or Time-On-Task). Procedural inconsistency res-ulted in more errors being made, whereas Perceptual inconsistency prolongedtask completion time independently of the number of errors made (Table 3.2).Semantic inconsistency, on the other hand, tended to increase the number ofclicks necessary to select a subject on the contact page (p = 0.016).

The inconsistencies in the contact form had only weak effects on the users’satisfaction (i.e., SUS Score) of the web shop as a whole. The effects were notstatistically significant (Table 3.2), and overall the users gave the web shop ahigh SUS Score (see Figure 3.1; according to Tullis & Albert, 2008, 80 is afairly good score).

To our knowledge, the present study is the first in examining full factorialinteraction effects of inconsistencies from a user’s perspective. A few interest-ing, and unexpected, interaction effects were indicated, even though none werestatistically significant at the 0.01 level. Based on previous research, it was spec-ulated that any interaction effects should be cumulative. However, the presentresults indicate that Semantic, as well as, Perceptual inconsistency inhibitedthe effects caused by Procedural consistency. Semantic consistency tended toprotect against the negative influence Procedural consistency had on user sat-isfaction, whilst both Semantic and Perceptual consistency tended to suppressthe number of errors made in Procedural inconsistent versions of the web shop.

That Perceptual inconsistency had negative impact on user performancecoincide with the findings of Adamson (1996), Ozok and Salvendy (2000), andAlTaboli and Abou-Zeid (2007) on physical inconsistency. However, they foundevidence that it increased errors made by users, whereas the present study foundthat it prolonged task completion time.

The results that Procedural inconsistency increased errors made by usersagrees with the findings of Adamson (1996) with regard to communicationalinconsistency.

Kellogg’s (1987) manipulation of conceptual consistency was close to thedefinition of Semantic consistency of the present thesis. Although the presentstudy lacked statistically significant results for the effects of Semantic consist-ency, it did show tendencies to correspond with the findings of Kellogg (1987)that it may affect user performance.

4.1 Implications

The present experiment shows that graphical designers should avoid designinginconsistent procedures within web sites from a user’s point of view. Inconsist-ency in procedure may lead to users making more errors. To achieve proced-urally consistent web design, developers must analyse the procedures from theuser’s perspective.

The present experiment also indicates that Semantic inconsistency may pro-tect against Procedural inconsistency’s negative impact on user satisfaction.This could be that once one changes the appearance or placement of an object,it indicates to the user that a different procedure is necessary. This potential in-teraction effect indicates that users may regard an object, which has more than

21

one inconsistency in relation to another object, as a completely new object, andtherefore judge it differently.

An interesting result from the present experiment is that Perceptual incon-sistency affects Time-On-Task independently of the number of errors, and thenumber of clicks a user made. This suggests that the prolonged completiontime was due to cognitive load of the inconsistency. This implies that change invisual appearance makes users think more about their actions.

Taken together, the results show that inconsistencies might be used to sig-nal change. However, designers should be sure to signal this with perceivablechanges in order to minimise the risks of errors being made.

4.2 LimitationsApart from the two statistically significant main effects of Perceptual and Pro-cedural inconsistencies on user performance, some additional but weak effectswere found. That only two effects were statistically significant at the 0.01 levelmay be due to at least three factors: (1) the experimental manipulation wasfaulty, (2) the experimental manipulation was to weak, or (3) the sample sizewas too small.

4.2.1 Faulty Manipulation

The definitions of the three categories of consistency proposed in Section 1.2.3,were strictly adhered to in designing the eight versions of the web shop (W1-8). Therefore there should be no question that the manipulation of Perceptual,Semantic, and Procedural inconsistency was true to the proposed definitions.

4.2.2 Weak Experimental Manipulation

As explained in Section 2.5.3, users evaluated their experience with the wholeweb shop. This is possibly the reason why there were no strong results onthe effects of user satisfaction. More powerful inconsistencies might have to beintroduced in order to affect user satisfaction when only a part of a web siteis manipulated. Naturally, the evaluation could have been on just the contactform to achieve stronger results. However, as elaborated earlier, this could havehad a priming effect on the participants in the experiment.

No significant results were found for Semantic consistency. It can be arguedthat the manipulation of Semantic consistency was too simple to affect on theuser experience. The question arises whether the interchange of visual appear-ance (see Figure B.3) really made a semantic difference to the user. However,the effects on Click Rate are close to being statistically significant, with regardto Semantic inconsistency, and the notion that it affected user performance cantherefore not be rejected.

4.2.3 Too Small Sample

The most probable cause for the few statistically significant effects is that toofew participants were used. A couple of the values presented in Table 3.2 are

22

approaching statistical significance (i.e., Semantic inconsistency’s effect on ClickRate, and the two-way-interaction effect between Perceptual and Proceduralconsistency on Click Rate). This indicates that using a larger sample may yieldstronger results.

4.3 Further ResearchComputer science, and thereby user interface design, will likely keep on expand-ing into unknown territories. Studies on the impact of consistency can thereforehelp us predict, and counteract, future usability issues.

Previous research and the present experiment makes it evident that incon-sistency does affect user performance. It also shows that inconsistency can havean effect on user satisfaction, although, the present study did not find statistic-ally significant evidence for this.

An in-depth analysis of the manipulations carried out in all studies on con-sistency within HCI could help to home in on the essence of consistency. Thiscould help to understand how these manipulations of consistency lead to suchdifferent results. One probable reason might be that there is another level toconsistency than just categories. For example, it would be interesting to com-bine an experiment such as ours along with Norman’s (2001) seven stages ofaction (Figure 4.1), which suggests that a user goes through seven stages ofaction when interacting with, for instance, a web site. This would be a use-ful approach in order to see in what stage of interaction any of the proposedconsistencies would affect user performance and satisfaction.

It would also be useful to carry out separate experiments on each of thethree proposed consistencies and thereby test several different manipulations ofeach consistency.

Figure 4.1: Norman’s seven stages of action. Redrawn from Norman (2001).

23

4.4 Conclusions1. Inconsistent web design results in more user errors being made and in

longer task completion times.

2. Viewing the system from a user’s perspective aids developers and designersto build consistent web applications.

3. The results of the present thesis indicate that there are interaction ef-fects between different types of inconsistency, which needs to be furtherexplored.

24

AcknowledgementsSpecial thanks go to Dr. Östen Axelsson, Department of Psychology, StockholmUniversity, for advices on research methods, statistics and scientific writing, aswell as valuable comments on the text.

Special thanks also go to Mette Clausen-Bruun for proof reading and valu-able comments on the text.

Special thanks is also directed towards Dr. F. Layne Wallace, Department ofComputer and Information Sciences, University of Northern Florida, for findingand forwarding a copy of Adamson (1996).

Special thanks also go to the 21 persons who volunteered to participate inthe present experiment.

25

References

37signals. (2006). Getting real. Chicago, IL: 37signals.Adamson, P. J. (1996). A comparison between consistent and inconsistent graph-

ical user interfaces. Unpublished master’s thesis, University of NorthernFlorida.

AlTaboli, A. & Abou-Zeid, M. R. (2007). Effect of physical consistency of webinterface design on users’ performance and satisfaction. In J. A. Jacko(Ed.), HCI’07 proceedings of the 12th international conference on human-computer interaction: applications and services (pp. 849–858). Berlin:Springer-Verlag.

Barnard, P. J., Hammond, N. V., Morton, J., Long, J. B. & Clark, I. A. (1981).Consistency and compatibility in human-computer dialogue. Applied Psy-chology , 15 , 87–134.

Barnett, S. M. & Ceci, S. J. (2002). When and where do we apply what welearn? a taxonomy for far transfer. Psychological Bulletin(128), 612–637.

Blake, T. (1986). Introduction to the art and science of user interface design.Chatsworth, CA: Intuitive Software and Interactive Systems.

Brooke, J. (1996). SUS: A quick and dirty usability scale. In P. W. Jordan,B. Thomas, B. A. Weerdmeester & I. L. McClelland (Eds.), Usabilityevaluation in industry (pp. 189–194). London, UK: Taylor Francis.

Constantine, L. L. & Lockwood, L. A. D. (1999). Software for use: A practicalguide to the models and methods of usage-centered design. Reading, MA:ACM Press.

Encyclopedia Britannica. (2012). consistent. Retrieved July, 30, 2012, from:http://www.britannica.com/bps/dictionary?query=consistent.

Finch, W. H. (2007). Performance of the Roy-Bargmann stepdown procedureas a follow up to a significant MANOVA. Multiple Linear RegressionViewpoints, 33 (1), 12–22.

Foley, J. & van Dam, A. (1982). Fundamentals of interactive computer graphics.Reading, MA: Addison-Wesley.

Grudin, J. (1989). The case against user interface consistency. Communicationsof the ACM , 32 (10), 1164–1173.

Hurst, M. (2004). The page paradigm. Retrieved March, 20, 2012, from:http://goodexperience.com/2004/02/the-page-paradigm.php.

Kanizsa, G. (1979). Organization in vision: essays on gestalt perception. NewYork, NY: Barnes Noble.

Kellogg, W. A. (1987). Conceptual consistency in the user interface: effectson human performance. In H.-J. Bullinger & B. Shackel (Eds.), Human-computer interaction -INTERACT’87 (pp. 389–394). Amsterdam, Hol-land: North-Holland.

Kellogg, W. A. (1989). The dimensions of consistency. In J. Nielsen (Ed.),Coordinating user interfaces for consistency (pp. 9–20). San Fransico,CA: Morgan Kauffman Publishers.

Klein, G. (1999). Sources of power. Cambridge, MA: The MIT Press.Mayhew, D. J. (1992). Principles and guidelines in software user interface

design. Eaglewood Cliffs, NJ: Prentice Hall.Mendel, J. & Pak, R. (2009). The effect of interface consistency and cognitive

load on user performance in an information search task. Proceedings of theHuman Factors and Ergonomics Society Annual Meeting , 53 (22), 1684–

26

1688.Moran, T. P. (1981). The command language grammar: a representation for

the user interface of interactive computer systems. International Journalof Man-Machine Studies, 15 , 2–50.

Nielsen, J. (1989). Coordinating user interfaces for consistency. ACM SIGCHIBulletin, 20 (3), 15–16.

Norman, D. A. (2001). The design of everyday things. Cambridge, MA: MITPress.

Ozok, A. A. & Salvendy, G. (2000). Measuring consistency of web page designand its effects on performance and satisfaction. Ergonomics, 43 (4), 443–460.

Payne, S. J. & Green, T. R. G. (1986). Task-action grammars: A model of themental representation of task languages. Human-computer interaction, 2 ,99–133.

Perneger, T. V. (1998). What’s wrong with bonferroni adjustments. BritishMedical Journal , 316 , 1236–1238.

Reisner, P. (1981). Formal grammar and human factors design of an interactivegraphics system. IEEE Transactions on Software Engineering , SE-7 (2),229–240.

Satzinger, J. W. (1998). The effects of conceptual consistency on the end user’smental models of multiple applications. Journal of Organizational andEnd User Computing , 10 (3), 3–14.

Shermer, M. (2011). The believing brain. New York, NY: Times Books.Shneiderman, B. (1992). Designing the user interface: Strategies for effective

human-computer interaction (2nd ed.). Reading, MA: Addison-WesleyPublishing Company, Inc.

Smith, D. C., Irby, C., Kimball, R., Verplank, W. L. & Harslem, E. (1982).Designing the star user interface. Byte, 4 , 242–282.

Somberg, B. L. (2000). Procedural consistency in the user interface. Proceedingsof the Human Factors and Ergonomics Society Annual Meeting , 44 (2),307–310.

Tabachnick, B. G. & Fidell, L. S. (2007). Using multivariate statistics (5th ed.).Boston, MA: Pearson Education Inc.

Tanaka, T., Eberts, R. E. & Salvendy, G. (1991). Consistency of human-computer interface design: Quantification and validation. Human Factors:The Journal , 33 (6), 653–676.

The Free Dictionary. (2012). consistent. Retrieved March, 19, 2012, from:http://www.thefreedictionary.com/consistent.

Tullis, T. & Albert, B. (2008). Measuring the user experience. Burlington, MA:Morgan Kauffman Publishers.

Wiecha, C., Bennett, W., Boies, S. & Gould, J. (1990). ITS: a tool for rapidlydeveloping interactive applications. ACM Transactions on InformationSystems, 8 (3), 204–236.

Wolf, R. (1989). Consistency as process. In J. Nielsen (Ed.), Coordinatinguser interfaces for consistency (pp. 89–92). San Fransico, CA: MorganKauffman Publishers.

27

Appendices

A Baseline Choice Selections

Figure A.1: Newsletter choice at customer registration.

Figure A.2: Card expiry date choice at checkout.

B Variations of Inconsistency in Subject System

Figure B.1: Choice selection fully consistent with the baseline newsletter and expirydate selections (W1).

Figure B.2: Perceptually inconsistent choice selection (W2).

Figure B.3: Semantically inconsistent choice selection (W3).

Figure B.4: Procedurally inconsistent choice selection (W4).

Figure B.5: Perceptually and Semantically inconsistent choice selection (W5).

Figure B.6: Semantically and Procedurally inconsistent choice selection (W6).

Figure B.7: Perceptually and Procedurally inconsistent choice selection (W7).

Figure B.8: Perceptually, Semantically, and Procedurally inconsistent choice selection(W8).

C Task Compendium

Teknik och SamhalleDatavetenskap

Experiment i anvandbarhet

CBG: 0

ID: 123456782

Anton [email protected]

IntroduktionDetta ar ett experiment om anvandbarhet pa webben. Syftet ar att testaen webbapplikations funktionalitet i atta olika utformningar. Din uppgiftar att hjalpa atta personer att inhandla produkter fran foretaget Web-shoppen, samt hjalpa dem kontakta Webshoppen via webbsidans kontak-tformular.

Detta kompendium bestar av ett forsattsblad, denna introduktion, samtatta uppgiftssidor med den information, samt de instruktioner du behoverfor att genomfora experimentet.

Varje uppgift (1-8) bestar av foljande delmoment och skall utforas i an-given ordning:

1. Hitta och lagga de 3 angivna produkterna i varukorgen (observeraatt sokfunktionen for produkter ar avstangd).

2. Registrera konto via kassan med• angivna kontaktuppgifter.• angivet onskat nyhetsbrev.• angivet onskat losenord.

3. Ange kortuppgifterna och slutfor kopet.4. Skicka meddelande till Webshoppen med angiven forfragan via webb-

platsens kontaktformular.Nar meddelandet har skickats kommer du att omdirigeras till en sida dardu ombeds utvardera din upplevelse av den senast anvanda webbapplika-tionen. Nar du sparat dina svar kommer du att omdirigeras till Webshop-pen igen och du paborjar da uppgiften pa nasta blad.

Nar du genomfort samtliga atta uppgifter ber vi dig kalla pa experimen-tledaren som kommer att ge dig ett medgivandeformular for din under-skrift.

Speciellt viktigt att komma ihag under experimentet ar:• Den tid du tar pa dig samt de fel du gor kommer att matas. Darfor

ar det viktigt att du haller dig till uppgiften och forsoker utfora deatta uppgifterna sa fort du kan men med sa fa fel som mojligt.

• Inga matningar gors under utvarderingarna mellan uppgifterna, duhar da mojlighet att ta en paus om du sa onskar.

• Vissa uppgifter kan upplevas vara svara, men de gar att losa sa tappainte modet.

• Detta ar INTE ett test av dig och dina kunskaper eller faridgheter,utan ett test av hur val de olika utformningarna av webbapplikationfungerar.

Om tekniska problem uppstar eller om du undrar over nagot sa kontaktaexperimentledaren. Nar du kanner dig redo kan du paborja experimentetgenom att ange ditt niosiffriga ID. Ditt ID finner du pa forsattsbladet. Ex-perimentet borjar med ett mindre frageformular dar du anger fodelsear,kon och datorvana.

Uppgift 1KontaktuppgifterAlfred GoranssonTallgatan 11 C523 35 ULRICEHAMN

[email protected] - 159 595 55Onskat losenord: letmein

KortuppgifterKorttyp: MasterCardKortnummer: 5454 9963 1215 9875Giltigt tom: 09 / 2012Sakerhetskod: 951

InstruktionerGenomfor kopAlfred ber dig kopa foljande produkter:

• Boken ’Version Control with Git: Powerful Techniques for Certralized andDistributed Project Management’ (Data & IT)

• Filmen ’The Big Lebowski - Special Edition’ (Komedi)

• CD-skivan ’Free The Bees’ (Alternativt & Indie)

NyhetsbrevAlfred vill ha nyheter gallande alla produkter.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Alfred att du kontaktar Webshoppen for attberomma dem for snabb leverans, men inte for nagon sarskild produkt. Ga tillkontaktsidan och fyll i formularet. Ange foljande meddelande:

Jag fattar inte hur ni lyckas! Jag bestallde mina produkter igar och idag ar deredan har. Maste saga att ni gor ett fantastiskt jobb. Keep up the good work!

Uppgift 2KontaktuppgifterLisa StahlskioldExpressvagen 55168 53 BROMMA

lisa [email protected] - 961 963 91Onskat losenord: tellno1

KortuppgifterKorttyp: VISAKortnummer: 4646 8366 9991 0105Giltigt tom: 01 / 2013Sakerhetskod: 883

InstruktionerGenomfor kopLisa ber dig kopa foljande produkter:

• Filmen ’Eyes Wide Shut’ (Thriller & Skrack)

• CD-skivan ’We Sweat Blood’ (Rock)

• CD-skivan ’Cosmo’s Factory’ (Rock)

NyhetsbrevLisa vill ha samtliga nyhetsbrev gallande musik.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Lisa att du kontaktar Webshoppen for attkontrollera status pa en musikorder hon lagt tidigare i ar. Ga till kontaktsidanoch fyll i formularet. Ange foljande meddelande:

For drygt en manad sedan lade jag en order (nummer 9152138) hos er pa fler-talet CD-skivor, vissa skivor var da slutsalda. Nar kan jag forvantas fa dennaorder?

Uppgift 3KontaktuppgifterEva StrandsjoStorgatan 114214 22 MALMO

[email protected] - 193 124 25Onskat losenord: password12


InstruktionerGenomfor kopEva ber dig kopa foljande produkter:

• Boken ’Truth’ (Filosofi)

• Filmen ’Inception’ (Action & Aventyr)

• Filmen ’Scarface (Blu-ray)’ (Action & Aventyr)

NyhetsbrevEva vill ha erbjudanden gallande filmer.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Eva att du kontaktar Webshoppen for attbegara en retur av tidigare bestallda filmer som hon angrar att hon kopt. Gatill kontaktsidan och fyll i formularet. Ange foljande meddelande:

Hej, idag fick jag en leverans av filmer fran ett kop som jag inte trodde hadegatt igenom, darfor har jag redan inhandlat dem pa annat satt. Hur gor jag nu?

Uppgift 4KontaktuppgifterAbir SahlirBjorkgatan 13 D571 95 NASSJO

[email protected] - 987 537 12Onskat losenord: 789secret


InstruktionerGenomfor kopAbir ber dig kopa foljande produkter:

• Boken ’Design Patterns’ (Data & IT)

• Boken ’Information Retrieval’ (Data & IT)

• Filmen ’Taxi’ (Action & Aventyr)

NyhetsbrevAbir vill ha erbjudanden gallande bocker.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Abir att du kontaktar Webshoppen for attklaga pa produktavdelningen for deras begransade utbud pa bocker. Ga tillkontaktsidan och fyll i formularet. Ange foljande meddelande:

Jag har flertalet ganger kontaktat er i detta arende och jag far aldrig nagotordentligt svar pa varfor ni aldrig kan ta in nagra vettiga bocker om HTML5!

Uppgift 5KontaktuppgifterSuzanna KowalskiPressvagen 1977 53 LULEA

daemon [email protected] - 156 353 31Onskat losenord: daemons


InstruktionerGenomfor kopSuzanna ber dig kopa foljande produkter:

• Boken ’Gruppsykologi : om grupper, organisationer och ledarskap’ (Psykolo-gi & Kognition)

• Filmen ’The Shining’ (Thriller & Skrack)

• CD-skivan ’Fear Of Fours’ (Electronica)

NyhetsbrevSuzanna vill ha nyheter gallande alla produkter.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Suzanna att du kontaktar Webshoppen for attandra den nyss lagda ordern och byta ut en musikartikel. Ga till kontaktsidanoch fyll i formularet. Ange foljande meddelande:

Alldeles nyss lade jag en bestallning med ordernummer 9153616. Jag undrar omjag kan byta ut skivan “Fear of fours“ med Lamb till deras sjalvbetitlade istallet?

Uppgift 6KontaktuppgifterAnders BengtsonTunavagen 33424 17 ANGERED

[email protected] - 965 461 24Onskat losenord: qwerty


InstruktionerGenomfor kopAnders ber dig kopa foljande produkter:

• Filmen ’Faglarna (1963)’ (Thriller & Skrack)

• CD-skivan ’Pendulum’ (Rock)

• CD-skivan ’Blue Train’ (Jazz & Eklektiskt)

NyhetsbrevAnders vill ha nyheter gallande filmer.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Anders att du kontaktar Webshoppen foratt ta reda pa om det gar att andra betalmetod for den nyss lagda ordern in-nehallande flera typer av produkter. Ga till kontaktsidan och fyll i formularet.Ange foljande meddelande:

Precis nar ordern lades sag jag att det inte fanns tillrackligt med pengar pa mittkonto. Finns det mojlighet att andra sa att ordern gar pa faktura istallet?

Uppgift 7KontaktuppgifterLovisa GranstedtRisgrynsstigen 21654 55 KARLSTAD

[email protected] - 751 752 23Onskat losenord: passw0rd


InstruktionerGenomfor kopLovisa ber dig kopa foljande produkter:

• Filmen ’Nyckeln till frihet (Blu-ray)’ (Drama)

• Filmen ’Limitless’ (Action & Aventyr)

• CD-skivan ’I Am Not A Doctor’ (Electronica)

NyhetsbrevLovisa vill ha samtliga nyhetsbrev gallande alla produkter.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Lovisa att du kontaktar Webshoppen for attgora en allman forfragan om betalning, men inte for nagon sarskild produkt. Gatill kontaktsidan och fyll i formularet. Ange foljande meddelande:

Sag att ni bara erbjuder faktura och kortbetalningar som alternativ. Skulle niinte kunna erbjuda Paypal som ett alternativ? Det skulle underlatta en hel del.

Uppgift 8KontaktuppgifterClaes-Goran AhlBergshyttan290 62 VILSHULT

[email protected] - 553 785 12Onskat losenord: 1q2w3e


InstruktionerGenomfor kopClaes-Goran ber dig kopa foljande produkter:

• Filmen ’Singin’ in the Rain’ (Komedi)

• CD-skivan ’A Musical Ramance’ (Jazz & Eklektiskt)

• CD-skivan ’Lockar Och Skagg’ (Jazz & Eklektiskt)

NyhetsbrevClaes-Goran vill ha erbjudanden gallande musik.

Skicka kontaktmeddelandeEfter att du genomfort kopet vill Claes-Goran att du kontaktar Webshoppenfor att klaga pa kundtjanst, men inte om nagon sarskild produkt. Ga till kon-taktsidan och fyll i formularet. Ange foljande meddelande:

Varfor svarar ni aldrig? Jag har kontaktat er flera ganger via telefon men mankommer aldrig fram istallet hamnar man bara i nagot slags limbo. Skarpning!!!

D Consent Form

MedgivandeformulärExperimentledare: Anton Axelsson

[email protected]

Experiment: Inkonsistens påverkan på användbarhetBeskrivningDetta experiment går ut på att utröna hur användare påverkas av inkonsistens i webdesign.Experimentet består av åtta versioner av en webbapplikation. Till varje applikationsversion skalltre uppgifter utföras.Köp av varor

Du genomför ett köp av ett antal specificerade varor och registrerar kunduppgifter.

Utformande av meddelandeDu fyller i ett kontaktformulär och skickar ett meddelande till det fingerade företaget bakomwebbapplikation.

UtvärderingDu utvärderar uppgiften samt applikationsversionens utförande.

MedgivandeJag medger härmed att jag har . . .

. . . blivit informerad om experimentets syfte och tillåter att den insamlade informationen be-varas till dess att den publiceras.

. . . förstått att den insamlade informationen rapporteras och bevaras i anonym form och attexperimentledaren enligt svensk lag, under ovanliga omständigheter, kan tvingas lämna utinformationen till andra forskare för granskning av eventuella brister.

. . . förstått att min uttryckliga värdering av applikationsversionerna, uppgifterna samt ålder,kön och datorvana sparas.

. . . förstått innehållet i detta dokument

Ort och datum Namnförtydligande Underskrift

Kompletterande klausulJag ger mitt medgivande till att den insamlade informationen publiceras online vilket ger andraforskare möjligheten att använda sig av informationen i det syfte de finner lämpligt.

Ort och datum Namnförtydligande Underskrift

consistency in web design from a user perspectivebachelor's thesis 15 credits consistency in...

Documents