INOM EXAMENSARBETE MEDIETEKNIK,AVANCERAD NIVÅ, 30 HP

, STOCKHOLM SVERIGE 2019

Intrusiveness VS Awareness: Laying The Groundwork For Presenting Offers To Customers With AR In A Retail Environment

ANTON MARTINSSON

KTHSKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP

Abstract The term Augmented Reality (AR) was first coined back in 1968. Research on the subject would then for decades remain largely focused on technical aspects of the phenomenon. At the time, little to no attention was paid to the potential user audience or what would later be known as Human-Computer Interaction theory. Some previous studies have touched upon user satisfaction of general AR interfaces, but most studies that cover the topic of indoor navigation with AR tend to focus on technical solutions. Few try to establish any kind of visual language or research what visual interfaces are most intuitive, effective and user friendly.

Consequently, this thesis investigates how to visually seek the attention of the user to present offers in an AR application for smartphones meant to be used to navigate an indoor retail environment. It does so by conducting a user study in a real retail store in Stockholm, Sweden, where participants completed three laps around a certain part of the store using an AR indoor navigation application. For every lap, each participant tried out one of three different versions of the application. These three versions varied in how intrusive the presentation of offers was to the customer’s experience with the application. The participants filled in a Likert scale questionnaire for each of the three versions, as well as answered some more open-ended questions at the end of every test session.

The conclusion is that a balanced approach to intrusiveness is the wisest in order to make customers aware of discounts around them while not considerably annoying them. The most positively received approach presented an offer promptly to the user, but did not take up too much screen space or force the user to take any action towards it. Future studies could investigate whether there is a higher tolerance for visual intrusion among customers if the discount is considered big or very personally relevant. Subsequent studies could also use high-end AR head-mounted displays that might be more prominently used by everyday consumers in the future.

Sammanfattning Termen Augmented Reality (AR) myntades år 1968. Vetenskap och forskning kring ämnet kom sedan i decennier att fokuseras på fenomenets tekniska aspekter. Till en början lades nästintill ingen uppmärksamhet på den potentiella användargruppen eller på det som senare kom att kallas Människa-Dator Interaktions-teori. Under senare år har en ökad mängd studier berört just denna aspekt, där användarens tillfredsställelse med allmänna AR-gränssnitt satts i fokus. För studier fokuserade på inomhusnavigering med hjälp av AR har dock inte utvecklingen varit fullständigt i linje med ovanstående, då det är få som i dag undersökt vilken typ av visuella gränssnitt som är mest intuitiva, effektiva och användarvänliga. Studierna har istället fokuseras på den tekniska aspekten av ämnet.

Denna avhandling undersöker därav hur användarens uppmärksamhet bör sökas för att visuellt presentera erbjudanden i en AR-applikation anpassad för smarta telefoner vid navigation i detaljhandelsmiljö inomhus. För att undersöka detta genomfördes en användarstudie i en matbutik i Stockholm. Deltagare i studien fick vid tre direkt efterföljande tillfällen gå en bestämd rutt genom matbutiken där tre olika visuella presentationer av erbjudanden visades i en AR-baserad navigationsapp. De olika versionerna av presentationer var mer eller mindre visuellt påträngande på deltagarnas upplevelse. Resultatet av deras upplevelse kom senare att utvärderas med hjälp av Likert-skalor som deltagarna fyllde i efter vardera version som testades, samt öppna frågor i slutet av varje användartest.

Slutsatsen är att ett balanserat angreppssätt är det mest effektiva för att göra den potentiella kunden medveten om rabatter och andra erbjudanden inuti butiksmiljön. Mest positiv inställning hade deltagarna då erbjudanden visades på ett enkelt och avskalat vis, där presentationen varken upptog mycket skärutrymme eller krävde att användaren vidtog några åtgärder. Vidare skulle framtida studier kunna undersöka huruvida det finns en högre tolerans gentemot erbjudanden som presenteras på detta eller andra sätt om de anses ha en hög personlig relevans. Efterföljande studier kommer även kunna använda ny AR-teknik och se hur den kommer kunna användas mot framtida konsumenter.

Intrusiveness VS Awareness: Laying The Groundwork ForPresenting Offers To Customers With AR In A Retail

EnvironmentAnton Martinsson

antmar@kth.seKTH Royal Institute of Technology

Stockholm, Sweden

ABSTRACTThe term Augmented Reality (AR) was first coined back in 1968.Research on the subject would then for decades remain largely fo-cused on technical aspects of the phenomenon. At the time, little tono attention was paid to the potential user audience or what wouldlater be known as Human-Computer Interaction theory. Some pre-vious studies have touched upon user satisfaction of general ARinterfaces, but most studies that cover the topic of indoor navigationwith AR tend to focus on technical solutions. Few try to establishany kind of visual language or research what visual interfaces aremost intuitive, effective and user friendly.

Consequently, this thesis investigates how to visually seek theattention of the user to present offers in an AR application forsmartphones meant to be used to navigate an indoor retail envi-ronment. It does so by conducting a user study in a real retail storein Stockholm, Sweden, where participants completed three lapsaround a certain part of the store using an AR indoor navigationapplication. For every lap, each participant tried out one of threedifferent versions of the application. These three versions varied inhow intrusive the presentation of offers was to the customer's expe-rience with the application. The participants filled in a Likert scalequestionnaire for each of the three versions, as well as answeredsome more open-ended questions at the end of every test session.

The conclusion is that a balanced approach to intrusiveness isthe wisest in order to make customers aware of discounts aroundthem while not considerably annoying them. The most positivelyreceived approach presented an offer promptly to the user, but didnot take up too much screen space or force the user to take anyaction towards it. Future studies could investigate whether thereis a higher tolerance for visual intrusion among customers if thediscount is considered big or very personally relevant. Subsequentstudies could also use high-end AR head-mounted displays thatmight be more prominently used by everyday consumers in thefuture.

KEYWORDSaugmented reality, AR, retail, offers, user interface, usability, intru-siveness

ACM Reference Format:Anton Martinsson. 2019. Intrusiveness VS Awareness: Laying The Ground-work For Presenting Offers To Customers With AR In A Retail Environment.In Proceedings of ACM Conference (Conference’17). ACM, New York, NY,USA, 11 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 INTRODUCTIONThe most common definition of Augmented Reality describes it asa blend of real and computer generated information and imagery,all brought together onto the user's view of the real world [16] [2].This makes Augmented Reality (commonly referred to as AR) aprime candidate medium for visual assistance and guidance, as itcan superimpose digital directional information on the real world,where no such information exists naturally.

In the last decade, multiple AR applications for indoor naviga-tion have been realized in research scenarios [23][13][22][25]. Sincethe satellite based Global Positioning System (GPS) can not esti-mate position accurately enough inside buildings, many differenttechniques have been studied in hope of developing accurate in-door navigation for places like shopping malls [1] or museums [31].While certain success has been reached by using bluetooth-enabledbeacons [31], Wifi-triangulation [19] and fiducial markers [13][23],recent advances in camera and sensor technology housed inmodernsmartphones now enables true, markerless AR experiences. Thistype of geometric AR lends itself well to navigational applications,as the user likely wants to move around freely and not be restrictedby image anchors or poor WiFi coverage while navigating.

Some previous studies have touched upon user satisfaction ofgeneral AR interfaces [7][24][10], butmost studies that cover indoornavigation tend to focus more on technical aspects of the naviga-tional process. Few try to establish any kind of visual language orinvestigate what interaction cues are most intuitive, effective anduser friendly [10][14][18].

1.1 Research objectiveIn an attempt to establish a design foundation for AR applicationsmeant to be used in indoor retail environments, this thesis willfocus on how to present offers on the screen of an AR applicationused to navigate such a retail environment. The aim is to get a basicunderstanding of how discounted wares should present themselvesand seek the attention of the user. Furthermore, what level of intru-siveness is appropriate as to make users aware of the offers aroundthem, but not distract or irritate the user during his or her shoppingexperience? The research question to be answered is thereby; Howcan one effectively balance visual intrusiveness with awareness ofoffers when presenting discounts to users of an Augmented Realityapplication in a retail environment?

2 BACKGROUNDEver since the term Augmented Reality was first coined back in1968, [6], research on the subject would for decades remain largely

focused on technical aspects of the phenomenon [14] [11]. At thetime, little to no attention was paid to the potential user audienceor Human-Computer Interaction (HCI) theory in general.

Almost 40 years after the conception of AR, Dünser et al [12]conducted one of the first studies with the aim of applying commonHCI principles to AR applications and suggested a few basic guide-lines to follow. Since then, the technology has made huge strides,and most modern smartphones now house the capability of produc-ing more or less advanced, interactive AR content. The technologyis more accessible than ever before, and with a current estimatedinstall base of around a billion devices [5], smartphones is currentlythe biggest target platform for AR content. While other, more high-end wearable technologies are rapidly developing, predictions saythat smartphones will still account for an overwhelming majorityof the predicted mobile AR global audience of 2.5 billion users and$70 billion in revenue by 2023 [8]. Continued expansion and globaladaptation of smartphone-based AR is therefore very likely in allkinds of fields in the coming years, and as such, this study mainlycovers AR in the context of these aforementioned phones. Afterall, with such a huge global audience, making AR interfaces onsmartphones as user friendly as possible will be in the interest ofa broad audience, as it will be beneficial to both users, developersand investors while AR finds it footing.

Olsson and Salo [24] conducted a user study that documented 84different user's most and least satisfying experiences with mobileAR applications. The study concluded that the most satisfying ex-periences reported by users were related to the instrumental utilityvalue of the application, and its usefulness in so called informationacquisition activities. In other words, the most satisfying apps werethe ones that effectively delivered valuable information to its usersas a tool or extension of reality. The most unsatisfying experienceswere related to hardware deficiencies or instrumental expectationsnot being met.

Three years later, Kourouthanassis et al [18] studied HCI inAR with a similar goal in mind, which resulted in an additionalfew design principles. These include using the real world contextfor augmented content, delivering relevant-to-the-task content,and supporting interaction techniques that are akin to real worldbehavior, or similar to what the user is familiar with.

Ko et al [17] demonstrated that these type of design frameworkscan work, as all the knowledge derived from multiple proposed de-sign frameworks were summarized by conducting a comprehensiveliterature study. Subsequently, all of the suggested principles werenarrowed down into a list of 22, using a panel of ten UI/UX expertswith at least two years of professional experience. Finally, an ap-plication prototype that adhered to the findings of the study wasdeveloped. It was found that the developed prototype improveduser satisfaction and resulted in nearly 50% faster average com-pletion time in comparison to similar market applications, thusproving the importance of a user focused development process forAR applications.

2.1 AR navigationMultiple studies have covered the usage of AR to navigate an indoorspace using a visual interface on a screen [23][13][22], but thesestudies tend to focus on technological optimization or navigational

efficiency and accuracy. Few research questions related to HCItheory, visual language or how to most adequately get an intendedmessage across to the user [10][14].

One of these few is a recently conducted study that focused onuser experience [9] observed visual guidance systems in modernvideo games, with the goal of applying any lessons learned to aframework for AR navigation applications. While the study lacksany proper user testing, and acknowledges that video games and theAR space are different enough that not every lesson learned fromvideo games can be applied to AR, it provides a few interestingconclusions. It strongly suggest that ”Discover” cues (cues thatsubtly tell a user what is interactable and what is not) will beimportant as AR finds its footing, since everything in the physicalworld is not interactable in AR. It also points out that even thoughit is possible to map the reasons for using interaction cues and theevents that trigger them from video games to AR applications, thevisual design of these cues needs additional nuance in the currentresearch environment.

Another study focused its efforts on user personalization, andcreated a visual interface in which points of interest were colorcoded according to how well they matched the user’s request [27].However, the application used in the study did not try to directthe user's attention in any specific way, and thus differs from thisstudy's goal of making users aware of points of interest (in thiscase, offers) around them. A few other research projects have dis-cussed visualization [15] and user engagement [29] in AR, but thesestudies have either been heavily focused on technical aspects orconceptualized very broad, general guidelines not related to indoornavigation applications.

2.1.1 Design implications of geometric AR. It is worth pointingout that none of the above mentioned research papers studied orused so called geometric AR for their studies, and thus the designimplications of this particular type of AR (used in this study) onapplications used for indoor navigation are unclear. However, sincethe phone and its sensors need to be aware of the geometry of theenvironment around them, one clear natural design limitation isthat the phone needs to be held in a manner that lets the cameraand sensors see the environment in front of them.

2.2 IntrusivenessTo visually catch the customer's attention while using a smartphoneto navigate an indoor space with AR, one or multiple, more or lessintrusive visual clues will need to be presented on screen. Intrusive-ness, as defined by Li et al [20], is considered to be "a perception orpsychological consequence that occurs when an audience’s cogni-tive processes are interrupted". In this study’s case, the audience’scognitive process is considered to be the interaction with the navi-gational AR interface of the application used in the study, and anintrusive occurence will therefore be any kind of interruption thatthe customer experiences with that application.

MacIntyre et al [21] suggested that every new medium shouldestablish a visual language and certain conventions, which as evi-dent by the earlier research presented in this section, has not yetbeen done in terms of indoor navigation AR applications. That iswhy this study will apply the knowledge from this aforementionedresearch to an AR application designed for retail navigation, and

2

Figure 1: The scientific process.

try to establish early design conventions for presenting offers tocustomers in this space.

2.3 DeliminationsAs this study focused on getting a user's attention through visualcues, no additions to the application's already existing navigationalfeatures was developed or prototyped. Because of time restrictions,it was also entirely focused on visual cues, which means no timewas spent researching or developing any other form of cues such assound or haptics. Instead, it heavily focused on presentation of UIelements, how intrusive these elements should or should not be toavoid being perceived as distracting or annoying by the user, andhow the user should be able to interact with the information pre-sented to in some way improve or expedite the shopping experience.It also did not research any AR platforms besides smartphones, andso the results derived from this study can not necessarily be appliedto any other AR-capable hardware.

3 METHODThis chapter describes the scientific methodology used to answerthe research question of the study in greater detail.

The study was conducted in partnership with Stockholm-basedstartup Tangar Technologies AB, who are currently working onbringing an AR-based indoor navigation application to market. Thebulk of the study consisted of prototyping and development ofmultiple different visual interaction cues for the existing Tangarapplication, that were later tested and evaluated by users in a retailstore in Stockholm.

The development and the user studies conducted were supportedby the research background presented in section 2, which mostlyfocused on previous work in the field of navigation with AR in orderto grasp what previous research had focused on and to understandwhat evaluation methods would be appropriate for user studies inthis field. The research and development process is represented byfigure 1.

3.1 Literature reviewPreviously presented in section 2.

3.2 Prototype DevelopmentThe visual interfaces and interaction cues were developed usingUnity 2018.3.6, and later deployed to a Lenovo Phab 2 Pro smart-phone for testing. Since Tangar had a working prototype of theapplication, the focus of the development phase was to add visualsand interactions to the preexisting user interface to answer the re-search question. The prototype application was built on the Project

Tango framework for its AR capabilities, but the application couldalso be run in a simulative fashion in the Unity editor for quickertesting and efficient debugging when it was inconvenient to buildand run the application on a supported smartphone.

Three different ways of presenting an offer in a retail store en-vironment were developed. All three implementations shared thesame database of virtual points of interest attached to real worldgoods in the retail store. They also shared a piece of code that cal-culated the distance from the user to the nearest offer to determinewhether the offer should be presented on screen or not. Apart fromthis, the implementations varied in how much information theypresented to the user, and thus, how graphically intrusive they wereto the navigational experience.

Figure 2: Example screenshots of the three implementationsdescribed in section 3.2, taken in the store where the userstudy was conducted. From the left: Scale (least intrusive),Panel (somewhat intrusive) and Popup (most intrusive).

3.2.1 First implementation - Scale. The least intrusive implementa-tion, which will be referred to as the "Scale" implementation goingforwards, was inspired by the work done by Dillman, Kody R., etal. [9]. Thus, it was a variation on a form of what they would call amore subtle ”Discover ” clue in the AR environment. When usingthis implementation, the point of interest label for each offer wassimply scaled up as the user got closer, to catch his or her attention.When the label had reached approximately double its original size,some additional placeholder text was attached below the title of thelabel to exemplify an offer description for the user (see Figure 1).The following two implementations also used this scaling system,but incorporated additional visual cues as well.

3

3.2.2 Second implementation - Panel. A second implementation(nicknamed "Panel"), which was somewhat of a mixture betweenthe first and the third implementation, displayed a small 2D panelon the left side of the screen when the user was within a certaindistance of the offer. The panel displayed the name of the discountedarticle, an image of the offer, and a bright red rectangle that said"offer" in capital letters (see Figure 2). If the user were to tap onthis panel, the app would start navigating to the point of interestthe offer was associated with. If the panel was ignored and the userkept moving, the panel was dismissed automatically after a while.

3.2.3 Third implementation - Popup. The third and final implemen-tation was the most intrusive and inspired by the work done byOlsson and Salo [24], which suggested that a big emphasis shouldbe placed on an application's ability to deliver valuable information.This implementation thus presented a popup canvas (containing thename of the product, an image and a short description of the offer)right in the center of the screen, delivering all necessary informa-tion about the offer, but in turn blocking parts of the navigationalview as well. It gave the user the option to dismiss the offer ornavigate to it by tapping on one of two buttons. Just like the panelin the second implementation, this popup would also disappear ifthe user continuously moved further away from the offer.

3.3 Pilot studyBefore the user study was conducted, a pilot study was held toensure that the study would be as frictionless as possible. This pilotwas executed in the same retail store as the final study, and hadone person go through the test described in section 3.4. As smallchanges were made to both the application and the test itself afterthe pilot, the data collected in the pilot is not utilized in the findingsand results of this report.

3.4 User studyThe application was tested by a total of 13 people in a retail storein Stockholm. An effort was made to have an as equal genderdistribution as possible, and thus 7 participants were male and 6were female. The age of participant's ranged from 19 to 50, althoughmost (10 out of 13) were in their 20s.

Before the user tests began, three points of interest in a certainpart of the store were marked as made-up example offers, to be dis-covered by users. Each participant tested all three implementationsdescribed in section 3.2, one at a time, by completing three lapsaround a short path where they would run into all three of theseoffers consecutively.

Prior to each test session, the study participant was briefly in-troduced to the basic functionality of the Tangar app, althoughspecific features were not discussed in any way to avoid taintingthe participant's first impression of the application. The user wasthen guided along the path in the store without the applicationin hand before they were deemed ready to begin. After each lapthe participant completed, the phone was handed back to the testmoderator, who preceded to switch the application to a differentstate, in which the next implementation could be tested by theparticipant. The order in which each participant experienced thedifferent implementations was different for each test session tominimize bias.

3.4.1 Evaluative methods. The evaluative methods for the userstudy were partially chosen based on the recommendations givenby Samini and Palmerius [28] as well as Bach and Scapin [3]. Thus,after each lap was completed, the participant filled in a short 5 pointLikert scale questionnaire [4] about the implementation they justexperienced, before moving on to try the next one. The Likert ques-tionnaire contained both negative and positive statements aboutthe implementation, to keep users on their toes while reading andresponding. The 5 point scale ranged from "Strongly disagree" and"Disagree" to "Agree" and "Strongly Agree", with a neutral alterna-tive in the middle. When all implementations had been tested, theuser answered some final questions regarding what they thoughtof the application and their experience with it, including what im-plementation was their favorite. In addition to this data, the appitself logged certain key variables to text files that was saved locallyto the phone's storage. These included total time spent with eachimplementation. the positional coordinates of the user in x, y, andz space for each frame and the magnitude of the current positionalvector.

The variable that contained the magnitude of the positionalvector was used to observe the overall movement of the user. Everysecond, the app compared the current value of the variable to thevalue from one second ago, and determined whether the user hadmoved or not. While it is impossible to say exactly what should beconsidered a pause in movement and what shouldn't for people ofdifferent lengths, genders and ages, several precautions were takento get as accurate of an estimation as possible. First of all, accordingto the Biostatistics and Epidemiology Research Design and AnalysisCenter at the University of Oklahoma [30], the average walkingstride step length for a man is approximately 0.395 meters, and 0.33meters for a female. Thus, it seemed reasonable that the app shouldnot consider anything above 0.33 meters a pause in movement.

Second of all, several real world tests were conducted. In thesetests, upon app initialization, the phone would first be transportedforwards. It would then be brought to a standstill for a few seconds,before it would once again be moved until the app was shut down.This test was repeated many times, with varying amounts of move-ment both during walking and during pausing, to determine whatvalue would be appropriate to distinguish between movement andpauses in movement.

After analysis of the positional data recorded by the phone, aswell as video recordings of every test iteration, it was deemed thatusing a threshold of 0.15 meters per second would portray theuser as moving even while walking very slowly, while at the sametime allowing for some small degrees of movement while standingstill. As a result, it was concluded that the app should considermovements that did not exceed 0.15 meters in any direction duringone second as a pause in movement.

3.5 Data processing and analysisBefore data analysis could begin, some precautions were taken tomake sure the data was as accurate and easy to analyze as possible.The positional data was manually processed file by file to removedata that was recorded as the app was initialized, before each userstarted moving. For the Likert scale questionnaire, it was madesure that all the negative statements and the responses to those

4

Table 1: Likert statements

Number Likert statement1 This system would make my shopping experience better2 This system would not distract me during my shopping experience3 This system would not annoy me during my shopping experience.4 This system would speed up my shopping experience.5 This system would make me more aware of offers in stores.6 This system would be a good complement to more traditional offer displays in stores.

statements were reversed, to more easily be able to visually analyzeits results. For example, if a statement was "This system woulddistract me during my shopping experience", it was switched to"This systemwould not distract me duringmy shopping experience".Subsequently, the scale was inverted as well, meaning the numberof disagreements and agreements (both strong and "regular" ones)were swapped with eachother.

3.5.1 Visualizations. Once the processing was complete, a websitewas created using HTML, CSS. Javascript, d3.js and Papa Parse (forparsing CSV-files containing the positional data) to be able to bettervisualize the processed quantitative data recorded by the phone.This website 1 allows the visitor to observe key data collected dur-ing the user study. The different types of visualizations were chosenon a case by case basis. To visualize data concerning the amount ofmovement versus the amount of pauses, as well as the result fromthe question regarding which implementation was every partici-pant's favorite, a few simple donut charts that displayed percentagevalues were created. When the time came to visualize the data fromthe Likert scale questionnaire, recommendations by Robbins andHeiberger [26] were followed, and thus the Likert data was orga-nized into diverging stacked bar charts. Some visualizations fromthis website will be presented as Figures in the Results section ofthis report.

4 RESULTSThe results of the study will be presented in two subsections. Onedescribing the results from the quantitative data collected duringeach test sessions, and one describing the results from the qualita-tive data collected after each session.

4.1 Quantitative DataThe results from the quantitative data can be split into four mainareas; Time spent moving and pausing during each implementation,total time spent with each implementation, the responses from theLikert scale questionnaire and each user's preferred implementa-tion.

4.1.1 Movement and time. The movement data recorded by thesmartphone used during the tests can be split into two parts. Oneis the total amount of time spent with each offer visualization im-plementation, and one is how much of that total time was spentactively moving versus pausing. How much time each participantspent with each implementation varied quite a bit, with values1Accessible at antonmartinsson.github.io/thesis

ranging from a high of 163 seconds to a low of 47 seconds. Everyparticipant except one spent the most time in the first implementa-tion they experienced.

When all values for each participant are aggregated and dividedinto an average, time spent with each implementation is a little moreevenly distributed, as the average time spent is 83 seconds for thePanel implementation, 90 seconds for the Popup implementationand finally 84 seconds for the Scale implementation, indicating thatthere is no significant difference in time spent with the app based onhow offers are presented. While participants spent 7 seconds longerwith the Popup implementation than the Panel implementation onaverage, it does not hold any statistical significance as the standarddeviation in time spent is much higher, at 29.34 seconds for Paneland 34.97 for Popup.

Figure 3: A visualization of the difference in average timespent moving and pausing for all participants in all threescenes, in percent. From the left: Panel, Popup and Scale.

When it comes to amount of time spent moving compared topausing, the same phenomenon can be observed. The biggest amountof time spent pausing usually occurred during the first implemen-tation each participant tested. Apart from that, percentages spentmoving or pausing varied quite a bit as well, with active movementvarying from 90% to 38% for individual participants. When aver-aged, the percentages even out to 65% active movement for Panel,59% for Popup and 67% for Scale, with the remaining 35%, 41% and33% representing a pause in movement.

This can be observed in the visualization presented in Figure 3. Itindicates that the time spent with each implementation plateaued astotal time spent with the app increased, and indicates that no signif-icant effect on the amount of movement or pausing was introducedby any specific implementation. However, just as with the totaltime averages, the standard deviation for both active movementand pauses in every scene eclipse the differences in percentagesbetween every implementation, and thus no statistical conclusioncan be drawn from these numbers.

5

4.1.2 Likert scale questionnaire. Overall, the responses received inthe Likert scale questionnaire suggest that most people are quitepositive towards the Panel implementation, somewhat annoyed bythe Popup implementation, and rather satisfied but slightly moreindifferent when it comes to the Scale implementation. In Figure4, 5 and 6, one can observe a stacked bar graph visualizing theresponses to the six statements in Table 1 for every implementation.On top of each chart, the different possible answers are color codedand presented in a legend. An x-axis with percentage values isalso displayed on top. By observing this axis, one can see approxi-mately how many percent of answers were in agreement with thestatements and how many were not.

Figure 4: Likert responses for Panel implementation.

Figure 5: Likert responses for Popup implementation.

In general, a more favorable response coincides with strongeragreements from every participant. As evident by comparing Fig-ure 4,5 and 6, the Panel implementation has the most favorableresponse overall. It has the biggest amount of strong agreementsout of every implementation, and the agreements generally over-weigh the disagreements quite heavily in all but statement number

2 and 4. Regarding statement 2 ("This system would not annoyme during my shopping experience"), opinions were split exactlyfifty-fifty between disagreements and agreements. When it comesto statement 4 ("This system would speed up my shopping experi-ence"), it is worth noting that none of the implementations receiveda favorable response, as the majority of participants disagreed withthat statement for every single implementation.

For all other statements, the responses can be considered favor-able for the Panel implementation, as most participants were inagreement that it would not considerably annoy them during shop-ping, while also making them more aware of offers in stores. Mostparticipants also agreed that the implementation would make theirshopping experience better, and that it would be a good complementto more traditional offer displays.

In Figure 5, we can observe that everyone was in more or lessstrong agreement about the fact that the Popup implementationwould make them more aware of offers in stores, but at a cost.A cost which is revealed by some of the other bars in the graph.Most obviously, a significantly bigger portion of participants feltannoyed or distracted when presented with an offer in the form of amore intrusive popup window, as more than half of the participantsagreed or strongly agreed it would distract them during shopping.This also seems to have slightly affected some participants' overallimpression of the implementation, as a few more participants are instrong disagreement with whether it would improve their shoppingexperience or be a good complement to existing offer displays.

Figure 6: Likert responses for Scale implementation.

Finally, Figure 6 indicates that the Scale implementation is lesspolarizing than the other two and quite well liked, but possibly alsonot considered as useful for finding offers. Perhaps most impor-tantly though, the amount of strong disagreements are the lowestof any of the three implementations, and it is seen as the leastannoying and distracting of them all. However, it is relatively clearthat less participant's believe it would make them find offers instores more easily. So while the experience is considered less an-noying and intrusive to users, that seems to come at the expense ofcustomers not being as aware of what is an offer and what is not.

6

Table 2: Questions to participants after completing the test

Number Question1 What did you think of your experience with the app overall?2 What, if anything, would you change about the app you just tested?3 How do you think this digital way of presenting offers would affect your shopping experience?4 Do you have any other miscellaneous comments?

4.1.3 Preferred implementation. When it came to each participant'spreferred implementation, 62% declared the Panel implementationto be their favorite. The remaining 38% chose the Scale implemen-tation, which means none of the participant's considered the Popupimplementation to be their favorite, and that the Panel implemen-tation was the most popular among the participants.

4.2 Qualitative DataThe qualitative data collected after each complete test session isbased on answers to the questions in Table 2. A summary of theresponses to each question will be presented in subsections below.

4.2.1 The overall experience. In general, the application as a wholereceived mostly positive feedback from the test participants. Someof the most commonly used positive words describing the applica-tion were, "good" (4 mentions), "interesting" and "liked" (3 mentionseach) as well as "exciting", "smooth" and "cool" (2 mentions each).A couple of participants also praised the app as easy to use or intu-itive, and one described it as "surprisingly useful". Three additionalparticipants specifically said that they liked being informed of offersby the app in general, and one of these said it would be easier notto forget what's on offer while using an app like this one.

In terms of more critical opinions, one participant said "It feltweird to walk around and "record" (i.e pointing the camera for-wards, as if one were recording something) when other peoplewere shopping close by" and another echoed similar feelings saying"It was weird to put a camera that close to other people's faces".This, coupled with the occasional concerned comments from storeemployees and miscellaneous customers that wondered why wewere "going around filming everything and everyone", might cer-tainly be a concern to take note of. A couple of participants alsoexpressed worries about ergonomics while holding a phone upto your face for a long period of time, one posing the question"Would one always have a hand free to use the application [whileshopping]?".

Two participants specifically said that the app needs improve-ments for it not to become too intrusive or disruptive during theshopping experience, and another added that "popups with offerscould get annoying if they show up too often". Two participantsmentioned that they would "probably not use the app" in its cur-rent state, with one of them calling the app "clumsier and morelaborious than just traditional shopping". The same participant alsoquestioned the frame rate of the application, and said that as a resultthe image was a bit blurry when you moved the camera around.

4.2.2 Effect on shopping experience. In terms of how participantsthought the app would affect their shopping experience, six peoplesaid it would make them more aware of offers around the store.

Figure 7: Themost commonwords used to describe the over-all app experience. The bigger each word is, the more timesit was mentioned by participants.

Another said it would make him buy more discounted wares whileshopping, and another three said it would affect their shoppingexperience positively (one with the caveat "as long as it does notget too distracting"). Two participants said they "would probablyspend more money" while shopping with the app, to which bothagreed was both positive and negative.

4.2.3 Suggested changes and improvements. Many of the suggestedimprovements involved the blue points of interest (or POI's) (seeFigure 2) placed around the store. These include that POI's shoulddisappear once the user gets very close, that they should not growas much as they did to not block their surroundings, that theyshould be more accentuated, that discounted POI's should have adifferent color than the rest as well as suggestions on improvementsin occlusion technology and the frame rate of the application ingeneral. One participant suggested that the user should have theoption to look at all offers before entering the store, and addingwhich ones you would like to be guided to upon entering. Nosuggestions concerned the Panel and Popup from their respectiveimplementations.

Some suggestions did already exist in the app or are currentlyunder development at Tangar, but were not directly involved in thisstudy. For example, two participant wished for a search functionthat could find both discounted and non-discounted offers, whichalready existed in the app version that was used for this study (butwas not found by those participants). Another participant wishedfor the possibility of using a personal shopping list in the app, which

7

is a functionality currently under development but one that wasnot deployed in the app when this test took place.

4.2.4 Misc. comments. One participant mentioned that an offerpresented during testing of the Panel implementation suddenlydisappeared the second it appeared. Another stated that she wasnot particularly aware of offers when there was no kind of popupor panel showing up.

5 DISCUSSIONThe goal of this study was to establish a base of design conventionsfor presenting offers to customers in an AR application used tonavigate a retail environment. This section will thus analyze anddiscuss findings from the data collected and presented in section 4with the goal of producing a conclusion that could be useful whendeveloping similar apps for retail experiences in the future.

5.1 Active movement versus pausingAs the quantitative data about each user's movement indicates, notangible difference between the amount of active movement andpauses can be observed. On a surface level, this only implicatesthat the participants spent approximately the same amount of timeactively moving and pausing with each implementation. However,what the amount of time spent moving or pausing during eachimplementation ultimately means on a more granular level is upfor debate, and should be investigated and discussed further infuture studies. For example, exactly why is it that the amount oftime spent standing still is about the same for two implementationsthat are wildly different in their execution (one highly intrusivethat requires user action, and one very passive, requiring no useraction)?

As presented in section 4, one user specifically said that she wasnot particularly aware of offers while testing the Scale implementa-tion, and another two suggested to further accentuate the points ofinterest that were offers. Furthermore, most suggestions on howto improve the app overall ended up being suggestions on how toimprove visibility of the points of interest. Is it thereby conceivablethat some people spent more time pausing and "manually" lookingaround for offers during the Scale (which only had the points ofinterest scale up when user got close) implementation as it didn'tdeliver information about offers as promptly and directly as theother two implementations? Possibly.

In the same way, is it feasible that the reason people pausedwhile testing the Popup implementation to read the informationpresented and act on it? Probably. Thus, if one is to extract furtherknowledge from similar movement data and know exactly whatcauses a pause in active movement in a future study, deliberationslike these should be considered.

5.2 Point of interest designAs mentioned in the previous subsection, it's certainly worth not-ing that most suggestions on improvements overall concerned thepoints of interest. As a vast majority of participants had opinionsabout how they should behave differently from the current imple-mentation, it is clear that the design and the behaviour of thesepoints of interest need to be overlooked thoroughly if one is todiscover the optimal way to labelling real world products in the

AR space. As they were not considered as distracting or annoyingby the participants as the other two implementations, the goal ofhaving the implementation be less intrusive upon the experiencewas at least partially achieved.

What was not accounted for however, was the fact that a scaledup point of interest took up quite a bit of the viewport when theuser got close, somewhat blocking the surrounding real world prod-ucts and other points of interest. As such, it is likely more waysof accentuating points of interest need to be prototyped and re-searched in the future. For example, one might argue they couldshrunken in size, and instead accentuated using different colors orvarying grades of opacity depending on how relevant they are tothe customer.

5.3 Balancing intrusion and awarenessPerhaps most importantly, one should note that the results from theLikert scale questionnaire suggest that the more intrusive a presen-tation of an offer was considered, the more annoyed or distractedthe participant felt. In addition, the fact that none of the partici-pants chose the most intrusive implementation as their favoritespeaks volumes. At the same time, it is also crucial to recognizethat as intrusiveness took a back seat and offers were presentedvery discreetly (in comparison), awareness of offers around thestore was considered to be lower. This, similar to dilemmas onefaces while creating any other form of marketing, is something fordevelopers and retail stakeholders to keep in mind when designingand developing similar applications in the future.

On one hand, a possible solution is to simply find an imple-mentation that is as reasonable as possible. An implementationwhich strikes the best possible balance between intrusiveness onthe navigational experience and awareness of offers around thestore. With that said, the choice does not necessarily have to beblack and white, and one could instead leverage the strengths of allindividual implementations for different situations.

If a good balance between intrusive and more discrete ways ofpresenting offers will not annoy customers considerably while alsomaking them more aware of offers in store, one could potentiallyargue every store could and should implement a mixture of differentways to present different kinds of offers. Say for example there iscurrently a personal, big loyalty discount on a product often boughtby the customer. Perhaps there would be a higher tolerance towardsintrusiveness in exchange for more awareness about that specificoffer?

In other cases, where an offer is not as personal and more generic,a more scaled back presentation might be wiser as the customermore likely could find it distracting or intrusive. If it is somethingin between, use an implementation that actively presents the offerto the user in a more discrete manner, but does not require any useraction to go away. In other words; Intrude if deemed necessary, butdo so carefully and selectively.

5.4 Privacy and physical fatigueSpeaking of intrusiveness, there is another form of intrusion onehas to consider when creating applications like these; Intruding onother people's private space with a camera. As evident by the twousers who said they felt uncomfortable holding the phone like they

8

were filming other people, people are not used to having strangersindirectly pointing a camera at them. Neither are they used tobeing on the opposite side of the issue, holding up a smartphonein front of one’s own face for a longer period of time. And whilethere are no direct solutions to these problems as long as the appis constricted to a traditional smartphone that needs to be visuallyaware (through use of the camera) of its surroundings to knowwhere offers are located in the store, these two issues share thesame couple of potential solutions that lie in the future of AR andindoor positioning.

One solution lies in what some call head-mounted displays, somecall AR/XR "glasses", where forthcoming evolutions and iterationsof products like the Microsoft Hololens and the Magic Leap headsethave the potential to normalize the use of AR in a public environ-ment. This could allow a version of the Tangar app deployed tothose platforms to possibly be considered less intrusive, as theyget more and more sophisticated and discrete. It also has a strongpossibility to reduce physical fatigue, as theoretically one wouldonly need to rest these more sophisticated glasses on one’s head,similar to how a regular pair of glasses work today.

Another lies in the future of indoor positioning, as it is feasiblethat as new technologies for indoor positioning are developed,positional accuracy can be achieved with less reliance on the viewof the hardware’s physical surroundings. If a new version of theapplication could make use of such technology, not only could itresult in less time spent indirectly pointing a camera at strangers,it could also reduce fatigue since a lesser amount of time would bespent on keeping the camera pointing forwards.

5.5 Technical concernsAlthough they were a clear minority, a few user's expressed con-cerns about the application's usefulness in its current technicalstate, and questioned whether they would personally use it or. It isimportant to note that some of the technical concerns these peoplehad likely can be adressed by developing a version of the applica-tion that can run on more recent, more powerful hardware using amore current AR framework.

The Lenovo smartphone used during the user test was releasedin 2016, and the Google Tango framework it used for the AR capabil-ities of the application has not been officially supported by Googlesince March 1st of 2018. Since then, more powerful mobile proces-sors have been released and implemented in flagship smartphones,and less performance-reliable AR frameworks by both Apple andGoogle have replaced the now cancelled Project Tango. However,it is important to note that while these newer frameworks are lessdemanding when it comes to performance, their geometric capa-bilities are not as robust as on a Tango hardware product, and sothe application would need to be rewritten quite extensively in amanner as the one described in the previous subsection.

However, if such a rewrite is successful, it could potentiallyhave additional benefits beyond just modernizing the product andimproving performance. As these newer frameworks are less relianton the geometry of its physical surroundings, a rebuilt applicationcould possibly eliminate the need to hold the smartphone up infront of one’s face during extended periods of time, as described inthe previous subsection.

5.6 EfficiencyFinally, it is notable that a majority of participants did not considerany of the implementations to necessarily speed up their shoppingexperience. This is definitely something to consider when develop-ing similar applications in the future, as customer appreciation islikely to increase if the app can save them time.

What would be required to change this perception is hard tosay though. It is possible that all it takes is a quicker, snappierapplication, but it could also be severely more complicated thanthat. It could involve an overhaul of the entire User Experience ofthe application, or better integration and communication betweendifferent features. It is also entirely feasible that the limited func-tionality of the application version tested by participants had aneffect on this feedback, and that a more fully featured applicationcould have positively altered people’s perception of its efficiency.

5.7 Method CritiqueThis subsection describes what might have been done differentlyto in order to affect the outcome of the thesis.

It was clear that most users spent the most amount of time ex-ploring the first implementation they tested. This could possiblyhave been amended by having each participant testing only oneimplementation, instead of having all participants test every imple-mentation. However, as the time available in the store where thetests were conducted was very limited and as many test participantsas possible was seen as advantageous, it was decided that everyparticipant should test every implementation.

Subsequently, it is important to note that the amount of partici-pants in the test is still quite small. More participants and therebymore data could possibly have produced additional insights andfindings.

Another shortcoming of this study is the lack of an extensivedesign process. With more time or better time management, thestudy would have benefited from iterating more and collectingfeedback on the design of each implementation. Such iteration couldhave improved the visuals and changed the participants'perceptionof the application, and thus also changed the results of the study.

5.8 Future WorkAs mentioned previously in this report, a few participants in thisstudy raised concerns about fatigue and when holding up a smart-phone in front of one's face for extended periods of time. Anotherfew raised concerns about privacy intrusion, and others had doubtsabout the smartphone medium as a vehicle for this type of applica-tion.

All of these doubts could be highly solvable in the future withAR head-mounted displays, like the Magic Leap and the MicrosoftHololens. Not only could current and future iterations of productslike these normalize the use of AR to the point that people donot consider them as intruding on their private sphere, it couldalso eliminate the fatigue of holding up the smartphone and let thecustomer have both hands freely available throughout the shoppingexperience.

In other words, if the headsets can be fully positionally awarethroughout the store, it is easy to imagine a concept where thedigital graphics are projected onto the real world in a similar way

9

to how they are today, but without the need of holding on to anyhardware with one’s hands. It also enables additional new inter-action modalities. Not only do these headsets have the ability totrack human hands if one wishes to bring hands back into the mix,they also possess the ability to track gaze. What one could researchwith gaze tracking in a similar scenario is not for this study toanswer, but it is possible it could help with everything from hands-free actions, to helping the system decide which offers should bepresented to the customer and which should not, based on wherethe customer is looking. With the increasing amount of rumors ofseveral big tech companies working on the next generation of ARin the form of similar high-end glasses, it would be very interestingto see a similar study with some kind of head-mounted high-endAR headset once the technology allows it, to see if the results areany different from this study.

Future research should also tap into the personalization systemscurrently used by stores to offer certain promotions to certaincustomers, and tentatively use them to decide how to present anoffer based on its relevance to the customer.

6 CONCLUSIONThe results of this study give developers and various retail stake-holders several things to consider when developing a digital ARproduct for indoor retail navigation with smartphones. The quanti-tative data produced, while not the most statistically definitive, givean indication that people appreciated a middle ground approachwhen it comes to balancing intrusiveness and awareness of offers,to not find themselves feeling annoyed or distracted while shop-ping. This may warrant a one way approach that strikes a goodbalance between being intrusive and making sure customers areaware of offers, or possibly warrant implementing several differentpresentational methods to present different kinds of offers basedon importance or customer interest. The qualitative data presentedmainly tells a story of suggestions for future iterations of the prod-uct, but also one of optimism towards the future. A future wheremodern AR frameworks can improve performance and where newhardware can lessen physical fatigue and privacy concerns. Eventhough some raised a few eyebrows at certain aspects of the ap-plication tested, the summarized opinions of participants reveal apositive and hopeful attitude towards the future of AR as a helpfultool in an everyday shopping experience.

In a future study, advancements could include more UI and UXdesign interations, incorporating a personalization system wherecertain offers are accentuated more based on relevance to the in-dividual customer, or using high-end AR hardware that might bemore prominently used by everyday consumers in the future.

ACKNOWLEDGMENTSFirst of all, I would like to acknowledgemy supervisor Björn Thures-son, who has been a constant positive driving force behind me forthe duration of this thesis. I also would like to thank Axel Nor-denström and the rest of Tangar for giving me the opportunityto work with an AR application that I think is the beginning ofsomething truly special. Lastly, I want to give thanks to everyonewho participated in the user study, as this study would not existwithout you.

REFERENCES[1] Adhikari, A., Zheng, V. W., Cao, H., Lin, M., Fang, Y., and Chang, K. C.-C. (2015). In-telligshop: Enabling intelligent shopping in malls through location-based augmentedreality. In 2015 IEEE International Conference on Data Mining Workshop (ICDMW),pages 1604–1607. IEEE.

[2] Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators &Virtual Environments, 6(4):355–385.

[3] Bach, C. and Scapin, D. L. (2004). Obstacles and perspectives for evaluating mixedreality systems usability. In Acte du Workshop MIXER, IUI-CADUI, volume 4. Citeseer.

[4] Bertram, D. (2007). Likert scales. Retrieved November, 2:2013.[5] Boland, M. 990 million arkit arcore smartphones in the wild. https://arinsider.co/2018/12/27/990-million-arkit-arcore-smartphones-in-the-wild/. Accessed: 2019-05-28.

[6] Caudell, T. P. and Mizell, D. W. (1992). Augmented reality: An application ofheads-up display technology to manual manufacturing processes. In Proceedings ofthe twenty-fifth Hawaii international conference on system sciences, volume 2, pages659–669. IEEE.

[7] Dhir, A. and Al-kahtani, M. (2013). A case study on user experience (ux) evaluationof mobile augmented reality prototypes. J. UCS, 19(8):1175–1196.

[8] Digi-Capital. For AR/VR 2.0 to live, AR/VR 1.0 must die. https://www.digi-capital.com/news/2019/01/for-ar-vr-2-0-to-live-ar-vr-1-0-must-die/. Accessed: 2018-09-02.

[9] Dillman, K. R., Mok, T. T. H., Tang, A., Oehlberg, L., and Mitchell, A. (2018). Avisual interaction cue framework from video game environments for augmentedreality. In Proceedings of the 2018 CHI Conference on Human Factors in ComputingSystems, page 140. ACM.

[10] Dirin, A. and Laine, T. (2018). User experience in mobile augmented reality:emotions, challenges, opportunities and best practices. Computers, 7(2):33.

[11] Dünser, A., Grasset, R., and Billinghurst, M. (2008). A survey of evaluation tech-niques used in augmented reality studies. Human Interface Technology LaboratoryNew Zealand.

[12] Dünser, A., Grasset, R., Seichter, H., and Billinghurst, M. (2007). Applying hciprinciples to ar systems design.

[13] Huey, L. C., Sebastian, P., and Drieberg, M. (2011). Augmented reality basedindoor positioning navigation tool. In 2011 IEEE Conference on Open Systems, pages256–260. IEEE.

[14] Irshad, S. and Rambli, D. R. B. A. (2014). User experience of mobile augmentedreality: A review of studies. In 2014 3rd International Conference on User Science andEngineering (i-USEr), pages 125–130. IEEE.

[15] Kalkofen, D., Mendez, E., and Schmalstieg, D. (2008). Comprehensible visualiza-tion for augmented reality. IEEE transactions on visualization and computer graphics,15(2):193–204.

[16] Kipper, G. and Rampolla, J. (2012). Augmented Reality: an emerging technologiesguide to AR. Elsevier.

[17] Ko, S. M., Chang, W. S., and Ji, Y. G. (2013). Usability principles for augmentedreality applications in a smartphone environment. International Journal of Human-Computer Interaction, 29(8):501–515.

[18] Kourouthanassis, P. E., Boletsis, C., and Lekakos, G. (2015). Demystifying thedesign of mobile augmented reality applications. Multimedia Tools and Applications,74(3):1045–1066.

[19] Levchev, P., Krishnan,M. N., Yu, C., Menke, J., and Zakhor, A. (2014). Simultaneousfingerprinting and mapping for multimodal image and wifi indoor positioning. In2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pages442–450. IEEE.

[20] Li, H., Edwards, S. M., and Lee, J.-H. (2002). Measuring the intrusiveness ofadvertisements: Scale development and validation. Journal of Advertising, 31(2):37–47.

[21] MacIntyre, B., Bolter, J. D., Moreno, E., andHannigan, B. (2001). Augmented realityas a new media experience. In Proceedings IEEE and ACM International Symposiumon Augmented Reality, pages 197–206. IEEE.

[22] Möller, A., Kranz, M., Diewald, S., Roalter, L., Huitl, R., Stockinger, T., Koelle, M.,and Lindemann, P. A. (2014). Experimental evaluation of user interfaces for visualindoor navigation. In Proceedings of the SIGCHI Conference on Human Factors inComputing Systems, pages 3607–3616. ACM.

[23] Mulloni, A., Seichter, H., and Schmalstieg, D. (2011). Handheld augmentedreality indoor navigation with activity-based instructions. In Proceedings of the13th international conference on human computer interaction with mobile devices andservices, pages 211–220. ACM.

[24] Olsson, T. and Salo, M. (2012). Narratives of satisfying and unsatisfying experi-ences of current mobile augmented reality applications. In Proceedings of the SIGCHIconference on human factors in computing systems, pages 2779–2788. ACM.

[25] Rehman, U. and Cao, S. (2017). Augmented-reality-based indoor navigation: Acomparative analysis of handheld devices versus google glass. IEEE Transactions onHuman-Machine Systems, 47(1):140–151.

[26] Robbins, N. B., Heiberger, R. M., et al. (2011). Plotting likert and other ratingscales. In Proceedings of the 2011 Joint Statistical Meeting, pages 1058–1066.

[27] Ruta, M., Scioscia, F., Ieva, S., De Filippis, D., and Di Sciascio, E. (2015). In-door/outdoor mobile navigation via knowledge-based poi discovery in augmented

10

reality. In 2015 IEEE/WIC/ACM International Conference on Web Intelligence andIntelligent Agent Technology (WI-IAT), volume 3, pages 26–30. IEEE.

[28] Samini, A. and Palmerius, K. L. (2017). Popular performancemetrics for evaluationof interaction in virtual and augmented reality. In 2017 International Conference onCyberworlds (CW), pages 206–209. IEEE.

[29] Scholz, J. and Smith, A. N. (2016). Augmented reality: Designing immersiveexperiences that maximize consumer engagement. Business Horizons, 59(2):149–161.

[30] Thompson, D. Stride analysis. https://ouhsc.edu/bserdac/dthompso/web/gait/knmatics/stride.htm. Accessed: 2019-05-20.

[31] Wang, C.-S. (2018). An armobile navigation system integrating indoor positioningand content recommendation services. World Wide Web, pages 1–22.

11

www.kth.se

TRITA -EECS-EX-2019:444