Mobile Marker-based Augmented Reality asan Intuitive Instruction Manual

HANNAH REUTERDAHL

Master’s Degree ProjectStockholm, Sweden May 2014

ii

Abstract

Instruction manuals are supposed to be used to ensure safe and proper useof medical devices. The instruction manuals could be paper or digital docu-ments, animations or videos. Conveying information and instructions throughthese media might not be the most efficient approach, especially in healthcarewhere physicians and nurses experience high cognitive load and prefer hands-onexperience with the devices [1].

This thesis investigates how a possibly more efficient and intuitive instructionmanual could be developed using mobile marker-based Augmented Reality; atechnology that combines a virtual and physical world.

By analyzing and comparing different marker-based Augmented Reality sys-tems and Software Development Kits (SDKs), the most suitable system, in termsof availability and functionality, was chosen for the implementation. Two mobileAR instruction manual prototypes were developed, as a proof-of-concept, usingUnity3D with Qualcomm’s Vuforia extension.

The resulting prototypes demonstrate the opportunity to integrate informa-tion and instructions in our physical world and the difficulties associated withthe interaction and user experience with the relatively novel Augmented Realityinterface.

v

Sammanfattning

Instruktionsmanualer tas fram för att se till att medicinteknisk utrustninganvänds på ett korrekt och säkert sätt. De instruktionsmanualer vi ser idag ärantingen pappers- eller digitala dokument, animeringar eller videos. Att erhållainformation och instruktioner från dessa medier behöver inte vara det mest effek-tiva tillvägagångsättet, speciellt inom sjukvården där personal ofta är utsatta fören hög kognitiv belastning och föredrar att lära sig ny utrustning genom praktiskträning [1].

Denna rapport utreder hur en eventuellt mer effektiv och intuitiv instruktions-manual skulle kunna utvecklas med hjälp av mobil, markörbaserad AugmentedReality; en teknik som kan visualisera virtuella element i vår fysiska värld.

Genom att analysera och jämföra olika markörbaserade Augmented Realitysystem och Software Development Kits (SDKs) valdes ett system som var till-gängligt och med tillräcklig funktionalitet för implementationen. Med hjälp avUnity3D och Qualcomm’s Vuforia utvecklades två stycken Augmented Realityinstruktionsmanualer för mobila enheter.

De resulterande prototyperna demonstrerar fördelen med att kunna integrerainformation och instruktioner i vår fysiska värld. Svårigheterna med teknologinverkar vara kopplade till användarupplevelsen och interaktionen i det relativt nyagränssnittet i Augmented Reality applikationer.

vi

Contents

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Main Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 52.1 Augmented Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 Healthcare Applications . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Method 113.1 Literature study and Article Search . . . . . . . . . . . . . . . . . . 123.2 Technical Study and Analysis . . . . . . . . . . . . . . . . . . . . . 123.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.4 User Experience Evaluation . . . . . . . . . . . . . . . . . . . . . . 12

4 State of the Art 134.1 Augmented Reality Systems . . . . . . . . . . . . . . . . . . . . . . 14

4.1.1 Markerless Systems . . . . . . . . . . . . . . . . . . . . . . . 154.1.2 Marker-based Systems . . . . . . . . . . . . . . . . . . . . . 15

4.2 Marker-based System Architecture . . . . . . . . . . . . . . . . . . 164.2.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2.2 Marker Detection . . . . . . . . . . . . . . . . . . . . . . . . 174.2.3 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2.4 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2.5 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.3 User Interaction and Visualization . . . . . . . . . . . . . . . . . . 184.4 Comparison of different Marker-based Systems . . . . . . . . . . . 214.5 Comparison of different Augmented Reality SDK’s . . . . . . . . . 244.6 Choice and Motivation for the Implementation . . . . . . . . . . . 25

4.6.1 Vuforia SDK and Unity3D . . . . . . . . . . . . . . . . . . . 25

5 Implementation 275.1 The Instruction Manual Devices . . . . . . . . . . . . . . . . . . . 275.2 Instruction Manual Implementation . . . . . . . . . . . . . . . . . 28

5.2.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.2.2 Building the models . . . . . . . . . . . . . . . . . . . . . . 29

vii

5.2.3 Introducing the markers . . . . . . . . . . . . . . . . . . . . 315.2.4 Providing information . . . . . . . . . . . . . . . . . . . . . 325.2.5 Creating the interface . . . . . . . . . . . . . . . . . . . . . 335.2.6 Extra features . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Result 356.1 Coffee Machine Instruction Manual . . . . . . . . . . . . . . . . . . 356.2 GM Counter Instruction Manual . . . . . . . . . . . . . . . . . . . 386.3 User Experience Evaluation . . . . . . . . . . . . . . . . . . . . . . 40

7 Discussion 437.1 Suggestions for Future Work . . . . . . . . . . . . . . . . . . . . . 45

8 Conclusion 47

A Camera Pose Estimation 49

B The GM counter Data Sheet 52

C Example of Scripts 54C.1 OnTouchDown.cs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54C.2 SmoothCameraFilter.cs . . . . . . . . . . . . . . . . . . . . . . . . 55

viii

List of Figures

1.1 Gartner’s Hype Cycle . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Translation of information . . . . . . . . . . . . . . . . . . . . . . . 62.2 Information augmented in the physical world . . . . . . . . . . . . 72.3 Augmented Reality applications in healthcare . . . . . . . . . . . . 9

3.1 The method of the thesis . . . . . . . . . . . . . . . . . . . . . . . 11

4.1 Mapping of different Augmented Reality systems . . . . . . . . . . 144.2 Examples of markers . . . . . . . . . . . . . . . . . . . . . . . . . . 154.3 The process of an AR marker-based system . . . . . . . . . . . . . 164.4 The perspective transformation of the marker. . . . . . . . . . . . . 184.5 Reactable, a sound interaction table running on ReacTIVision . . . 194.6 On-screen interaction . . . . . . . . . . . . . . . . . . . . . . . . . . 204.7 MIT Media Lab prototype using fluid interaction . . . . . . . . . . 204.8 Comparison of different marker-based systems . . . . . . . . . . . . 234.9 Comparison of different Augmented Reality SDK . . . . . . . . . . 24

5.1 The instruction manual devices . . . . . . . . . . . . . . . . . . . . 285.2 The implementation setup . . . . . . . . . . . . . . . . . . . . . . . 295.3 Modeling the coffee machine and the GM counter in Unity3D . . . 305.4 The positioning of the markers . . . . . . . . . . . . . . . . . . . . 315.5 Jura Instruction Manual . . . . . . . . . . . . . . . . . . . . . . . . 32

6.1 User pointing device towards marker and device . . . . . . . . . . . 356.2 The coffee machine AR instruction manual . . . . . . . . . . . . . 366.3 Instruction guidance for the "Coffee Cleaning"-program . . . . . . . 376.4 The resulting model applied to the real world GM counter . . . . . 386.5 Operation guide for the GM counter . . . . . . . . . . . . . . . . . 396.6 One of the users testing the coffee machine application . . . . . . . 40

A.1 World-to-Image-transformation . . . . . . . . . . . . . . . . . . . . 51

B.1 GM counter data sheet . . . . . . . . . . . . . . . . . . . . . . . . . 53

ix

x

Chapter 1

Introduction

1.1 MotivationInstruction manuals could be the most awkward part of a user’s experience witha device. Users are avoiding manuals as long as possible, while designers tryto develop interfaces so there will be no need for them. Yet, both users anddesigners, agree upon the continued presence of the manuals [2].

In healthcare, the instruction manual is supposed to be used to ensure safe andproper use of a medical device. Since healthcare is turning into an environmentwith complex devices, all using different interfaces, software and hardware, therehave been guidelines developed on how to write proper instruction manuals. Stillthe manuals seem to be comprehensive and complicated documents. Documentsthat seldom are being used by the healthcare personnel, who seem to prefergetting hands-on experience with the device or getting instructions from a moreexperienced colleague [1].

Different approaches have been taken in order to create more efficient methodsto use instruction manuals. These approaches include the digitization of theinstruction manual as well as the creation of animations or video instructions.Nevertheless, the problem seems to be that the manual tries to explain somethingout of its own context. It uses illustrations and 2D images, to describe thefunctionality of a device in our three-dimensional world. The translation fromthe "manual’s representation of the world" to "the real world" is then left to theuser. This adds an extra cognitive effort and might divert the user’s attentionfrom the message the manual tries to convey.

Augmented Reality, AR, is a technology that is creating an interface betweenour digital and physical world. By populating the world with information it couldprovide the user with instructions in the correct context, removing the translationthe user otherwise is required to do.

Even though AR has been around for almost two decades, the technology isstill being seen in its infancy. Gartner’s Hype Cycle is a methodology that pro-vides enterprises with processes to take smarter innovation adoption decisions bygiving an idea of how mature different technologies are [3]. Figure 1.1 illustratesGartner’s Hype Cycle for 2013 where it can be notice that Augmented Realityhave passed the peak of the Hype curve and is now going through a phase whereit is trying to mature as a technology. At this point, only successful technologieswill start to climb towards the Plateau of Productivity.

1

Figure 1.1: Gartner’s Hype Cycle for emerging technologies, 2013 [3].

One success factor for AR seems to be the fast development of mobile devices.Due to their availability, flexibility and pricing, they are becoming attractive tar-gets for AR. One report published in March 2014 [4] has estimated the market forAR between 2013-2018 and concludes that the development of AR strive towardssmartphone applications and that the maximum share of global AR revenue willbe held by the mobile device-based AR.

Together with the better performing mobile devices and also with the wearabletechnologies entering the market, AR seem to move into a position where it willcreate useful applications, especially in healthcare;

"AR technology presents massive lifesaving potential for the medicalindustry." [5]

"Medicine is becoming one of the strongest catalysts for augmented reality, asthe technology is having a dramatic impact in improving patient outcomes." [6]"This technology is going to play an important role in the future of medicine." [7]

1.2 Main GoalThe goal of this master thesis is to investigate how a mobile AR instructionmanual could be developed. By implementing such an application as a proof-of-concept, an evaluation could further assess different opportunities, difficulties andwhether such an application could create more efficient and intuitive instructionmanuals.

The thesis will focus on marker-based AR systems using optical detection,i.e. a camera will be used to perceive our world populated with markers. Thevisualization of virtual objects will be made using monitor-based visualization,i.e. using a mobile device.

2

1.3 Problem StatementHow can an instruction manual application be designed and implemented so thatit will:

- use the technology of marker-based Augmented Reality;

- run on a mobile device;

- create a more efficient way to employ information and instructions and

- add intuitiveness for its users.

1.4 Thesis OutlineIn Chapter 2 the reader is presented with the background of the found problemas well as the definition of AR. The current and potential applications of ARin healthcare are further described and the related work is presented in its ownsection.

Chapter 3 is covering the method for the thesis project.The state of the art is presented in Chapter 4. It includes an overview of

different AR systems as well as a more detailed description of the marker-basedsystems. The reader is further presented with the important aspects of userinteraction and visualization in AR applications. Different AR marker-basedsystems and SDKs are being compared and the choice of the system for theimplementation is motivated in the last section.

Chapter 5 presents the implementation of the AR instruction manuals. Theinstruction manual devices are being briefly described and the development pro-cess is explained.

The resulting prototypes are presented in Chapter 6 together with the resultsfrom the user evaluation of the prototypes.

The thesis is summarized in Chapter 7 and Chapter 8 with a discussion,suggestion for future work and a conclusion.

3

4

Chapter 2

Background

Medical device manufacturers are required by regulators (i.e. U.S. Food and DrugAdministration - FDA and EU Commission) to include instruction manuals withtheir medical devices [8]. The regulations are included in the labeling requirementunder 21 CFR 801 [9] and IVD Directive 98/79/EC [10] respectively, and statethat instruction manuals should supply users with necessary information for thesafe and proper use of medical devices.

In 2003, FDA approved the use of electronic labeling for medical devices.The motivation was the efficient storage of vast amounts of information, ease ofretrieval and ability to better manage changing information [11]. Even thoughthe digitization of the manuals increased the accessibility of information andinstructions, problems are still being perceived.

Problem 1. The instruction manual requires a design so that it can be readby biomedical engineers, sales personnel, government regulators, resulting in acomprehensive document containing information targeting users with differentexperiences and knowledge [12].

Problem 2. The people who in fact will use the device, i.e. the nurses andphysicians, tend to engage in hands-on training or consulting a colleague overreading the instruction manual [12].

Problem 3. Today’s instruction manuals, require the user to translate the in-formation and instructions provided in the instruction manual to the physicaldevice, (concept illustrated in Figure 2.1).

5

Operating elements JURA IMPRESSA X7-S

1. Drip tray2. Drip tray cover3. Ground container4. Heigt-adjustable coffee and cappuccino spout 5. Draw-off button 2 coffees6. Draw-off button 2 Espressos7. Draw-off button 1 coffee8. Draw-off button 1 Espresso9. Draw-off button Cappuccino10. Selector key for grinder11. Display dialogue system12. Grinders13. Funnel for cleaning tablets14. Water tank15. Cover for programming keys16. Draw-off button steam17. Draw-off button hot water18. Pivoting nozzle for hot water/steam19. Accessory drawer

Figure 2.1: The information translation between the traditional instruction man-ual and the physical device.

The translation requires extra cognitive effort from the user and could divertthe user’s attention. Further the procedure, of conveying information by tradi-tional means, might be frustrating and time consuming. In healthcare, it mighteven induce errors since nurses and physicians need to be able to operate devicesunder high cognitive load and in stressful situations.

FDA, provides a database called Manufacturer and User Facility Device Expe-rience (MAUDE) [13] which stores error reports associated with medical devices.Searching for event reports associated with instruction manuals for medical de-vices, the result below is given, for a ten year period:

- Device difficult to setup or prepare: 124

- Installation-related problem: 193

- Programming issue: 188

- Instruction for use: 43

6

One of the events found in the database is described as follows: "Receivedreport from user facility. An adult male pt1 had multiple crush injuries. The ptwent to cardiac arrest twice on scene then was airlifted to hosp2. Once at thehosp, staff attempted to use the device in question to warm blood for transfusionto pt, the fluid warming device was alarming and was not warming the blood. Thedevice was replaced with a functioning unit. The pt expired due to his injuries.The bio-medical dept3 at the user facility determined the alarming unit was dueto operator error. The device was set-up incorrectly. The user facility declined toreturn the device to the mfr4 as they determined there was no malfunction of thedevice."

When such an error occurs the blame often falls on the end-user. The vul-nerabilities are although often embedded into the system long before the eventtook place. Since medical device manufacturers are required to include properinstruction manuals with their devices and since users are required to complywith this information, there should not exist such errors. Manufacturers shoulddo all in their power to minimize misuse of a medical device.

Augmented Reality might be a technology that the manufacturers could usein order to create more efficient and intuitive means of conveying information,solving the above-mentioned problems by:

1. adapting information according to the user and its previous experience andknowledge;

2. allowing the user to perform hands-on tasks with the device while receivinginstructions;

3. removing the translation of the information between the real device and themanual, (compare Figure 2.1 and Figure 2.2).

Cappucino

Accessory drawer

Drip tray

Double espresso

Selector key for grinder

Figure 2.2: Information augmented in the physical world, (compare to Figure 2.1).1patient2hospital3department4manufacturer

7

2.1 Augmented RealityAugmented Reality (AR) is a technology where our physical world merges with avirtual world, creating an enhanced environment for its users. Navigation routesare being displayed on the road itself, surgeons use iPads for guidance in minimallyinvasive surgery and technicians watch how to assemble devices on-spot. AR isa great tool to visualize complex data, but also to be used in applications thatbenefit from a merge between the digital and physical world.

The definition of AR includes three properties describing its (i) interactivityand real-time environment; (ii) registration of virtual objects in 3D and; (iii) com-bination of real world elements and virtual objects [14].

By definition, the virtual objects presented in an AR environment are notonly being positioned in the physical world, but should also co-exist in the world,allowing interaction between the objects and the physical world as well as inter-action with its users. The definition excludes specification of any kind of senseaddressed by AR and therefore includes vision as well as smell, touch and hearing.Further, the definition does not limit AR to feature-enhancements but includesapplications aiming towards the replacement of missing senses. AR applicationscould augment sight for blind people or visualize sounds for hearing impairedpeople [15].

8

2.2 Healthcare ApplicationsCurrently, AR healthcare applications can be distinguished between the following;

(i) applications targeting surgery, (ii) educational applications for medicalstudents and (iii) applications focusing on patient consultation, education andtreatment.

a. Surgical Checklist [20]. b. Treating PLP - Phantom Limb Pain [24].

c. Penetrating renal calix guided with an iPad [18].

d. Exploring anatomy. ©4D Anatomy.

e. MedicAR Google Glass app for surgical education. ©Droiders

f. Treating phobias [25].

Figure 2.3: Augmented Reality applications in healthcare.

Using AR in minimally invasive surgery could improve the precision of sur-geons by visualizing the interior of the patient together with the instruments andthe patient itself. The iPad has been used in two different cases, one guiding thesurgeon to penetrate a patient from the skin to the renal calix [16] (Figure 2.3c),and another one helping surgeons to remove tumors located in the liver withoutdamaging important vessels [17]. Further a surgery checklist has been imple-mented by Droiders and Stanford University to improve efficiency and accuracyof the traditional surgical safety checklists [18], (Figure 2.3a).

In the field of medical education, students are practicing surgery [19] andexploring anatomy using AR [20], (Figure 2.3d).

Apart from surgery and education, AR is making contributions in assistivetechnology and rehabilitation [21], e.g. treating PLP - Phantom Limb Pain [22],

9

(Figure 2.3b). There have also been studies investigating whether AR can beused to help and treat people with physiological disorders such as phobias [23,24],(Figure 2.3f).

AR applications in healthcare are thought to have a breakthrough with thewearable technologies entering the market [5]. In the category of wearables,Google Glass has been showing promising results. One such application wasdemonstrated at Boston’s Beth Israel Deaconess hospital where a doctor couldsecurely access internal data while at the same time talking to patients in theEmergency Department [19]. The doctors testing the application noted greatpossibilities regarding real-time updates and notifications connected to locationservices, providing them with the right data at the right place.

At the Academic Medical center of Amsterdam, Google Glass have furtherbeen used to record a surgery [25]. This feature enables efficient documentationas well as great educational support. Further a Stanford surgeon used GoogleGlass during a surgery on a model. The Google Glass application, developedby Droiders [26], provided him with a step by step illustrated guide of a certainprocedure superimposed on the skin of a model, (Figure 2.3e).

2.3 Related WorkTang A. et al. have explored the efficiency of using Augmented Reality as aninstructional tool [27]. The study evaluates whether spatially overlaid informa-tion can improve human performance and provide cognitive support in assemblytasks. The conclusion was that using AR to overlay and register information ina workspace could improve the human performance and ease some of the user’smental workload.

One industry that has developed AR instructional tools is the car industry.This has resulted in applications covering fields such as maintenance, assemblytasks and instruction guidances. Car manufacturers noticed that users didn’t payattention to the instruction manual even though the interface of a car is ratherdifficult, including different controls, symbols and settings. As an example, Audideveloped a mobile application that can detect and return information regardingdifferent individual vehicle elements and also augment maintenance instructionson the vehicle [28].

A healthcare application developed by ANGELS [29] explores contextual in-formative AR to improve risk management in hospitals. The system was imple-mented as part of an ongoing project co-funded by the European Commissionand displays relevant risk information on-screen depending on the location of theuser.

Nilsson and Johansson have concluded that the need for time consuming ed-ucation and training of new healthcare personnel could be decreased using Aug-mented Reality [30, 31]. Their studies investigated how AR could be used asan instructional tool in a healthcare setting. Nilsson and Johansson created anAR system using a helmet-mounted device to show users instructions on how toactivate a surgical diathermy apparatus and how to assemble a trocar.

A review made by Zhu et al. concluded that there is an acceptance amongstthe users towards AR and that there is indeed potential for improving competen-cies by introducing AR in healthcare [32].

10

Chapter 3

Method

This chapter is presenting the method that was used for the thesis project. Themethod is summarized in Figure 3.1.

7

Operating elements JURA IMPRESSA X7-S

1. Drip tray2. Drip tray cover3. Ground container4. Heigt-adjustable coffee and cappuccino spout 5. Draw-off button 2 coffees6. Draw-off button 2 Espressos7. Draw-off button 1 coffee8. Draw-off button 1 Espresso9. Draw-off button Cappuccino10. Selector key for grinder11. Display dialogue system12. Grinders13. Funnel for cleaning tablets14. Water tank15. Cover for programming keys16. Draw-off button steam17. Draw-off button hot water18. Pivoting nozzle for hot water/steam19. Accessory drawer

RequirementsAssumptions

Discussion &Evaluation

Analysis

Implementation

AR what now?

?!

.

ProblemStatement

Background

Result

Figure 3.1: The method of the thesis.

11

3.1 Literature study and Article SearchThe literature study and article search was made in order to gain a deeper knowl-edge of AR and its current application in healthcare. Questions such as how ARis being used today, what the most recent technology is and how the future ARwill look like were sought to be answered. The focus lay among the healthcareindustry and instructional applications but general AR applications were alsoexamined.

3.2 Technical Study and AnalysisBy comparing different techniques and systems related to AR, a strategy for theimplementation was formed. A use case was conducted in parallel with the systemcomparison in order to map important features and requirements of the systemchosen for the implementation.

The results of the technical study and analysis are being described and sum-marized in Chapter 4.

3.3 ImplementationThe implementation phase was begun with a brainstorm focusing around thecomprehensive questions; what, for whom and where? What kind of informationis the user in need of? Who is the user and where is the user located? Differentfeatures and interesting aspects were written down in form of a wish-list.

The applications and features were implemented using Unity3D together withthe Vuforia extension. The programming was made within MonoDevelop usingC#. The implementation is further described in Chapter 5.

3.4 User Experience EvaluationThe final phase included a minor user evaluation, to get a first impression of howusers would interact and experience an AR instruction manual. Five differentusers, with different degrees of mobile application experience, got to test one ofthe final prototypes and their thoughts, ideas and behavior were noted. Ques-tions were asked in an open-question style encouraging the users to freely expresstheir experiences with the application. The questions were focused around theapplicability, ease of use, user experience and information content. The users didnot get any prior information about the application.

12

Chapter 4

State of the Art

This chapter presents the state of the art in Augmented Reality. A general intro-duction to different AR system architectures is given with a focus on marker-basedAR systems. The important aspects regarding user interaction and visualizationassociated with AR is covered in its own section. A comparison between differentmarker-based AR systems and SDKs is given together with the motivation of thefinal choice of system for the implementation.

13

4.1 Augmented Reality SystemsAugmented Reality systems rely on technologies that recognize and track thephysical world around us with the goal to render and visualize virtual objects inthat world.

The architecture of AR systems differs and can be categorized depending onthe scene capturing, detection/tracking method, user interaction and visualization(Figure 4.1).

USER INTERACTION

Tangible

Embodied

DETECTION/TRACKING

VISUALIZATIONARtechnologies

HARDWARE/SCENE CAPTURING

Handheld

Video see-through

Projection

Optical see-through

Optical

Accelero-meter

GPS

Gyroscope

Compass

Magnetic

Monitor

Marker-based Markerless

Fluid

On-screen

Wearables

Figure 4.1: Mapping of different Augmented Reality systems.

The scene capturing can be done using wearables (e.g. head-mounted displays,Google Glass) or handheld devices (e.g. smartphones, tablets). The detectionand tracking methods include the use of GPS, compass, inertial tracking (usinggyroscopes and accelerometers), optical tracking (markerless or marker-based) ormagnetic tracking (using magnetometers). The visualization can be made usingvideo see-through (opaque head-mounted display and video stream), optical see-through (transparent head-mounted display), monitor based (hand-held devices)and projector based technologies (projections onto real world objects).

This thesis focuses on optical detection using handheld devices and on-screeninteraction. The optical detection and tracking perceives the world through acamera, e.g. a smartphone camera, and the detection can be made using mark-erless or marker-based techniques.

14

4.1.1 Markerless SystemsThe markerless systems is gaining popularity and the mobile implementation seemto become feasible together with the more advanced mobile devices and sensorsentering the market. The advantage of using a markerless system is that theenvironment doesn’t need to be prepared with markers but can be tracked as itis. The current limitations of these systems are the needed computational powerand the sparse availability of natural features in the world around us.

Markerless tracking detects interest points, so called natural features, in thephysical world using methods called Natural Feature Tracking (NFT). Popularalgorithms that are being used for detection and tracking are Scale InvariantFeature Transform (SIFT), Speed Up Robust Features (SURF) or the faster bi-nary detectors Oriented FAST and Rotated BRIEF (ORB), Fast Retina Keypoint(FREAK) and Binary Robust Invariant Scalable Keypoints (BRISK) [33]. Thesealgorithms detect and describe local features in an image by certain metrics.

In general, the procedure of markerless systems is first to find interest pointse.g. corners or blobs in the image. These interest points should be repeatableand recognized as the same under different viewing conditions [34]. When reliableinterest points are found the next step is to describe them by some metric. This isoften done with a feature vector describing the neighborhood of the interest point.The feature vector can be called descriptor and should be robust to differentlighting conditions, in different angles and together with noise. The final stepincludes matching of the descriptor from different views. The matching can bedone by calculating distances between the feature vectors [35].

An emerging method amongst the markerless techniques is SLAM (Simul-taneously Localization and Mapping), first proposed by Leonard and Durrant-Whyte [36]. SLAM was originally developed to be used in robotics but is nowfinding its way into AR applications. The general function of the algorithm isto create a map, or 3D model, that is representing the camera’s unknown envi-ronment and also to locate and track the camera in this environment. Both theenvironment and localization will affect each other, i.e. the better the map isknown the easier it is to get the position and the other way around.

4.1.2 Marker-based SystemsA marker-based system, or a so called fiducial marker system, is defined as alibrary of patterns or markers together with computer vision algorithms to detectand track the patterns or markers. A fiducial marker can be defined as an artificialobject with a known dimension positioned in a scene [37]. Examples of commonmarkers can be seen in Figure 4.2.

Figure 4.2: Example of markers.

Ever since the first marker-based system was introduced, there has been re-search focusing on the design of the markers. The research has resulted in a vastamount of different-looking markers and detection techniques. Today there are

15

square markers estimating the scale and pose accurately, frame markers beingseen as less obtrusive and dot markers providing accurate positioning.

Several studies have been focusing on comparing marker detection systems bytheir efficiency, accuracy, usability and reliability [38–40]. According to Fiala et.Al there are several metrics that define the quality of a marker system [37]:

- false positive rate, how often a marker is erroneously reported

- false negative rate, how often a marker is missed

- inter-marker confusion rate, how often a marker is mistaken for another

- lighting immunity, how well the system handles different lighting conditions

- occlusion immunity, how well the system handles occlusion of the marker

- perspective/affine projection support, how distorted the marker can be and stillbe detected

- planarity restriction, if the marker could be bent and still detected

- library size, how many different markers the system can detect

- minimum/maximum image size, how large or small the marker can be and stillbe detected

- speed performance, e.g. frame rate

Compared to the markerless systems, markers-based systems can efficientlyrun on small processors, provide us with accurate information regarding the scaleand position of real world objects and information contained within the marker,e.g. an ID. Due to this, marker-based systems are continuing to be an active re-search domain where people are contributing with faster and more robust systemse.g. Hirzer’s marker system [41] or Fiala’s ARTag system [37].

4.2 Marker-based System ArchitectureThe general approach of an AR marker-based system is illustrated in Figure 4.3.The different steps will be briefly described below.

7

Image binarization - Global Thresholding- Adaptive Thresholding

MARKER DETECTION

Countor �nding/Blob detectionRejection/Acceptance test

IDENTIFICATION

Binary decoding

TRACKING

Camera poseestimation

VISUALIZATION

IMAGE INPUT

PREPROCESSING

Figure 4.3: The process of an AR marker-based system.

16

4.2.1 PreprocessingThe images that are being preprocessed are taken from the camera’s video stream.In order to process images efficiently an image binarization is made [42]. Thebinarization operation makes sure that every pixel is either 0 or 1 and is performedusing a thresholding algorithm. There are different algorithms that can be used.The simplest but fastest one is a global threshold. Global thresholding appliesone threshold to the entire image, making a pixel black or white depending onone given value. This method is highly dependent upon the lighting and intensitychanges. A better method is to use adaptive thresholding. Adaptive thresholdinguses different thresholds depending on small patches of the image, making itrobust against different lighting conditions.

4.2.2 Marker DetectionThe detection method will depend on the design of the marker. Square markersare detected searching for lines and contours while circular markers are detectedsearching for blobs.

Suzuki [43] proposes one contour detection algorithm for square markers thattakes advantage of the fact that the markers are four-sided quads with a blackborder. If the detected lines are contours, or if they belong to a quad, they arestored as marker candidates together with their corresponding corners.

To keep the algorithms as low-cost as possible a fast rejection/acceptance testis often followed by the detection to ignore obvious non-markers. The test canbe done in various ways e.g. defining a minimum and maximum accepted size ofthe marker, checking how gradients change along a direction, counting holes inthe marker or checking if the numbers of points describing the contour are moreor less than four. If a candidate fulfills the requirements it is stored as a marker.

4.2.3 IdentificationOnce the marker have been detected, the code contained within it has to beextracted to check whether the marker is valid. The information encoded in amarker is often an ID number which can be used to connect the marker to certainmeta data. The square markers are usually divided into a number of smallersquares e.g. 7x7 of which 5x5 contains the code and the rest defines the externalborder.

To be able to extract the code the perspective projection of the marker needsto be removed (Figure 4.4). The perspective transformation is a transformationthat takes the marker’s corners in image space and finds a mapping to the realcorners of the marker.

17

[T]

Figure 4.4: The perspective transformation of the marker.

When the marker is unprojected the code can be read and validated. Markersystems often use Hamming distances for error correction. The Hamming distance[44] can be seen as a measure of how easily two markers are confused. It is a simplesum of the differences between two digital sequences, for example the Hammingdistance between the sequences 10001 and 00011 is 2. Hamming distances canalso be used to check the rotation of the marker since the correct rotation willhave a hamming distance of 0.

4.2.4 TrackingThe tracking plays an important role in order to make the augmentation feel asseamless as possible. Tracking means that the camera pose is being calculatedfor each frame. The camera pose describes the relation between the camera andthe marker and will be used to position the virtual objects.

For the positioning to be made accurately, the relative camera movement needsto be tracked with six degrees of freedom (6DOF), three variables for position(x, y, z) and three variables for orientation (roll, pitch and yaw). The trackingwill return a transformation matrix containing a rotation and translation. Thecamera pose estimation procedure is further explained in Appendix A.

4.2.5 VisualizationThe final step is to position the virtual elements in the physical world. This isdone using the transformation given by the pose estimation. The concept behindvisualization is further explained in 4.3 User Interaction and Visualization.

4.3 User Interaction and VisualizationThe goal of Augmented Reality is to visualize virtual objects in the physical world.Since the physical world can’t be controlled, nor the background, there is a needto be considerate when implementing the visualization and user interaction.

The main challenges when designing mobile AR experiences include the fol-lowing:

1. there are several different ways a user can interact with an AR application;

18

2. the camera view is being used as the user interface instead of a blank screen;

3. what appears on the screen will be dependent upon the location of the userand

4. the augmented objects need to be seamlessly integrated with the applicationand yet draw the user’s attention to important information [45].

The different ways a user can interact with an Augmented Reality applicationcan be categorized by the following:

(i) Embodied interaction. Embodied interaction means that the applicationsense and interpret the user’s actions and intentions using gyroscopes and ac-celerometers e.g. tilting, rotating and moving.

(ii) Tangible interaction. Tangible interaction means that the user is interact-ing with digital information via the physical environment. This could for examplebe a virtual, interactive button or object positioned somewhere in the real, phys-ical world. When the user interacts with the button or object the applicationwill register the action and provide feedback to the user accordingly. Tangibleinteraction has been used, for example, to create a system called Reactable [46]running on the ReacTIVision system (Figure 4.5). By moving objects placed ona screen the user can control different sounds.

Figure 4.5: Reactable, a sound interaction table running on ReacTIVision [47].

(iii) On-screen interaction. On-screen interaction is the way a user interactswith a device’s touchscreen, i.e. tapping, dragging and pinching. These inter-actions can be differentiated between the interaction with virtual elements posi-tioned in the physical world, but interactive through the screen, static GraphicUser Interface (GUI) elements that appear on screen (e.g. a static menu or but-tons) and real world objects being interactive through the screen (Figure 4.6).

19

Figure 4.6: On-screen interaction illustrated in Volkswagens AR service manual[48].

Massachusetts Institute of Technology (MIT) Media Lab has taken the userinteraction one step further combining tangible interaction with graphical on-screen interaction, something they call fluid interface [49]. The fluid interfaceis aiming towards the creation of a more natural AR interface. One of theirprototypes demonstrates a user walking up to a door whereby a lock code isaugmented on the door, if the user enters the right combination on the screen,the door will unlock (Figure 4.7).

Figure 4.7: MIT Media Lab prototype using fluid interaction. Copyright MITMediaLab [49].

The book "Designing the AR Experience: Tools and Tips for Mobile Aug-mented Reality UX Design" investigates new tools to be used in order to aid thedesign of mobile AR experiences [45]. These tools focus on user research, iterativedesign, rapid prototyping and user testing.

Prototyping is the first step to establish the functions and interface of anapplication. Usually prototyping is conducted by drawing different interfaces.For mobile AR applications this becomes rather difficult since the background isgiven by the camera stream, which can’t be controlled nor drawn easily. Keating

20

G. et al. explores prototyping by placing 2D notes and 3D objects in the physicalworld while studying them using a pass-through paper.

User testing should be introduced early in the development of an applicationin order to explore different kinds of user mental models and to explore howeffective the first design concepts are. The user study should include notes beingtaken during testing and could also be followed by a debriefing interview includingfollow-up questions.

Keating et al. concluded that the current Augmented Reality design processesneed to grow and adapt to the novel AR interface.

Kosara R. et al. propose another general visualization technique, called Focusand Context [50]. The method is a powerful instrument to visually communicateinformation when there is a lot of relevant structures in an image, commonlythe case in an AR view. This technique draws the users attention to a portionof data, Focus, while at the same time keeping the overall spatial relationshipof neighboring information, Context. The Focus-part can for example includehighlighting, opacity changes, zooming, modifying geometry, distorting data ormanipulating saturation. The Context is being decided depending on the situationand should be considered from case-to-case. The Focus and Context-method havebeen used in an AR application by Kalkofen D. et al. with promising results [51].

4.4 Comparison of different Marker-based Sys-tems

The method to detect, identify and track markers vary from system to system.ARToolKit [52] was among the first released marker tracking systems. It once

was a development standard and have been most commonly seen in experimentalor research-oriented AR-projects. The first ARToolKit system used templatematching for the marker detection and tracking. Template matching is basedon correlation between a found feature vector and a library of known featurevectors. Garrido-Jurado S. et al. [53] and Hirzer [41] stated that ARToolkit wassuffering from severe limitations. These limitations were based on the fact thatthe algorithm used global thresholding which resulted in a high false detectionrate and inter-marker confusion rate. Further the detection rate was slow due tothe correlation calculations and would become ineffective for a large database offeature vectors.

Following these limitations a system called ARTag [37] was developed. TheARTag system follows an edge-based approach using binary features. This ap-proach is being used by other marker systems as well, such as Hirzer [41], andworks by finding closed contours in the images. ARTag includes 2002 differentmarkers and has better inter-marker confusion rate since it includes methods forcorrecting errors.

The results and improved performance of ARTag inspired the further de-velopment of ARToolkit, resulting in ARToolKitPlus which finally evolved intoStudierstube Tracker. The Studierstube system [54] offer a number of differentmarkers, i.e. split markers, frame markers and dot markers. Studierstube wasdesigned to work with low processing power and less memory so that it couldrun on mobile devices. Studierstube has a specialized marker system for medicalapplications called Studierstube Medical which focuses on real-time acquisition,consistent data models and highly accurate visualization. The system have been

21

used in medical software applications as described by Garrido-Jurado S. et al [55].ARTag and Studierstube are both systems that require licensing.

Other systems that have been developed alongside are Matrix, BinARyID, In-stitute Graphische Datenverarbeitung, (IGD), Siemens Corporate Research (SCR)and Hoffman Marker System (HOM) [39]. Matrix has been used in the US PostalServices and also as an early digital information communication tool. HOM hasbeen used in maintenance and industrial information and documentation tools.HOM, SCR and IGD are closed-source systems. VisualCode is derived from theQuick-Response (QR) technology, but can also be used for tracking.

Research groups have contributed with their own open-source marker systemssuch as Chilitags [56] developed by the CHILI lab at École Polytechnique Fédéralede Lausanne, AprilTags [57], developed by APRIL Robotics Laboratory at theUniversity of Michigan or AruCo [53] developed by Aplicaciones de la VisiónArtificial at Universidad de Córdoba.

Chilitags and AprilTags are based on the ARTag approach. The systemsprovide robustness against occlusions and the ability to handle distorted markers.AruCo is based on a method using a board of markers to create a more stabledetection. This will allow for different lighting settings, fast camera movementsand large occlusions.

Other systems are studying the use of circular and dot markers. These systemsuse blob detection, taking advantage of the fact that circles become ellipses underprojection. The circular systems deliver high accuracy regarding the position ofa marker, but a single center coordinate is unfortunately not enough to estimatethe camera pose. These systems need four markers to derive the camera posewhich results in a slower performance, compared to the square markers. Exampleof circular systems are ReacTIVision [58] and RUNE-tag [59]. RUNE-tag is asystem created by a Dipartimento di Scienze Ambientali, Informatica e Statisticaat Universita Ca’ Foscari Venezia. RUNE-tag uses dots distributed on a circlewith a certain pattern coding. The system has been shown robust against largeocclusions. ReacTIVision have been used to create the Reactable system [46].

Figure 4.8 summarizes the comparison of different marker-based systems.

22

Chilitags

Binary Image

Edge-based

Decode pattern

1024

Open source

Geometry education

AprilTags

Binary Image

Edge-based

Decode pattern

1024

Open source

Robotics

TracksDetection

Nbr of markersAvailabilityApplications

TracksDetection

Nbr of markersAvailabilityApplications

ARTag

Binary Image

Robust Digital Encoding

2002

Require licensing

Mars Science Labratory

ARToolKit

Binary Image

Quadrilateral boundaries

Threshold-based

4096

Open source, well documented

Academic and research

ARToolKit Plus

Template image

Threshold-based

Correlation/template matching

-

Open source, well documented

Academic and research

Matrix

Binary Image/Natural features

Decoding patterns

4096

Commercial licence

Medical, visualization,

entertainment

Studierstube

Binary Image

Polygon approximation

Decode pattern

1024

Open source

-

AruCo

Blobs

Locate ellipses

Sector decoding

32 768

Open source

Maintenance,

assembly tasks,

biomechanics

Blobs

Locate ellipses

Sector decoding

20 000

Open source

-

Intersense RUNE-tag

VisualCode

SCR

HOM

Topology

Region adjacency graph

Tree pattern matching

216

Open source

Sound synthesizer

ReacTiVision

Figure 4.8: Comparison of different marker-based systems.

23

4.5 Comparison of different Augmented RealitySDK’s

The potential of Augmented Reality have resulted in companies starting to de-velop complete AR solutions targeting developers. An independent AR softwaredeveloping company have listed the top ten different AR SDKs for developing mo-bile AR applications [60]. Currently these are Qualcomm Vuforia, Metaio SDK,Total Immersion D’fusion, Wikitude SDK, Layar SDK, 13th Lab, ARToolKit,HP Aurasma SDK, Obvious Engine and String. The various specifications can befound at the Augmented Reality World Expo’s website [60]. Five of these systemsare compared and explained below (Figure 4.9).

DetectsTrackingPlatformVisualiaztionLicenseApplications

Wikitude SDK

2D images, location

NFT, GPS

iOS, Android, Blackberry

HTML, Wikitude Studio

0-$2 499

Entertainment, Navigation,

Fashion

Qualcomm Vuforia

2D images, 3D, anywhere

NFT, GPS, 3D, SLAM

iOS, Android

Unity3D

0-$8 500

Branding, instrucional,

entertainment

Metaio SDK

2D images, markers, text

NFT, marker, text

iOS, Android

Unity3D

Free

Education, entertainment

advertising, shopping

2D images, location

NFT, QR, GPS

iOS, Android

Layar Creator

0-$3 000

Advertising, branding

location services

2D images, 3D, anywhere

NFT, 3D, SLAM

iOS, Android

Unity3D

0-$5 000

Gaming, navigation

13th LabLayar SDK

Figure 4.9: Comparison of different Augmented Reality SDK.

Qualcomm Vuforia Vuforia is a software platform offering AR solutions fordevelopers [61]. It is a flexible system, designed to perform well on mobile devices.Vuforia supports both Android and iOS. The SDK is free to use and free to publishwith. Vuforia provides developers with a well established developer community.The SDK allows tracking of 2D images, markers and also text. Vuforia offers anextension to Unity3D, a tool that can be used as a rendering engine to visualizethe virtual objects.

Metaio SDK Metaio SDK is a complete AR solution which means that ithandles both tracking and rendering [62]. Metaio supports both iOS and Android.It requires licensing to be used without a watermark. Metaio detects 2D images,location and 3D objects using NTF, GPS and SLAM, respectively. The SDKfurther allows the use of 2D barcodes and QR-codes. One special feature Metaiohas implemented and patented, is to add a sense of gravity to the augmentedobjects. Metaio does not have the large developer community that Vuforia offers.Metaio could be integrated with Unity3D for the visualization and rendering ofvirtual objects.

Wikitude SDK Wikitude support detection of 2D images and location [63].The tracking uses NTF or GPS respectively. The system runs on iOS, Android

24

and Blackberry, but requires licensing.

Layar SDK Layar tracks 2D images and location using NTF and GPS respec-tively [64]. It runs on iOS and Android and requires licensing. It has a wellestablished support community.

13th Lab Point Cloud 13th Lab is a small Swedish company that is exploringSLAM to handle markerless AR [65]. The system runs on both iOS and Androidand require licensing. 13th Lab could also be integrated with Unity3D to renderand visualize virtual objects.

4.6 Choice and Motivation for the Implementa-tion

By comparing the above mentioned Augmented Reality marker-based systemsand SDKs, a system for the implementation was chosen. The choice was made inaccordance with the following requirements;

The system should:

- detect and track markers in real time on a mobile device;

- allow for occlusion during tracking;

- allow for interaction;

- be cross platform

- use fairly small, unobtrusive markers and

- be available to free.

To be able to comply with the requirements and stay within the time con-straint of the project, the choice was to use a complete AR system for the imple-mentation. Using such a system means that detection, tracking, modeling andrendering can be managed all together.

The chosen system for the implementation was Qualcomm’s Vuforia togetherwith Unity3D as the rendering and visualization component.

4.6.1 Vuforia SDK and Unity3DVuforia’s SDK includes different recognizable real world objects, so-called track-ables [61]. The detection and tracking algorithm uses Natural Feature Detectionand Tracking, which allows the system to match visible elements in the imagewith targets in a predefined set. Different elements have different ease of be-ing recognized, characterized by the detail level and repetitiveness. The currenttrackables Vuforia handles are image targets, cylinder targets, multitargets, framemarkers and words.

The frame markers were chosen for the implementation, since those markerscan be used in small sizes and are the least obtrusive ones. The frame markerconsists of a black squared frame with connecting bits of information. The blackframe is used to recognize the marker in real-time and the binary pattern is used

25

to decode the ID of the specific marker. The inside of the marker is not beingevaluated and could therefore have any appearance.

There are a total number of 512 different markers which due to the low pro-cessing power could be used in a single application.

The frame marker has to be entirely visible for the detection, but could bepartly occluded during tracking. When the marker is detected, Vuforia returnsa trackable result containing references to the matching target, a status (i.e un-known, undefined, not found, detected or tracked) and a pose matrix. The posematrix represents the current position and orientation of the marker with respectto the camera’s coordinate system.

The content rendering can be done using Unity3D. Vuforia provides an exten-sion package to Unity that allows developers to use vision detection and trackingwithin the Unity Integrated Developer Environment (IDE).

Unity is originally a cross-platform 3D game engine [66]. It allows for thecreation of simple geometrical objects such as spheres, cubes or rectangles, but isessentially a tool to enable behavior of 3D models imported from 3D programs i.e.Blender or Maya. The behavior of the models is specified using scripts written ineither C#, Javascript or Boo.

Unity3D comes with different licensing types depending on the number ofextra features one is interested in. The basic version is free and allowed to bepublished both privately and commercially.

26

Chapter 5

Implementation

This chapter introduces the chosen instruction manual devices. Further it ex-plains the implementation of the instruction manual applications in six differentsteps; setup, building the models, introducing the markers, providing information,creating the interface and adding extra features.

5.1 The Instruction Manual DevicesTwo devices were chosen to demonstrate two different scenarios where there wouldbe practical use of an AR instruction manual. Due to the time constraint ofthe project, these devices where chosen because of their availability. The finalprototypes of the AR instruction manuals are being seen as a proof-of-conceptand therefore the implementation could be feasible for any kind of device.

The first device was a Jura Coffee Machine and the second a Geiger MullerCounter (Figure 5.1).

27

Figure 5.1: The devices chosen for the prototype applications; a Jura CoffeeMachine and a Geiger Muller Counter.

The coffee machine has a rather complicated interface with unintuitive buttonsand display messages that are difficult to decode. The device comes together witha 17 pages instruction manual.

The GM counter is a beta contamination meter. It measures the intensity ofbeta radiation on a contaminated surface. The particular device is being used inhospital nuclear medicine centers and radiation chemistry laboratories of researchcenters. The device has a simple interface with one display and one control. Thedisplay holds a logarithmic scale that can be read in pulses per second or directlyin Bequerels for the radioisotopes C-14 or S-35, Ca-45 and P-32. Even thoughthe interface is simple the device could benefit from further on-spot informationregarding the device’s measuring modes and how the device should be operated.The device comes together with a data sheet (Appendix B).

5.2 Instruction Manual Implementation5.2.1 SetupThe setup for the implementation includes the building environments Unity3Dand XCode as well as the Vuforia package for Unity (Figure 5.2). This setup isused to create iOS applications, to create Android applications, Eclipse will beneeded instead of XCode.

28

Figure 5.2: The implementation setup.

The project was built in Unity3D. A separate scene for each application wascreated in order to build the AR environment. By importing the Vuforia pack-age to Unity, an "ARCamera" can be assessed in order to control the detectionand tracking of the scene. There is also a number of different so-called prefabsincluded, that can be recognized by the ARCamera, e.g. frame markers, imagetargets, words, cylinder targets etc. In our case the frame marker’s prefab wasused.

5.2.2 Building the modelsSince there were no 3D models available for the chosen objects, these were modeledin Unity3D using simple geometrical shapes. The accuracy of the virtual objects’positioning will depend upon how similar the model is compared to the real object.

The IDE of Unity can be seen in Figure 5.3. The figure illustrates how thecoffee machine and GM counter are being modeled according to their respectiveframe marker prefab.

29

Figure 5.3: Modeling the coffee machine and the GM counter in Unity3D.

30

5.2.3 Introducing the markersWhen the model is in place, the markers have to be positioned onto the physicaldevice. The size of the markers is important since it determines the actual scaleof the model to fit the real world object. Vuforia recommends using markers insizes between 3-10 cm.

The relation between the camera-to-marker distance and marker size createsan important trade-off. The markers should be unobtrusive, not taking too muchspace on the real device. Although, the smaller marker being used, the moreunstable the detection becomes. Different markers ranging from 2-6 cm weretested on the objects. The goal was to use as small markers as possible butstill have a robust detection of the marker. The setup that gave the most stabledetection, was two markers on the coffee machine, 5 cm and 3 cm, and a 3 cmmarker for the GM counter (Figure 5.4).

Vuforia further states that there should be a clearance area around the markerin order to detect the marker most efficiently. This clearance should be twice thesize of the marker’s black border to ensure that the contrast around the borderis sufficient. This concept was applied to all markers used.

Figure 5.4: The positioning of the markers.

31

5.2.4 Providing informationThe information and instructions included in the AR application were gatheredfrom the instruction manual and data sheet respectively (Figure 5.5 and AppendixB).

Users had already experienced problems with the interface and display mes-saging of the coffee machine. A particular problem was observed when the displayread Coffee Cleaning. Figure 5.5 illustrate what the user is facing opening theinstruction manual. The specific Coffee Cleaning instructions involved severaldifferent steps, all referring to the buttons displayed in Figure 5.5.

Figure 5.5: Jura Instruction Manual.

An instruction guidance for the Coffee Cleaning program was implemented aswell as information regarding the buttons in Figure 5.5.

The GM counter has a relatively simple interface and would rather benefitfrom an instruction manual containing information about certain concepts andphysics related to the device. The instruction manual was therefore included withthe theory behind radiation measuring and demonstrates the respective simpli-fied decay scheme to estimate the radiation, as seen in the results part in Chapter6. Further included information was the minimum beta efficiency and the con-version between Bequerel and counts-per-second. An instruction guidance wasimplemented to demonstrate how the device is supposed to be operated.

32

5.2.5 Creating the interfaceAR is dealing with a novel interface between the physical world and the digitalworld, an interface that is relatively new in the eyes of many users. It is thereforeimportant to put good consideration into the creation of intuitive interfaces anduser experiences. How should information be visualized and in what way shouldthe user interact with the information and the device? Due to the time constraintof this project, on-screen interaction was chosen. Different types of on-screeninteractions were explored in order to evaluate how the interaction procedureswould be perceived in practice. The name of the devices were implemented asvirtual objects, positioned in the scene. The buttons of the devices were notvisible but interactive via the screen. Further there was a regular static menuwith buttons the user could interact with.

The interface was created in Unity by creating and setting behavior of so-called "GameObjects". The behaviors were created with scripts written in C#,e.g. OnTouchDown.cs and SmoothCameraFilter.cs, see Appendix C. The scriptswere then applied onto the object/objects they would affect, e.g. SmootCamer-aFilter.cs was applied onto the ARCamera positioned in the scene.

Touches were handled by casting rays from the touch position on the device’sscreen into the Unity scene. If a ray hits an object, the event will be detected,using the OnTouchDown-script. The script will send a message to the affectedobject, telling it to run its OnTouchDown-method.

For example, if the user taps the virtual coffee machine name, the followingwill happen;

1. the OnTouchDown-script will send a ray from the user’s touch position intothe scene;

2. the ray hits the GameObject associated with the coffee machine name;

3. the OnTouchDown-script notices there was a hit and sends a message tothe GameObject, notifying it that it has received a tap from the user;

4. the GameObject picks up the message and runs its OnTouchDown-method;

5. the OnTouchDown-method redirect the user to the webpage of the device.

The coffee machine’s AR instruction manual was implemented with an instruc-tion guidance for the specific Coffee Cleaning-program. The guidance is accessedby tapping the augmented display screen. The user follows the instructions bytapping the "buttons" on the device’s screen.

The AR instruction manual for the GM counter allows the user to interactwith the control, via the device’s screen, to understand the different modes ofmeasuring. A virtual button, named "Instruction", is included in the scene whichwill start a guide to demonstrate different concepts behind measuring radiationusing the GM counter.

The visualization of the information was made using semi transparent blackboxes together with white text. This in order to maximize the user’s view ofthe physical world, and at the same time draw the user’s attention to the textelements. None of the virtual buttons positioned on the coffee machine werevisualized, but could be displayed with an extra feature, described below. Thismethod was used in order to make an attempted to comply with the Focus andContext-method, proposed by Kosara R. et al [50].

33

5.2.6 Extra featuresThe applications provide three extra features that can be controlled via a staticmenu. The first function controls the jitter of the camera pose. Since the algo-rithm is detecting and updating the camera pose in every frame, it seems like themodel is jittering. To solve this problem a smoothing filter was applied to theARCamera. The filter calculates an average of the pose from the last N numberof frames. Using a large number of averaging frames will create a feeling thatthe model is drifting, not following the camera movement. The trade-off betweenthe camera jitter and the drift can be controlled via the menu, by increasing ordecreasing the averaging number of frames, so called smoothness. This conceptcan be seen in the resulting demonstrated videos, see link in Chapter 6.

Another function allows the user to see the edges and buttons of the devices.The button Toogle Edges will display the model, created in Unity, as well as theinteractive objects. This can be used to see how well the model fits the real worlddevice or to realize which objects are interactive.

The third function is a Pause/Unpause function. This function allows the userto freeze the frame whenever they would like to continue reading the instructionwhere the marker is not visible.

34

Chapter 6

Result

The resulting prototypes are demonstrated on an iPad Mini Retina, (Figure 6.1),but have been tested on smartphones and tablets running on Android and iOS.

Figure 6.1: User pointing device towards marker and device.

6.1 Coffee Machine Instruction ManualWhen the user points the iPad towards the coffee machine, the application willdetect the marker placed on the machine. The marker will be identified and theapplication will augment the model of the coffee machine to the physical world,according to the calculated camera pose. The user will see the coffee machine

35

together with its model name and a couple of interactive elements, augmentedto the physical world. The user can then navigate through different buttonson-screen to find information, (Figure 6.2).

Jura Co�eeMachine

Figure 6.2: The coffee machine AR instruction manual.

The Coffee Cleaning-program can be accessed via the augmented displayscreen. If the user pushes the display, on-screen, they will be presented withan option to enter a text input. If the user enters Coffee Cleaning, the instructionguide will begin, (Figure 6.3).

36

Step 1 Step 2

Step 3 Step 4

Step 5 Step 6

Instruction Guide

Figure 6.3: Instruction guidance for the "Coffee Cleaning"-program.37

6.2 GM Counter Instruction ManualThe instruction manual for the GM counter includes an interactive control dis-playing information regarding different modes of measuring, (Figure 6.4).

Geiger-MullerCounter

Figure 6.4: The resulting model applied to the real world GM counter.

By toggling the control the user is presented with the respective simplifieddecay scheme, being the foundation for the Bequerel conversion. Informationregarding how the user can translate counts per second, CPS, to Bequerels isalso given. Further there are instructions on how to operate the device presentedtogether with information regarding the physics behind the radiation calculations,(Figure 6.5). The physics concepts cover activity, radiation and the number ofemitted particles versus the number of detected particles.

38

OperationGuide

Figure 6.5: Operation guide for the GM counter.

The final running applications can be seen on Vimeo using the following links:

AR Instruction Manual for Jura Coffee Machine: https://vimeo.com/95712660AR Instruction Manual for Geiger Muller-counter: https://vimeo.com/95719036

39

6.3 User Experience EvaluationThe user evaluation was made in order to get a first impression on how a userwould experience the coffee machine application. Five users with different degreesof mobile application experience were chosen for the evaluation. These kind ofevaluations have their own entire research field and there are multiple differentmethods of performing user experience evaluations. Amongst the first to developsuch methods for designing and evaluating different aspects of product designswas Jordan [67]. Due to the time constraint of the project the evaluation wasmade in its simplest form and should therefore not be considered as exhaustivenor as unbiased.

During the evaluation the users were free to use the application as they wishedbut they were informed that they should seek information regarding the "CoffeeCleaning" program. The users didn’t get any prior information regarding the ap-plication. The users were observed during their testing and questions were askedin an open-question style, encouraging the users to freely express their experi-ences with the application. The questions asked, focused around the perceivedease of use, perceived performance, intuitiveness, user experience, interface andinformation content.

Figure 6.6: One of the users testing the coffee machine application.

40

The following statements were noted during the user testing:

- Interesting and intuitive way to convey information and instructions

- Difficulties concerning the interaction with the marker, "Where should I holdthe device so the marker is being detected?"

- Difficulties realizing which objects are interactive, "Where is the informationlocated?"

- Does not realize when the marker is lost, "Why don’t I see anything?"

- Tangible or on-screen interaction, "How should I interact with the objects?"

The first thing to notice was that the users were not certain how they weresupposed to interact with the marker. One user went too close trying to alignthe marker with the camera, as one does with QR-codes, while another user wastoo far away for the application to even detect the marker. When the markerwas detected neither of the users had any problems to proceed and retrieve theinstructions they were seeking. Another apparent problem was users not realizingthat the buttons on the coffee machine were interactive via the screen. Theusers did not know if the application required tangible interaction or on-screeninteraction. Two users tried to push the buttons on the physical device insteadof the iPad’s screen.

All users found the information relevant and saw benefits such as the ability toadjust information to the current user or translating instructions and informationto another language, on-spot. The fact that the information was tightly connectedwith the device was appreciated amongst the users. The users saw potential inthe application and thought it was an intuitive way to convey instructions.

41

42

Chapter 7

Discussion

This thesis has investigated how mobile marker-based AR instructional manualscould be developed. Together with the conclusions from Nilsson and Johans-son [30] and Zhu et al. [32], there seem to be potential for such applications tocreate more efficient and intuitive means to convey instructions. The resultingprototypes are seen as a proof-of-concept of the opportunity of providing infor-mation in its correct context, removing the information translation users haveto make when conveying information from traditional instruction manuals. Nev-ertheless, there seem to be several difficulties that need to be addressed andconsidered before the technology can mature.

Two instruction manual prototypes were developed using marker-based Aug-mented Reality. Marker-based systems have the advantage that they can estimatethe camera pose from a single marker positioned in the world. The markers canfurther contain information that can be used to access other data. The disadvan-tage with the marker-based system is that the environment has to be preparedwith markers, due to this, the popularity for markerless systems are increasing,for example SLAM methods.

The implementation was made using Unity together with Qualcomm’s Vuforiaextension package. The Unity IDE allows a simple AR environment to be set upand running rather quickly, being an advantage due to the time constraint of thisproject. The system worked well overall, but there was a couple of found areas forimprovement. This included a case when the application lost track of the markerand broke the camera pose. In this case it would be better if the applicationsaved the last updated model until it finds the marker again. Such improvementsto the system are further discussed in 7.1 Suggestions for Future Work.

The method of performing a thorough investigation of articles and differentsystems led to the decision of using a complete Augmented Reality system. Thedecision was although also due to the time constraint. In a larger scope theproject would have benefitted from a comparison between the marker-based andmarkerless system, not making this a delimitation early in the project. Thedecision of using Unity together with Vuforia was partly based upon the decisionthat the systems are free to use. Another system could have been chosen if itwould have been an opportunity to pay for the licenses. In general, the chosenapproach and method for the thesis was perceived to be appropriate for this kindof proof-of-concept implementation.

The difficulties concerning the Augmented Reality Instruction manuals were

43

found in designing the user interaction and user experience, but also in the factthat these kind of applications might neglect the importance of fully understand-ing how a device is functioning.

Augmented Reality is creating a new kind of user interface with different formsof interactions that can all be mixed together. There are regular GUI elements,interactive virtual on-screen objects, interactive physical objects on-screen, virtualbuttons positioned in the physical world etc. The user evaluation indicated thatthere might be a confusion concerning the interaction with an AR application.

It seem to be important to provide the users with some kind of guidance;should the user tap the screen, tap an object in the physical world or tap avirtual object on screen? It was further noticed that the user needs feedbackto know when the application is working as intended. Such feedback should begiven whenever tracking is started or interrupted and if the user is standing toofar away. The feedback can be in the form of objects disappearing when themarker is lost or text informing the user that they have to move closer. The usershould also be encouraged to stay within the area so that the marker can be seen.

In general, there seem to be a need to further develop user interaction con-cepts around mobile AR applications. There is an opportunity for user experiencedesigners to step in and explore how the digital and physical world can be seam-lessly integrated. The concepts should focus on how users can be provided withinformation in the most efficient way, while taking the user’s cognitive capabili-ties and limitations into consideration. How should the data and information bepresented in order to create intuitive, natural experiences for the users?

Although, while trying to create the most efficient way that a user can conveyinformation, thorough understanding of a device or concept could be lost. Thefact that Augmented Reality could remove the information translation the userhave to make when reading a traditional instruction manual might help the userto perform a certain task, but won’t help the user to understand the concept be-hind the device or how it is functioning. The implementation of the GM countermade an approach towards this problem, trying to not only demonstrate whatthe different control modes mean, but also to explain some of the concepts be-hind the device, how radiation is being calculated and why the device has to beoperated in a certain manner. It would have been interesting to evaluate whetherthis instruction manual could aid students in getting a better understanding ofradiation measurement.

Further noticed problems were associated with the use of mobile devices asthe visualization medium. The user evaluation demonstrated that the users wereexperiencing difficulties holding the device, pointing it towards the marker and atthe same time interacting with the content within the application. In healthcarethis could become a hinder rather than a help since the personnel need to keeptheir hands free most of the time. The screen size of the mobile devices furtherlimits the area that could be used to visualize virtual objects. With this in mind,it is not a surprise that the wearable technologies and tangible interfaces areemerging in the AR field.

By the thorough review of articles and applications there was a notice thatthe Google Glass might be the device that will take Augmented Reality to thenext level, providing its users with a more natural interface by using speech andhand gestures etc. Although, the mobile devices still have the major advantageof their vast availability all over the world, pricing and unobtrusiveness. GoogleGlass includes larger social barriers to overcome before it could be used in a wider

44

context and in a more accepted way.The analysis of environmental sustainability and ethical aspects were out of

the scope for this thesis but have been considered. A few remarks can be made;the potential use of digital AR instruction manuals could decrease the numberof printed instruction manuals. The information and instructions can be spreadmore easily, which could deliver better cost-efficient solutions of conveying infor-mation. This could perhaps be a less expensive method rather than educatingprofessionals using other more experienced personnel. It could also be providedto anyone anywhere and due to the extensive availability of mobile devices.

7.1 Suggestions for Future WorkMost of the identified future work lies in the field of user experience and userinteraction. There is a need to build concepts to assure that the applicationswill be used as intended and that the user will have no problems interacting withthe application. Since some users in the evaluation thought they were supposedto push buttons on the real device instead of the screen, it would have beeninteresting to explore tangible interaction together with on-screen interaction.

As one can notice from the figures in Chapter 6, the models are not fitting thereal world devices perfectly. In a future application accurate 3D models of thedevices should be used to ensure the correct positioning of the virtual elements.

Vuforia seems to have a tendency to break the camera pose when trackingis lost. A solution would be to add a delay on the update of the model so thealgorithm has time to find the marker again. If it does not find the marker apreferable behavior would be that the model hides and the user gets notifiedthat tracking was lost. Another problem that was noticed was the fact thatthe system could get stuck in a "deadlock" position where it could not track themarker correctly and the only solution would be to restart the application. Thisproblem has to be solved in future applications.

An implementation of a text recognizer for the coffee machine application wasbegun but unfortunately not finished. The purpose of the text recognizer wasto allow the user to scan a display message, e.g. "Coffee Cleaning", rather thanprovide the application with a text input. Another interesting feature would be toinclude speech recognition, or for the GM counter application, a sound recognitionto calculate the number of counts.

Following this thesis the next step would be to implement a mobile AR in-struction manual for a medical device and evaluate the application in a healthcaresetting.

45

46

Chapter 8

Conclusion

Instruction manuals are important in healthcare since they assure safe and properuse of medical devices. However, the traditional instruction manuals might not bethe most efficient way to instruct users. Augmented Reality is a technology thatmight be able to introduce more intuitive, efficient instruction manuals, especiallyin healthcare where the cognitive load is high.

This thesis has investigated how instruction manuals could be implementedusing a mobile AR marker-based system. Two prototypes were developed, asproof-of-concept, to identify different opportunities and difficulties. The oppor-tunity seems to be the fact that an AR instruction manual could provide infor-mation directly in our physical world, creating a more intuitive experience. Thedifficulties seemed to be associated with the user interface and user experience.The technology need user experience designers to step in and explore how AR canintegrate our digital and physical world most efficiently. When the interface andexperience comes into place, there could to be a possibility that AR instructionmanuals could provide users with more intuitive and efficient instruction manuals.

The final AR instruction manuals were demonstrated as a use case by Bon-touch at the Augmented Reality World Expo in California the 27-29 May 2014.

47

48

Appendix A

Camera Pose Estimation

To be able to visualize our virtual objects correctly in the physical world, thereis the need for a robust camera pose estimator that tracks our markers posi-tioned in the physical world. When knowing the camera pose we can not onlyposition our virtual objects correctly but also keep the position regardless of thecamera movement. The pose estimation is equivalent of finding the six degrees-of-freedom, DOF, of the current camera. The problem of finding the 6DOF fromn point of correspondences is called the Perspective-n-Point (PnP) problem andwas coined by Fischler and Bolles [68]. Fischler and Bolles state that if we aregiven four known points that are coplanar and not collinear we end up with oneunique solution to the problem.

There are several approaches solving the PnP-problem, one using Direct Lin-ear Transform, DLT. The concept of the DLT algorithm is that given four verticeswe can solve the relation between image coordinates and world coordinates. Gen-erally this means calculating the homography between two frames. Homographyis describing the geometrical relationship between a camera frame and an inter-est point in an image e.g. a marker. The definition of homography is stated byHartley [34] as follows: Given a point (x, y, z) in one image and a correspondingpoint (x’, y’, z’) in a second image, then homography is given by H:xy

z

= H

x′

y′

z′

The homography estimation is often not accurate enough whereby a refine-

ment step is needed. One used approach is to make a first approximation of thehomography using a non-iterative algorithm e.g. DLT and then iterate to geta more accurate pose [69]. One of these iterative algorithms is the Levenberg-Marquardt minimization algorithm [70]. Levenberg-Marquardt algorithm mini-mize the reprojection error which means that it minimizes the sum of squareddistances between the observed projection’s image points and the projected objectpoints. This algorithm has been stated being one of the most reliable optimizationmethods in terms of accuracy and robustness against outliers and noise. Otheralgorithms that calculate the transformation matrix are the Hung Yeh-Harwoodpose estimation algorithm [71], the Rekimoto 3D position reconstruction algo-rithm [72] and the Orthogonal Iteration Algorithm by Lu et. al [73].

49

A precise camera calibration is essential for the initialization of the camerapose tracker. The camera calibration is often made offline prior to the registrationof the physical world and will provide us with the perspective transformation andlens distortion of the specific camera being used.

The principle behind camera calibration is to find projected points in an im-age given 3D points in the physical world. One method is to use images of achessboard taken at several different angles. Since the dimensions of the chess-board are known we can estimate the projection by calculating the homographybetween the different frames. There are different tools that have implementedcamera calibration methods. Matlab offers the Camera Calibration Toolbox [74]and openCV [75] have implemented a couple of functions to facilitate the calibra-tion.

The result of the camera calibration is the so called intrinsic matrix, K, de-scribing parameters of the camera such as focal length, skew and principal point:

K =

αx s x00 αy y00 0 1

These parameters provide us with a mapping between an ideal image and the

camera sensor coordinates and defines how the image will form on the sensor.Given the fact that today’s cameras have squared pixels and that columns

and rows often are straight we get;

s = 0 (A.1)αx = αy (A.2)

The intrinsic matrix is not enough to describe a rigid transformation, we needthe rotation and translation. The rotation and translation are given by the so-called extrinsic matrix, E.

E =[R T

]=

r11 r12 r13 Tx

r21 r22 r23 Ty

r31 r32 r33 Ty

By combining the intrinsic, 3x3, and extrinsic matrix, 4x4, we will get the

final expression translating 3D world coordinates to 2D screen coordinates. Thismatrix is called the camera calibration matrix, C 3x4. The intrinsic cameraparameters will stay constant for a given camera while the extrinsic parameterswill change together with the camera movement and will need to be calculated(using the above- mentioned approach) for every frame.

Below is the entire expression transforming a 3D world coordinate to the 2Dimage plane:xu

yv

1

=[K] [R T

] xw

yw

zw

=

αx s x00 αy y00 0 1

[R T] xw

yw

zw

Having all the above we know the combined effect of object-to-world, world-

to-camera and camera-to-image transformations and we can estimate the posecorrectly, (Figure A.1).

50

[RT]

Xc

Zc

Yc

Xw

Zw

Yw

Figure A.1: World-to-Image-transformation.

A further transformation might be needed from image coordinates to screencoordinates since the resolution of the device and the image might not be thesame, e.g. a video resolution of 1080x720 will not map correctly to the iPhone5’sresolution of 1136x640.

51

52

Appendix B

The GM counter Data Sheet

Figure B.1: GM counter data sheet.

53

Appendix C

Example of Scripts

C.1 OnTouchDown.cs

using UnityEngine;using System.Collections;using System.Collections.Generic;

public class OnTouchDown : MonoBehaviour {

void Update () {

int touch = 1;

// Create a new ray to cast in sceneRaycastHit hit = new RaycastHit();

for (int i = 0; i+touch < Input.touchCount; ++i) {if

(Input.GetTouch(i).phase.Equals(TouchPhase.Began)){

// Construct a ray from the current touchcoordinates

Ray ray =Camera.main.ScreenPointToRay(Input.GetTouch(i).position);

if (Physics.Raycast(ray, out hit)) {hit.transform.gameObject.SendMessage("OnTouchDown");

}}

}

}}

54

C.2 SmoothCameraFilter.cs

using UnityEngine;using System.Collections;using System.Collections.Generic;

[RequireComponent(typeof (QCARBehaviour))]public class SmoothCameraFilter : MonoBehaviour, ITrackerEventHandler {

public int numberOfSmoothingFrames = 10;private QCARBehaviour qcarBehavior;

private Quaternion smoothRotation;private Vector3 smoothPosition;

private Queue<Quaternion> rotations;private Queue<Vector3> positions;

// Update is called once per framevoid LateUpdate () {

transform.rotation = smoothRotation;transform.position = smoothPosition;

}

public void OnTrackablesUpdated() {if (rotations.Count >= numberOfSmoothingFrames) {

rotations.Dequeue();positions.Dequeue();

}

rotations.Enqueue(transform.rotation);positions.Enqueue(transform.position);

Vector4 avgr = Vector4.zero;foreach (Quaternion singleRotation in rotations) {

Math3d.AverageQuaternion(ref avgr, singleRotation,rotations.Peek(), rotations.Count);

}

Vector3 avgp = Vector3.zero;

foreach (Vector3 singlePosition in positions) {avgp += singlePosition;

}

avgp /= positions.Count;

smoothRotation = new Quaternion(avgr.x, avgr.y, avgr.z,avgr.w);

smoothPosition = avgp;}

55

// Use this for initializationvoid Start () {

rotations = new Queue<Quaternion>(numberOfSmoothingFrames);positions = new Queue<Vector3>(numberOfSmoothingFrames);qcarBehavior = GetComponent<QCARBehaviour>();qcarBehavior.RegisterTrackerEventHandler(this, true);

}

void IncreaseSmoothnessFactor() {numberOfSmoothingFrames = numberOfSmoothingFrames + 3;

}

void DecreaseSmoothnessFactor() {numberOfSmoothingFrames = numberOfSmoothingFrames - 3;

}}

56

Bibliography

[1] M. E. Wiklund, “Animated videos guide user interactions withcomplex medical devices.” http://www.mddionline.com/article/animated-videos-guide-user-interactions-complex-medical-devices,August 2008.

[2] J. Spolsky, “Designing for people who have better things to dowith their lives.” http://www.joelonsoftware.com/uibook/chapters/fog0000000062.html, April 2000.

[3] Gartner, “Gartner’s 2013 hype cycle for emerging technologies maps outevolving relationship between humans and machines.” http://www.gartner.com/newsroom/id/2575515, Aug. 2013.

[4] Markets and markets, “Augmented reality & virtual reality market bytechnology types, sensors (accelerometer, gyroscope, haptics), components(camera, controller, gloves, hmd), applications (automotive, education,medical, gaming, military) by geography - global forecast and analysisto 2013 - 2018.” http://www.marketsandmarkets.com/Market-Reports/augmented-reality-virtual-reality-market-1185.html, March 2014.

[5] H. Papagiannis, “How medical augmented reality will seam-lessly save your life.” http://thenextweb.com/dd/2014/01/01/medical-augmented-reality-will-seamlessly-save-life/, January2014.

[6] R. Sandhu, “5 applications of augmented reality technol-ogy.” http://newtech.about.com/od/augmentedreality/tp/5-Applications-Of-Augmented-Reality-Technology.htm, January2014.

[7] N. Douglas, “Role of augmented reality in medical industry.”http://www.augmentedrealitytrends.com/augmented-reality/medical-augmented-reality.html, January 2014.

[8] U.S. Department of Health Human Services, “Design control guidance formedical device manufacturers.” http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm070627.htm,Febuary 2014.

[9] U.S. Food and Drug Administration, “General device labeling requirements.”http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/

57

Overview/DeviceLabeling/GeneralDeviceLabelingRequirements/, May2012.

[10] European Commision, “In vitro diagnostic medical devices.”http://ec.europa.eu/enterprise/policies/european-standards/harmonised-standards/iv-diagnostic-medical-devices/index_en.htm, Febuary 2013.

[11] Director, Office of Device Evaluation, “Section 206 of the medical de-vice user fee and modernization act (mdufma) (new section 502(f) ofthe federal food, drug, and cosmetic act) electronic labeling for pre-scription devices intended for use in health care facilities.” http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/Overview/MedicalDeviceUserFeeandModernizationActMDUFMA/ucm109200.htm,March 2003.

[12] M. E. Wiklund, “Medical device user manuals: Shifting to-ward computerization.” http://www.mddionline.com/article/medical-device-user-manuals-shifting-toward-computerization,January 2002.

[13] U.S. Department of Health Human Services, “Maude - manufacturerand user facility device experience.” http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfMAUDE/search.CFM, March 2014.

[14] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre,“Recent advances in augmented reality,” Computer Graphics and Applica-tions, IEEE, vol. 21, pp. 34–47, Nov 2001.

[15] J. Carmigniani, B. Furht, M. Anisetti, P. Ceravolo, E. Damiani, andM. Ivkovic, “Augmented reality technologies, systems and applications,”Multimedia Tools and Applications, vol. 51, no. 1, pp. 341–377, 2011.

[16] C. Bichlmeier, “Mobile medical augmented reality appfor the apple ipad, an interview with prof. hans-petermeinzer.” "http://medicalaugmentedreality.com/2012/03/mobile-medical-augmented-reality-app-for-the-apple-ipad-interview-with-prof-hans-peter-meinzer/",March 2012.

[17] Augmented Reality Trends, “Doctors perform surgerywith augmented reality technology.” http://www.augmentedrealitytrends.com/augmented-reality/doctors-perform-surgery-with-augmented-reality-technology.html, 2013.

[18] Medical Augmented Reality, “Droiders releases augmented realityapp medicar for google glass.” http://medicalaugmentedreality.com/category/applications/?lang=en_us, March 2014.

[19] B. Levine, “This stanford surgeon shows us the future of medicine:Augmented reality google glass.” http://venturebeat.com/2014/03/12/forceps-scapel-google-glass/, March 2014.

58

[20] P. Fallavollita, “Research in Medical Augmented Reality.” http://campar.in.tum.de/Chair/ResearchIssueMedAR, June 2013.

[21] S. Ong, Y. Shen, J. Zhang, and A. Nee, “Augmented reality in assistivetechnology and rehabilitation engineering,” in Handbook of Augmented Re-ality (B. Furht, ed.), pp. 603–630, Springer New York, 2011.

[22] L. Clark, “Ar helps amputee experience first pain-free night in48 years.” http://www.wired.co.uk/news/archive/2014-02/25/phantom-limb-pain-cure, Feburary 2014.

[23] M. C. Juan, C. Botella, M. Alcaniz, R. Banos, C. Carrion, M. Melero, andJ.-A. Lozano, “An augmented reality system for treating psychological dis-orders: application to phobia to cockroaches,” in Mixed and Augmented Re-ality, 2004. ISMAR 2004. Third IEEE and ACM International Symposiumon, pp. 256–257, Nov 2004.

[24] M. C. Juan, M. Alcaniz, C. Monserrat, C. Botella, R. M. Banos, and B. Guer-rero, “Using augmented reality to treat phobias,” IEEE Comput. Graph.Appl., vol. 25, pp. 31–37, Nov. 2005.

[25] Medical Augmented Reality, “Google glass surgery at academic medicalcenter of amsterdam.” http://medicalaugmentedreality.com/2014/04/googleglass-surgery-at-amc-netherlands/, April 2014.

[26] P. G. Alburquerque, “MedicAR; Surgical Glassware with Augmented Realityat Stanford University Medical Center.” http://www.droiders.com/en/medicar-surgical-glassware-augmented-reality-stanford-university-medical-center/,March 2014.

[27] A. Tang, C. Owen, F. Biocca, and W. Mou, “Experimental evaluation ofaugmented reality in object assembly task,” in Mixed and Augmented Real-ity, 2002. ISMAR 2002. Proceedings. International Symposium, pp. 265–266,2002.

[28] Metaio, “Metaio & audi release interactive augmented reality man-ual.” http://www.metaio.com/customers/case-studies/case-studies/audi-ekurzinfo-app/, August 2013.

[29] Inglobe Technologies, “Augmented reality to support risk managementin hospitals.” http://arblog.inglobetechnologies.com/?p=1200, June2013.

[30] S. Nilsson and B. Johansson, “Acceptance of augmented reality instructionsin a real work setting,” in CHI ’08 Extended Abstracts on Human Factorsin Computing Systems, CHI EA ’08, (New York, NY, USA), pp. 2025–2032,ACM, 2008.

[31] S. Nilsson, “A cognitive systems engineering perspective on the design ofmixed reality systems,” in Proceedings of the 13th European Conference onCognitive Ergonomics September 20-22, Zurich 2006, pp. 154–161, 2006.

[32] E. Zhu, A. Hadadgar, I. Masiello, and N. Zary, “Augmented reality in health-care education: an integrative review,” PeerJ PrePrints, vol. 2, p. 335, 32014.

59

[33] Computer Vision Talks, “A battle of three descriptors: Surf,freak, brisk.” http://computer-vision-talks.com/articles/2012-08-18-a-battle-of-three-descriptors-surf-freak-and-brisk/,August 2012.

[34] R. I. Hartley and A. Zisserman,Multiple View Geometry in Computer Vision.Cambridge University Press, ISBN: 1849517827, second ed., 2004.

[35] H. Bay, T. Tuytelaars, and L. V. Gool, “Surf: Speeded up robust features,”in Proceedings of the ninth European Conference on Computer Vision, May2006.

[36] J. Leonard and H. Durrant-Whyte, “Simultaneous map building and local-ization for an autonomous mobile robot,” in Intelligent Robots and Systems’91. ’Intelligence for Mechanical Systems, Proceedings IROS ’91. IEEE/RSJInternational Workshop on, pp. 1442–1447 vol.3, Nov 1991.

[37] M. Fiala, “ARTag, a fiducial marker system using digital techniques,” inComputer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Com-puter Society Conference on, vol. 2, pp. 590–596 vol. 2, 2005.

[38] M. Muja and D. Lowe, “Fast matching of binary features,” in Computer andRobot Vision (CRV), 2012 Ninth Conference on, pp. 404–410, May 2012.

[39] X. Zhang, S. Fronz, and N. Navab, “Visual marker detection and decodingin ar systems: a comparative study,” in Mixed and Augmented Reality, 2002.ISMAR 2002. Proceedings. International Symposium on, pp. 97–106, 2002.

[40] M. Fiala, “Comparing artag and artoolkit plus fiducial marker systems,”in Haptic Audio Visual Environments and their Applications, 2005. IEEEInternational Workshop on, p. 6 pp., Oct 2005.

[41] M. Hirzer, Marker Detection for Augmented Reality Applications . PhD the-sis, Inst. for Computer Graphics and Vision Graz University of Technology,Austria, October 2008.

[42] D. Lélis Baggio, S. Enami, D. Escrivá Millián, K. Ledgen, N. Mahmood, J.Saragih and R. Shilkrot, Mastering OpenCV with Practical Computer VisionProjects. Packt Publishing, ISBN: 0521540518, first ed., 2012.

[43] S. Suzuki, “Topological structural analysis of digitized binary images by bor-der following,” Computer Vision, Graphics, and Image Processing, vol. 30,no. 1, pp. 32 – 46, 1985.

[44] R. Hamming, “Error detecting and error correcting codes,” Bell SystemsTechnology Journal, vol. 29, pp. 147–160, 1950.

[45] G. Keating, D. Guest, A. Konertz, N. Padovani, and A. Villa, “Designingthe ar experience: Tools and tips for mobile augmented reality ux design,”vol. 6770, pp. 135–141, 2011.

[46] Reactable, “Reactable system.” http://www.reactable.com/, May 2014.

[47] Universitat Pompeu Fabra Barcelona, “Reactable.” http://mtg.upf.edu/project/reactable(2014-05-20).

60

[48] Metaio, “Metaio and volkswagen announce marta aug-mented reality service support for new xl1 conceptcar.” http://www.metaio.com/press/press-release/2013/volkswagen-marta-augmented-reality-service-support/, August2013.

[49] V. Heun, S. Kasahara, and P. Maes, “Smarter objects: Using ar technologyto program physical objects and their interactions,” in CHI ’13 ExtendedAbstracts on Human Factors in Computing Systems, CHI EA ’13, (New York,NY, USA), pp. 2939–2942, ACM, 2013.

[50] R. Kosara, H. Hauser, and D. L. Gresh, “An interaction view on informationvisualization,” 2003.

[51] D. Kalkofen, E. Mendez, and D. Schmalstieg, “Interactive focus and contextvisualization for augmented reality,” in Mixed and Augmented Reality, 2007.ISMAR 2007. 6th IEEE and ACM International Symposium on, pp. 191–201,Nov 2007.

[52] ARToolKit, “Artoolkit.” http://www.hitl.washington.edu/artoolkit/(2014-03-01).

[53] S. Garrido-Jurado, R. Muñoz-Salinas, F. Madrid-Cuevas, and M. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducialmarkers under occlusion,” vol. 47, pp. 2280–2292, 2014.

[54] “Studierstube augmented reality project.” http://www.icg.tugraz.at/project/studierstube(2014-03-01).

[55] S. Garrido-Jurado, R. Muñoz-Salinas, F. Madrid-Cuevas, and M. Marín-Jiménez, “Integrated medical workflow for augmented reality applications,”Pattern Recognition, no. 0, 2014.

[56] Q. Bonnard, S. Lemaignan, G. Zufferey, A. Mazzei, S. Cuendet, N. Li, andP. Dillenbourg, “Chilitags 2: Robust fiducial markers for augmented realityand robotics.,” 2013.

[57] E. Olson, “AprilTag: A robust and flexible visual fiducial system,” in Pro-ceedings of the IEEE International Conference on Robotics and Automation(ICRA), pp. 3400–3407, IEEE, May 2011.

[58] M. Kaltenbrunner and R. Bencina, “Reactivision: A computer-vision frame-work for table-based tangible interaction,” in Proceedings of the 1st Inter-national Conference on Tangible and Embedded Interaction, TEI ’07, (NewYork, NY, USA), pp. 69–74, ACM, 2007.

[59] F. Bergamasco, A. Albarelli, E. Rodola, and A. Torsello, “Rune-tag: A highaccuracy fiducial marker with strong occlusion resilience,” in Computer Vi-sion and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 113–120, June 2011.

[60] Augmented World Expo, AWE 2013, “Tutorial: Top 10 augmentedreality sdks for development.” http://augmentedworldexpo.com/news/tutorial-top-10-mobile-augmented-reality-sdks-for-developers/(2014-03-30).

61

[61] Qualcomm Vuforia, “Vuforia sdk.” https://developer.vuforia.com/resources/sdk/(2014-03-26).

[62] Metaio, “Metaio sdk.” http://www.metaio.com/sdk/(2014-03-26).

[63] Wikitude, “Wikitude sdk.” http://www.wikitude.com/products/wikitude-sdk/(2014-03-26).

[64] Layar, “Layar sdk.” https://www.layar.com/documentation/layar-sdk/(2014-03-26).

[65] 13th Lab, “13th lab sdk.” http://13thlab.com/(2014-03-26).

[66] Unity Technologies, “Unity game engine.” https://unity3d.com/unity/download(2014-03-26).

[67] P. W. Jordan, Designing Pleasurable Products. Taylor Francis, 2000.

[68] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigmfor model fitting with applications to image analysis and automated cartog-raphy,” Commun. ACM, vol. 24, pp. 381–395, June 1981.

[69] S. Siltanen and V. T. Tutkimuskeskus, Theory and Applications of Marker-based Augmented Reality. VTT science, 2012.

[70] K. Levenberg, “A method for the solution of certain non-linear problemsin least squares,” Quarterly Journal of Applied Mathmatics, vol. II, no. 2,pp. 164–168, 1944.

[71] Y. Hung, P.-S. Yeh, and D. Harwood, “Passive ranging to known planar pointsets,” in Robotics and Automation. Proceedings. 1985 IEEE InternationalConference on, vol. 2, pp. 80–85, Mar 1985.

[72] J. Rekimoto and Y. Ayatsuka, “Cybercode: Designing augmented realityenvironments with visual tags,” pp. 1–10, 2000.

[73] C.-P. Lu, G. Hager, and E. Mjolsness, “Fast and globally convergent poseestimation from video images,” Pattern Analysis and Machine Intelligence,IEEE Transactions on, vol. 22, pp. 610–622, Jun 2000.

[74] J.-Y. Bouguet, “Camera calibration toolbox for matlab.” http://www.vision.caltech.edu/bouguetj/calib_doc/, Dec. 2013.

[75] OpenCV Dev Team, “Camera calibration with opencv.” http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html, Dec. 2013.

62