mobile room schedule viewer using augmented reality

Code:________________

Faculty of Engineering and Sustainable Development

MOBILE ROOM SCHEDULE VIEWER USING

AUGMENTED REALITY

Yunyou Fan

June 2012

Bachelor Thesis, 15 credits, C

Computer Science

Study Programme for a Degree of Bachelor of Science in Computer Science

Examiner: Stefan Seipel

Supervisor: Peter Jenke

Mobile Room Schedule Viewer

Using

Augmented Reality

by

Yunyou Fan

Faculty of Engineering and Sustainable Development

University of Gävle

S-801 76 Gävle, Sweden

Email:

[email protected]

Abstract

This work aims to present a historical overview and analysis on the current technical details

within one category of Augmented Reality systems – Mobile Augmented Reality (MAR). In

addition, this research shows what practical and enjoyable mobile applications can be made

with MAR. A sample application has been developed for demonstration. The general aim of

this sample application is to develop a room number recognition system using Augmented

Reality technology. The work has demonstrated that an Android mobile phone equipped with

this sample application can overlay augmented room schedule onto the mobile screen. Several

experiments were carried out to evaluate the application. Recognition rate is an average of 91%

in continuous real time testing. The application is also tested for varying viewing angles as

well as different distances between the hand-held device and the targets to be tracked. Some

cases of failure have been identified and shown. Future work and results of an evaluation are

also discussed.

Contents

1 Introduction .............................................................................................................. 1 1.1 Definition ........................................................................................................................ 1 1.2 Objective ......................................................................................................................... 2

2 Related works ........................................................................................................... 3 2.1 Previous research ............................................................................................................. 3 2.2 Current AR based mobile applications ............................................................................ 5

2.2.1 Location-aware applications ................................................................................ 5 2.2.2 Context-aware applications .................................................................................. 6

3 Development of MAR demonstrator ...................................................................... 7 3.1 Overview ......................................................................................................................... 7 3.2 Implementation ................................................................................................................ 8 3.3 Experimental Results ..................................................................................................... 10

4 Discussion ............................................................................................................... 11 4.1 Hardware ....................................................................................................................... 13 4.2 Software ........................................................................................................................ 13 4.3 Privacy issues ................................................................................................................ 13

5 Conclusion .............................................................................................................. 14

6 Future work ............................................................................................................ 14

Acknowledgement ...................................................................................................... 15

Bibliography ............................................................................................................... 16

Appendix 1: Usability Survey for Further Evaluation ........................................... 19

Appendix 2: List of figures ........................................................................................ 21

Appendix 3: List of tables.......................................................................................... 22

Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality

1

1 Introduction

Augmented Reality (AR) technology has recently been a subject of much interest, and

has being used widely on PC such as 3D-game playing and information searching.

But now, thanks to the development of mobile hardware, this technology appears

destined for the mobile market, brought AR into everyday life.

1.1 Definition

There are several definitions in the literature for Augmented Reality. In 1994, Paul

Milgram and Fumio Kishino [1] provide taxonomy of mixed reality systems that

include Virtual Reality (VR) and Augmented Reality (AR) apart from Augmented

Virtuality (AV). Just a few years later, in 1997, Ronald Azuma [2] defines Augmented

Reality to be a technology that combines real environment surrounding with virtual

objects. Azuma notes that Augmented Reality shares three common characteristics:

Combines virtual characters with the actual world

Interactive in real time

Registered in 3D

The 2011 annual Horizon Report [3] published by the New Media Consortium

(NMC) describes Augmented Reality as the following:

“Augmented reality (AR) refers to the addition of a computer-assisted contextual

layer of information over the real world, creating a reality that is enhanced or

augmented.”

Figure 1. Milgram’s definition of virtuality continuum.

According to Milgram‟s definition of virtuality continuum [1](Figure 1), Mixed

Reality (MR) environments are those in which real world and virtual world objects are

presented together on a single display. Virtual Reality (VR) simulates physical

presence in places in the real world with computer-simulated environments.

Augmented Virtuality is a term that merges of actual world objects into virtual worlds.

Unlike Virtual Reality creating a simulation of reality and Augmented Virtuality using

virtual worlds replace reality, Augmented Reality enhance the user experience by

projecting the virtual objects onto the observer‟s perception. In other words, with

Augmented Reality, virtual objects only supplement the real world, the real world

objects are still in the real world.

Combined with handheld displays, cameras, acceleration, compass, GPS sensors,

Mobile Augmented Reality has been one of the fastest growing research areas in

Augmented Reality, partially due to blending virtual imaging into the video stream of

a mobile device's camera in real-time and ubiquitous platforms for supporting MAR.

The MAR technology can be applied widely, such as in navigation, situational

awareness, and geography located information. Applications in practice include a

Mixed Reality

Real Environment

Augmented Reality

Augmented Virtuality

Virtual Environment


2

university information system [4] [5], shooting game [6] and car finder with GPS [7],

and so on.

Based on how augmented information is aligned with actual world, there are two

primary types of MAR implementations: Markerless and Marker-Based.

The key difference in Markerless AR is the method used to place virtual objects

in the user‟s view. Markerless AR typically uses the GPS feature of a smartphone to

locate and interact with AR resources. Markerless AR is often more reliant on the

capabilities of the device being used such as the GPS location, velocity meter, etc.

With Markerless AR, no special markers are required. The software is able to

recognize natural features, and more complex visual information. Aside from

expanding the scope of what can be recognized and in turn augmented, it also opens

up increased opportunities to create more natural interactions between digital and

physical world elements which have the potential for a wide variety of applications.

However, tracking and registration techniques are much more complex in Markerless

AR. Drawbacks of Markerless AR including real/virtual image discrepancy, system

calibration, or registration problems such as observers fail to adjust their perceptions

have to be overcome.

When location data isn‟t used, a marker is often used. Traditional markers, for

example, data matrix codes and QR codes, need special black and white regions or

codes to be recognized while Image Targets do not. However, in order to have more

features to be found, high local contrast is needed. A Marker-based implementation

utilizes the traditional marker (see Figure 3), such as QR Code/2D barcode to produce

a result when it is sensed by a reader, typically a camera mounted on the mobile

screen. Unlike traditional markers, Image Targets do not need special black and white

regions to be recognized. The Vuforia SDK [8] uses sophisticated algorithms to detect

and track the natural features that are analyzed in the target image itself. It recognizes

the Image Target by comparing these natural features against a known target resource

database. Once the Image Target is detected, the SDK will store these extracted

features in a database and then compared at run-time with features in the live camera

image. Targets that are accurately detected should be rich in detail, have good contrast,

must be generally well lit and not dull in brightness or color and should not have

repetitive patterns. Figure 2 is an example of a bad image target that has repeated

pattern that cannot be distinguished by the detector, small yellow crosses that appear

around the tree shows the features can be used to match images.

Figure 2. An example of a bad image target.

Marker-based AR is generally easier to build thus it becomes more prevalent in

the mobile application market.

1.2 Objective

The objective of this research is to provide an overview of the previous research as

well as the current applications in the field of Mobile Augmented Reality. A sample

application for the Android platform is developed using AR technology for


3

demonstration and evaluation. The general aim of this sample application is to develop

a room recognition system to act as a room schedule viewer for University of Gavle.

The digits recognition is achieved by OCR (optical character recognition). In

particular, this sample application is context-aware and is able to tell both teachers and

students whether the room is free or occupied by loading schedules for the specific

room published in KronoX. This also allows a comparison between an AR based

application and a non-AR based application.

2 Related works

2.1 Previous research

In 1995, Jun Rekimoto and Katashi Nagao started to experiment with handheld AR

based on the color codes detection method [9]. One year later, Rekimoto [10]

proposed his new method using a 2D matrix barcode (see Figure 3) as landmarks for

both object identification and registration when producing an Augmented Reality

system. It was one of the first marker systems to enable the use of a 2D matrix barcode

to identify a large number of objects.

Figure 3. 2D matrix marker.

The first outdoor Mobile Augmented Reality System (MARS), Touring Machine

(see Figure 4) [11] was presented by Feiner et al. in 1997 featured tracking in a

handheld display with touchpad interface. The system observed the environment

through an HMD (Head Mounted Display), and showed additional information such as

the names of buildings, historical events at the observed locations.

Figure 4. Touring Machine [12].

Five years later, authors demonstrated an AR restaurant guide on their Touring

Machine. Information such as reviews, menus, and website for nearby restaurants

were overlaid onto the display [13] [14] .

ARToolKit [15] is a free and open source library under GPL license for

developing AR applications. Similar to Rekimoto‟s method, it is a flexible pose

tracking library allowing six degrees of freedom (6DoF) movement in 3D space. That

is, controllers are free to move in 3-dimensional direction. The system uses a template


4

matching approach and square markers for object recognition, and achieves AR using

a video or webcam. The project was launched in 1999 and is still very popular among

AR developers. A number of well-known AR libraries, including ARTag [16], AndAR

[17], NyARToolkit [18] and FLARToolKit [19], are inspired by ARToolkit. Wagner

and Barakonyi [20] created the first self-contained AR marker tracking system on a

PDA using ARToolKit.

In the same year, Höllerer et al. [21] presented their MAR system that allowed

visitors to experience Augmented Reality outdoors and receive a guided campus tour

with additional 3D overlay on top of the real world. This was the first MAR system to

use RTK GPS and an inertial-magnetic orientation tracker.

In 2000, Bruce Thomas et al. presented ARQuake [22], an extension to the

popular desktop game Quake, and showed how it could be used for outdoor gameplay.

ARQuake was a first-person perspective application based on a 6DOF tracking system

[23]. Users equipped a HMD on the head, and a wearable computer, ran around in the

real world whilst playing a game in the virtual world. The game could be played in- or

outdoors where the usual keyboard and mouse commands for movements and actions

were performed by movements of the user in the real environment and using the

simple input interface. ARQuake was the first fully working Augmented Reality game

created for outdoor use.

CyberCode [24], introduced in 2000, was a conventional tagging system which

allowed the computer to track position and orientation in three dimensions space.

System was based on the 2D barcode technology and could be used to determine the

3D position of the tagged objects.

Newman et al. presented the Bat system [25], a PDA-based MAR system in the

year 2001. Bat system used two different ways to build an indoor AR system. The first

method used an HMD connected to a laptop. The second method used a PDA with a

fixed configuration to provide a convenient portal with which the user can quickly

viewed the augmented information on the PDA‟s screen.

Reitmayr et al. [26] introduced a mobile collaborative augmented reality system

in 2001. The ideas of mobile computing and collaborative augmented reality were

combined and merged into a single augmented system. The communication between

users was done through LAN and wireless LAN (WLAN).

In 2008, Mobilizy launched Wikitude World Browser [4], an application that

combined position data with Wikipedia entries. In 2011, Wikitude announced the

Wikitude ARchitect Engine at the ARE conference in Silicon Valley [27]. ARchitect

is very flexible; developers are allowed to engage directly with their users on their

terms. A similar step up, Layar [5], works by retrieving digital information over a

network connection and augmented onto top of the screen with a combination of

devices‟ built-in sensors.

Sony is doing a project called SmartAR [28] which was published in 2011. It is a

great experimental demonstration of how the technology is evolving with Markerless

AR. According to Sony, SmartAR (see Figure 5) makes use of the Markerless

approach by combining the traditional object recognition technology with their

proprietary probabilistic matching method in the 3D space. With this method, the 3D

overlay will be retrieved from the database with minimal calculations when

recognizing an object from actual world.


5

Figure 5. Sony develops “SmartAR” integrated AR technology.

2.2 Current AR based mobile applications

Mobile phones have becoming a part of almost everyone‟s lives recently. Indeed, AR

based mobile applications have made way to a lot of comfort and convenience to a lot

of mobile phone users already, partially due to people always want to know all the

details about their interests. Such applications cover a wide variety of technologies,

devices and goals.

I‟ve grouped the AR based mobile applications into two categories, location-

aware applications and context-aware applications, based on its key features.

2.2.1 Location-aware applications

Nowadays, location-aware technologies, like assisted GPS satellite service, video

camera, and digital compass allow people to share their real-time location with friends

and family on their mobile devices [29] [30].

Current MAR systems typically use the devices‟ built-in sensors, like camera,

compass and accelerometer to determine the direction of view, working with GPS

tracking service or Network Location Provider, to assign the augmented information

to the obtained location coordinates. The location information can be obtained from

GPS service or Network Location Provider. However, GPS is limited to outdoor use

only while the Network Location Provider would be suitable for both indoor and

outdoor use. Here, camera captures the world as seen through its lens and shows it on

the screen; GPS and Network Location Provider obtain the observer‟s location and

compass and accelerometer determine the field of view.

2.2.1.1 How it works

It is a simple process that involves four basic tasks:

#Step one: Obtaining user location

Figure 6. Obtaining user location.

Return The Location

Acquire User

Location

GPS/

Location

Provider

Mobile

devices


6

#Step two: Request/receive observed location information from ISP (Information

Service Provider)

Figure 7. Request/receive observed location information from ISP.

#Step three: Determine the lens direction using built-in sensors (e.g. gravity

sensor, digital compass).

#Step four: The camera captures the world as seen through its lens and shows the

receiving updates about users‟ surroundings on the screen.

Figure 8. Combiner.

While MAR is emerging, it is currently rather limited. Most of these Location-

based applications rely heavily on GPS Services. But GPS itself is limited due to

many factors - accuracy, usually between ten and thirty meters, only works outdoor.

Besides the GPS limitation, mobile network requirements, the limited battery power,

as well as camera preview callback memory issues can affect the usefulness of these

location based application.

Another important issue to think about is the ability to continue operation in case

the communication with the tracking system is temporarily lost.

Location-aware applications might touch upon privacy issues, since it allows

people to check where a person is without consent. The problems (e.g. privacy issues)

are discussed in a later section.

2.2.2 Context-aware applications

Context-aware application can be considered as a complement to location-aware

application since context can be applied more flexibly with mobile computing with

any moving entities. For instance, wireless subscribers can provide real-time

information about observers‟ surroundings, overlay information about locations,

surrounding buildings and friends who might be nearby, and these information is

stored and shared with others via online services (see Figure 9).

These activities result in detailed maps and allow additional information that

otherwise cannot be identified by visual perception to be displayed on the mobile

devices. Objects captured by camera can be quickly recognized and tracked at high-

speed along with the movement of the camera, as it is displayed over the actual 3D

space [28].

Receive Data from

ISP

Send a Request to

ISP

Mobile Device

Information Provider

Location Provider

Mobile

devices

ISP


7

Figure 9. Looking at the world through AR [31].

When location data isn‟t used, a marker is often required. Marker-based AR

applications are the most common in the AR market. According to Azuma‟s definition

of AR, the ability of interacting in real time is important since it makes the AR not as

simple as an information rendering tool. But it is in terms of increasing reality in real

time where the prospects seem the most impressive. AR based mobile applications

built with Vuforia SDK [8] must have a known target dataset and good features that

can be used to match images in the camera view. In some cases registration problems

might happen since real scene coordinate system and virtual object coordinate system

should be aligned with each other. Registration problems can cause observers fail to

adjust their perceptions.

Perhaps this is the future to wear glasses. The idea of eyewear has been put into

practice by Vuzixs who offers glasses Wrap 920AR [32] to display 3D images in the

real world. Similar to Vuzixs, Nokia was showing off its own alternative to use

camera supported mobile phones as the platform for sensor-based, video see-through

mobile augmented reality [33]. At the beginning of the year 2012, Google has

announced Project AR Glass [34] [35].

3 Development of MAR demonstrator

Imagine a mobile phone is a lens into the real world that shows the schedule at the

University of Gävle by displaying virtual information on the top of reality – users are

allowed seeing that with the real world environment right in front of them. This is the

key of Mobile Augmented Reality and also is what I am trying to illustrate.

3.1 Overview

The project is developed using Eclipse with a plugin called ADT (Android

Development Tools). Eclipse is appreciated since it can directly invoke the tools that

needed while developing applications. The Tesseract OCR Engine [36], the ZXing [37]

library for image processing, Android SDK, NDK (a companion tool to the Android

SDK that allows building application in native code), Cygwin and Git are also used to

compile the project. The application of the framework is implemented in Java

language (Oracle‟s JDK7u4).

Environmental information will be obtained via mobile camera. The video of the

real world surroundings and the images are combined, and then displayed on the

screen. Figure 10 shows the concept of how the system is built.


8

Figure 10. The conceptual diagram of Mobile based AR.

3.2 Implementation

A procedure to carry out the task is as follows in Figure 11. The system comprises of

three subparts, Scene generator, Virtual objects generator and the Combiner.

Figure 11. System architecture.

The details of the procedures are described below:

#Step one: Real world generator

The user is required to predefine the position of the number plate using a

bounding box and then the application will automatically inspect the room number

within the given area in order to recognize the room number. Camera data is provided

by the android framework through the Camera API and Camera Intent. The common

method to custom a surface view where the live camera preview will be loaded is

using the class SurfaceView.

In order to be more feasible and effective, I set up a target using model-based

recognition. In the campus of University of Gävle, room numbers, composed by five

digits, together with “:”, with a clear plastic cover are printed on the wall. If only one

letter (A-Ö) is detected, it is rejected since it is not part of target string and thus cannot

be a room number. A whitelist “0123456789:” is created for rejection. The use of

whitelist can simplify the digit recognition and improve quality of recognition while

achieving good performance. The room number model is shown in Figure 12.

User

Combiner

Android device video camera on

mobile device

Scene

generator

Tracker

Room Number

Recognition

Scheme

information

video of real world

surroundings

Real world

generator

• Prepping the camera

• OCR by running the Tesseract OCR engine to recognize room number in images captured by the device's camera.

Virtual objects

generator • Using web services(KronoX)

Combiner • Achieve a real-time display


9

Figure 12. Room number model.

As the images differ from each other, it is not possible to use a static global

adaptive threshold because there could be a certain amount of loss of information.

Thus, image pre-processing is required as a prerequisite to the OCR. The Tesseract

OCR Engine consists of a main program, a word level recognizer, a low-level

character classifier and one or more dictionaries. The word level recognizer is a

module that organizes text into lines and words. The character classifier works with

dictionaries to classify the words. The digit recognition is achieved by a numbers of

outline comparisons between characters for separation, and then applying a Prototype-

Feature-Matching process on these extracted outline fragments. Each scaled character

will be matched once with each template, and the one with the highest confidence will

be the recognition result. However, even the highest confidence can still very low

sometime. The last step is a combination of broken characters. In order to identify the

room number on the Android platform, Android APIs for the Tesseract library are

used [38] [39], which provides a set of Android APIs for accessing natively-compiled

Tesseract OCR libraries. However, some failure of character recognition may happen;

this will be discussed in the later section.

#Step two: Virtual objects generator

Since the augmented information is the schedule website, the processing of

virtual objects generator can be regarded as the processing of link generator which is a

made up of fixed URL and recognized the room number ( Figure 13 ).

Figure 13. Link generator.

#Step three: Combiner

The final step is to bring the camera view and augmented schedule together onto

the observer‟s perception. This step is a process of registration which is vital for the

AR. AR requires accurate registration of virtual objects in order to render them into

the actual world. As mentioned before, the magic of AR is the idea that drawing

something on top of it. For mobile devices, this process is as simple as draw

something over the camera and can be accomplished by various ways. In this

application, the magic is done by adding two views to the selected layer. In order to

achieve a real-time display, the WebView class is used to ensure recognition result

and web page is shown at the same time.

Schedule Link

Fixed URL Room number

http://code.google.com/p/tesseract-android-tools/


10

The running result is shown in Figure 14. The TextView (blue) displays the

room number that interested, the WebView showing the activities for the room.

Figure 14. Running result.

3.3 Experimental Results

I have conducted experiments in order to examine the performance of the application

under varying conditions and to validate the feasibility of AR based room recognition

system in real time. The experiments were carried out under a natural light condition

in the building 99, University of Gävle. These experimental results are only in case of

recognizing specific room number „99:518‟.

The system was tested in real time and gave reliable room recognition. The major

achievement of this work is the recognition rate of correct identification, which is 91%

in continuous real time testing. The recognition rate of correct identification varied

while capturing images in the different angle of view and in the different distance

between observer and door.

The application is tested on over the different angle of view as well as the

different distance between observer and door. While capturing these images, the angle

of view and the distance between observer and door varied according to the

experimental setup. These images were subjected to pre-processing which comprises

of some standard image processing algorithms. The resultant images were then fed to

the proposed OCR system.

The first experiment measures the correct identification while capturing images

in the different angle of view. The result is shown in Table 1 and Figure 15. In Table 1,

recognition rate refers the number of the successful rate in which the application can

recognize room number correctly. The failure is considered to have been caused by

errors in the camera orientation. So the angle of view is crucial for room number

recognition.

Table 1. Recognition rate according to different angle of view.

(deg)


11

Figure 15. Recognition rate according to different angle of view.

The second experiment measures the correct identification while capturing

images in the different distance between observer and door. The result is shown in

Table 2 and Figure 16. In Table 2, recognition rate is the number of the successful rate

in which the room numbers were recognized correctly. As the results suggested,

normally when the distance between observer and door was bigger than 70 cm, the

digits were too small to be recognized. The recommend distance of the observer to the

door should be between 10 and 30 cm, not more than 50 cm.

Table 2. Recognition rate according to different distance.

93 83 41

Figure 16. Recognition rate according to different distance.

4 Discussion

From the experiment results, we can see that the application gives no reliable

recognition result if room number is distorted by perspective projection when the

angle of view θ is small or the distance of the observer to the door is large. False

recognition caused by errors in the angle of view is most likely due to a lack of

resolution in depth. Thus lower accuracy is often unavoidable. Any image taken at a

slanted angle will thus affect the recognition. With an increase of angle of view, the

0

20

40

60

80

100

40 50 60 90

recognition rate (%)

θ(deg)

θ(deg)

θ(deg)

0

20

40

60

80

100

10 30 50 70

recognition rate (%)

distance(cm)

distance(cm)

distance(cm)


12

image will become distorted. This problem is hard to handle because it very much

depends on the camera's depth of field and focus settings. The problem is even more

serious here since the plastic covering on the room number plate can cause reflections.

In Figure 15, there is a sudden drop between 50 to 60 degrees, which is mainly caused

by reflections. In this area, the amount of light reflected increases as the angle of view

increases. While the room numbers are at a quite distance from the user, digits are too

small and can be easily wrongly recognized. To overcome this, an alternate approach

is to use the camera phone featuring a high zoom ratio.

Besides, the recognition of room number is also affected by factors such as the

ambiguity of an image, motion blur effect and brightness level. Ambiguity makes an

image can easily be wrongly recognized (Figure 17). Motion blur (Figure 18) is

partially due to hand shaking. Blurred or ghostly images make it difficult to perform

recognition on. Handle motion blur from both hand shaking and object moving can be

minimized through several techniques such as separating individual frames from a

video stream.

Figure 17. The ambiguous problem.

Figure 18. The motion blur problem.

Brightness level is another crucial problem in room number recognition. Many

other problems are more or less subdued by its lack of suitable light. The experiments

are carried out under a natural light condition, but perhaps lighting conditions in the

surrounding still not enough. Low lighting conditions causes the input image to be

unclear and consists mainly of dark pixels which seriously affect the recognition. In

our case, local contrast should not one of the major problems that limiting the

recognition rate since the sample application only handle the black room numbers

printed on the white wall. However, when the angle of view θ is small or observer and

door is quite a distance away, room numbers are not be distinct enough for accurate

recognition. An input image has high local contrast features and yield easier to be


13

recognized. The higher contrast the input image has, the better OCR engine works.

Without a good local contrast, the shape and outline of each digit will be ambiguous.

Given all above problems, it is impossible to achieve 100% recognition rate. In

order to minimize the effects of above problems, an appropriate image pre-processing

could be applied on the original image before being processed by the OCR engine.

The most common techniques used to improve the brightness and contrast of an image

is histogram equalization where one attempts to map one intensity distribution of the

original image histogram to another wider and more uniform distribution so that the

intensity values can spread over the whole intensity range. This is done by distributing

the intensities of the input image from dark to bright, for example, the number of

pixels for each luminous intensity. After these improvements are applied, the problem

when the distributions of the original histogram are very close to each other could be

solved. All in all, lighting conditions should be carefully controlled to avoid

reflections and shadows.

As described in chapter 3.2, the Tesseract OCR engine verifies the recognized

number by voting the recognition result, the one with the highest confidence is

considered as a correct room number. In the experiment, the highest confidence is still

very low due to errors in the angle of view as well as the distance of the observer to

the door, which leads an increasing of false recognition rate. So it is risky to use the

highest confidence result without any filtering. Rejection is an alternative to evaluate

the reliability of the resultant data. This can be done by setting a threshold value for

the acceptable highest confidence result - if the highest confidence is lower than

certain threshold value, then it is rejected and will not assume to be a correct room

number. In this way, the false recognition rate could be reduced.

4.1 Hardware

The sample application is compatible with Android 2.2 and up. It doesn‟t working on

the older Android OS such as Android 2.1 and Android 1.6. This is because the

application uses YUV image format API to speed up compression and decoding.

The YUV data only supports Android 2.2 and up.

Most Augmented reality applications require the image processing which is

generally very computationally expensive. What‟s more, there is a limitation of

memory allocation for each process max memory use, it is normally 24 MB, but on

some older devices, the limit is even lower at 16 MB. The memory used by Bitmaps is

also included in the limit. When developing an AR application, it is pretty easy to

reach this limit and get the OutOfMemoryError exception. The problem is finally

solved by allocating memory from native code using the NDK. However, as new

Android OS get released, users will probably see more AR applications with cool

features.

4.2 Software

In the sample application, text drawings might suffice which means rendering is not

currently one of the critical problems. However, Mobile AR systems must be worn,

which challenges developers to minimize weight and maintain performance. One

disadvantage of augmented reality applications is the information overloading. This

occurs when too much information is offered than the user is able to process.

Reducing information overloading can be complicated; one approach is to reduce the

rate at which new updates appear by specifying an interval time.

4.3 Privacy issues

The sample application is context-aware and does not require any extra permission

(besides internet access permission and the use of camera permission). But usually,

most AR-based applications have access to much permissions, such as allow hardware


14

controls(take pictures and videos), full internet access, coarse (network-based)

location, fine (GPS) location, modify and delete USB/SD card storage contents, and

view and change network/Wi-Fi state [4] .

Possible risks include:

enabling the GPS and then streaming those feeds to a remote server

sending email/SMS/MMS(services that cost you money) without user‟s

confirmation

read personal information and send these data to other people

linking to unsafe websites

These application and services may run in the background and users didn't even

realize they are still running.

5 Conclusion

In this paper, I have presented an analysis on the current technical details and the

practical use in the field of Mobile Augmented Reality. Mobile Augmented Reality

technology offers very unique experiences for users to interact with surroundings, and

the use of mobile phones with MAR allows these experiences to be extended with

visibility. Moreover, I have developed a sample application that can simultaneously

recognize room number by means of optical character recognition using Mobile

Augmented Reality technology. The MAR demonstrator achieves the task that

superimposes augmented schedule information onto the mobile screen. Several

experiments were carried out to evaluate the application. A few problems in room

number recognition have been identified, including varying viewing angles and

different distances between observer and door. The experimental results showed room

numbers can be correctly recognized by using the MAR demonstrator. The recognition

rate was an average of 91% in continuous real time testing. This has been proved

through several experiments.

Despite their limited computation power, I think mobile device is the perfect

platform for who want a truly immersive augmented reality experience. Most of

mobile devices are equipped with a digital camera on the back, which provides a

nature way to achieve see-through effect at affordable cost.

6 Future work

According to the three common characteristics of Augmented Reality offered by

Azuma, the interaction between users and virtual information play an essential role in

augmented reality. Therefore it is important to evaluate user experience when using an

AR-based application and come up with further improvement.

Later, an experiment could be carried out for evaluation. The purpose of this

experiment should focus on the different user reactions when using an AR based

application and a non-AR based application (Figure 19).

I have offered a Usability Survey (Figure 20) (see Appendix 1) which is a part of

the application. The possibility of future evaluation was considered at the application

design stage.


15

Figure 19. GUI and Non-AR function.

Figure 20. Usability Survey for future evaluation.

Acknowledgement

I would like to thank Peter Jenke for his help of my research project.


16

Bibliography

[1] Paul Milgram and Fumio Kishino, "Taxonomy of Mixed Reality Visual Displays",

IEICE Transactions on Information Systems, pp. 1321-1329, December 1994.

[2] Ronald T. Azuma, "A Survey of Augmented Reality", Presence: Teleoperators and

Virtual Environments, pp. 355-385, August 1997.

[3] The New Media Consortium and the EDUCAUSE Learning initiative, "Two to

Three Years: Augmented Reality", The 2011 NMC Horizon Report, pp. 16-20,

February 2011.

[4] Wikitude GmbH. (2012, March) Wikitude World Browser. [Online].

https://play.google.com/store/apps/details?id=com.wikitude&hl=en

[5] Layar. Layar. [Online]. http://www.layar.com/

[6] ProjectX. (2012, March) AR Shooting. [Online].

https://play.google.com/store/apps/details?id=com.project_x.ARShooting&hl=sv

[7] Hexter Labs Software. (2012, January ) Car Finder AR. [Online].

https://play.google.com/store/apps/details?id=net.androoid.carxare&feature=search_

result#?t=W251bGwsMSwxLDEsIm5ldC5hbmRyb29pZC5jYXJ4YXJlIl0.

[8] Qualcomm. Augmented Reality (Vuforia) - Qualcomm Developer Network.

[Online]. https://developer.qualcomm.com/develop/mobile-technologies/augmented-

reality

[9] Jun Rekimoto and Katashi Nagao, "The world through the computer: computer

augmented interaction with real world environments", in 8th annual ACM

symposium on User interface and software technology(UIST '95), 1995, pp. 29-36.

[10] Jun Rekimoto, "Matrix: A Realitime Object Identification and Registration Method

for Augmented Reality", in 3rd Asia Pacific Conference on Computer Human

Interaction(APCHI 98), Kanagawa, 1998, pp. 63 - 68.

[11] S. Feiner, B. MacIntyre, T. Höllerer and A. Webster, "A Touring Machine:

Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban

Environment", in 1st International Symposium on Wearable Computers(ISWC '97),

Cambridge, MA, 1997, pp. 74-81.

[12] Steven Feiner, Tobias Höllerer, Elias Gagas, Drexel Hallaway, Tachio Terauchi,

Sinem Güven, and Blair MacIntyre. MARS - Mobile Augmented Reality Systems.

[Online]. http://graphics.cs.columbia.edu/projects/mars/mars.html

[13] B. Bell, S. Feiner, and T. Höllerer, "View Management for Virtual and Augmented

Reality", in UIST '01, Orlando, 2001 , pp. 101-110.

[14] B. Bell, T. Höllerer, and S. Feiner, "An Annotated Situation-Awareness Aid for

Augmented Reality", in UIST '02, Paris, 2002, pp. 213-216.

[15] Hirokazu Kato and Mark Billinghurst, "Marker tracking and HMD calibration for a

video-based augmented reality conferencing system", in 2nd International Workshop

on Augmented Reality (IWAR'99), San Francisco, 1999, pp. 85-94.

[16] Mark Fiala. (1999, February ) ARTag. [Online]. http://www.artag.net/

[17] ARToolWorks. andar - AndAR - Android Augmented Reality - Google Project

Hosting. [Online]. http://code.google.com/p/andar/

[18] ARToolWorks. (2008, May) NyARToolkit project. [Online].

http://nyatla.jp/nyartoolkit/wp/

[19] Saqoosha amd nyatla. (2008, october) saqoosha/FLARToolKit/en. [Online].

https://play.google.com/store/apps/details?id=com.wikitude&hl=en

http://www.layar.com/

https://play.google.com/store/apps/details?id=com.project_x.ARShooting&hl=sv

https://play.google.com/store/apps/details?id=net.androoid.carxare&feature=search_result#?t=W251bGwsMSwxLDEsIm5ldC5hbmRyb29pZC5jYXJ4YXJlIl0.

https://play.google.com/store/apps/details?id=net.androoid.carxare&feature=search_result#?t=W251bGwsMSwxLDEsIm5ldC5hbmRyb29pZC5jYXJ4YXJlIl0.

https://developer.qualcomm.com/develop/mobile-technologies/augmented-reality

https://developer.qualcomm.com/develop/mobile-technologies/augmented-reality

http://graphics.cs.columbia.edu/projects/mars/mars.html

http://www.artag.net/

http://code.google.com/p/andar/

http://nyatla.jp/nyartoolkit/wp/


17

http://www.artoolworks.com/products/web/flartoolkit-2/

[20] Daniel Wagner and Istvan Barakonyi, "Augmented Reality Kanji Learning", in 2nd

International Symposium on Mixed and Augmented Reality(ISMAR 2003), Tokyo,

2003, pp. 335-336.

[21] T. Höllerer, S. Feiner, and J. Pavlik, "Situated documentaries: Embedding

multimedia presentations in the real world", in 3rd International Symposium on

Wearable Computers(ISWC'99), New York, 1999, pp. 79-86.

[22] Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., and

Piekarski, W, "ARQuake: An Outdoor/Indoor Augmented Reality First Person

Application", in 4th International Symposium on Wearable Computers(ISWC'10),

October 2010, pp. 139-146.

[23] J. Park, B.Jiang, and U.Neumann, "Vision-based pose computation: Robust and

accurate augmented reality tracking", in 2nd IEEE and ACM International Workshop

on Augmented Reality (IWAR '99), San Francisco, California, 1999, pp. 3-12.

[24] Jun Rekimoto and Yuji Ayatsuka, "CyberCode: designing augmented reality

environments with visual tags", in ACM Conference on Designing Augmented

Reality Environments (DARE 2000), New York, 2000, pp. 1-10.

[25] J. Newman, D. Ingram and A. Hopper, "Augmented Reality in a Wide Area Sentient

Environment", in 2nd IEEE and ACM International Symposium on Augmented

Reality (ISAR 2001), New York, 2001, pp. 77-86.

[26] G. Reitmayr, and D. Schmalstieg, "Mobile Collaborative Augmented Reality", in

2nd International Symposium on Augmented Reality (ISAR 2001), New York, 2001,

pp. 114-123.

[27] Wikitude. (2011, September ) Wikitude reveals the ARchitect Developer Kit.

[Online]. http://www.wikitude.com/enwikitude-reveals-architect-developer-kit

[28] Sony Corporation. (2011, May) Sony develops “SmartAR” Integrated Augmented

Reality technology. [Online]. http://www.sony.net/SonyInfo/News/Press/201105/11-

058E/index.html

[29] G. Takacs, V. Chandrasekhar, S. Tsai, D. Chen, R. Grzeszczuk, and B.Girod,

"Unified real-time tracking and recognition with rotation-invariant fast features", in

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010,

pp. 934-941.

[30] D. Chen, S. Tsai, R. Vedantham, R. Grzeszczuk, and B. Girod, "Streaming mobile

augmented reality on mobile phones", in 8th International Symposium on Mixed and

Augmented Reality (ISMAR 2009), Orlando, October 2009, pp. 181-182.

[31] The Big Idea: Augmented Reality. [Online]. http://ngm.nationalgeographic.com/big-

idea/14/augmented-reality

[32] Vuzix Corporation. (2010, October) Wrap 920AR. [Online].

http://www.vuzix.com/consumer/products_wrap920ar.html

[33] Nokia. (2009) Nokia Mixed Reality. [Online]. http://research.nokia.com/page/695

[34] Bilton Nick. (2012, April) Google Begins Testing Its Augmented-Reality Glasses -

NYTimes.com. [Online]. http://bits.blogs.nytimes.com/2012/04/04/google-begins-

testing-its-augmented-reality-glasses/

[35] NICK BILTON. (2012, February) Google to Sell Heads-Up Display Glasses by

Year‟s End. [Online]. http://bits.blogs.nytimes.com/2012/02/21/google-to-sell-

terminator-style-glasses-by-years-end/

http://www.artoolworks.com/products/web/flartoolkit-2/

http://www.wikitude.com/enwikitude-reveals-architect-developer-kit

http://www.sony.net/SonyInfo/News/Press/201105/11-058E/index.html

http://www.sony.net/SonyInfo/News/Press/201105/11-058E/index.html

http://ngm.nationalgeographic.com/big-idea/14/augmented-reality

http://ngm.nationalgeographic.com/big-idea/14/augmented-reality

http://www.vuzix.com/consumer/products_wrap920ar.html

http://research.nokia.com/page/695

http://bits.blogs.nytimes.com/2012/04/04/google-begins-testing-its-augmented-reality-glasses/

http://bits.blogs.nytimes.com/2012/04/04/google-begins-testing-its-augmented-reality-glasses/

http://bits.blogs.nytimes.com/2012/02/21/google-to-sell-terminator-style-glasses-by-years-end/

http://bits.blogs.nytimes.com/2012/02/21/google-to-sell-terminator-style-glasses-by-years-end/


18

[36] Google. (1995) The Tesseract OCR engine. [Online].

http://code.google.com/p/tesseract-ocr/

[37] ZXing. (2007, November) zxing - Multi-format 1D-2D barcode image processing

library with clients for Android, Java. [Online]. http://code.google.com/p/zxing/

[38] Alan Viverette. Android APIs for the Tesseract OCR library. [Online].


[39] Robert Theis. (2011) Tess-two. [Online]. https://github.com/rmtheis/tess-two

http://code.google.com/p/tesseract-ocr/

http://code.google.com/p/zxing/


https://github.com/rmtheis/tess-two


19

Appendix 1: Usability Survey for Further Evaluation

Has the application run successfully? * 1.

a. Yes

b. No

Do you find the information you are looking for (room scheduling)? * 2.

a. Yes

b. No

Please state any difficulties you had when using the application, if any. 3.

a. can't connect to the internet

b. can't recognize the room number correctly

c. don't know how to use this application

d. other

Compared with non-AR based application I have used, I found an AR based 4.

application is* (concerning the OPERABILITY)

a. easier to use

b. about average

c. more difficult to use


application is* (concerning the EFFECTIVENESS)

a. take more time to find the information

b. take the same time to find the information

c. take less time to find the information


application is* (concerning the USEFULNESS)

a. more helpful

b. about the same

c. less helpful

Have you ever used any AR based application on your mobile phone? * 7.

a. Yes

b. No


20

c. Never Heard of

Any comment? 8.


21

Appendix 2: List of figures

Figure 1. Milgram‟s definition of virtuality continuum. ---------------------------------- 1

Figure 2. An example of a bad image target. ----------------------------------------------- 2

Figure 3. 2D matrix marker. ------------------------------------------------------------------- 3

Figure 4. Touring Machine [12]. ------------------------------------------------------------- 3

Figure 5. Sony develops “SmartAR” integrated AR technology. ------------------------ 5

Figure 6. Obtaining user location. ------------------------------------------------------------ 5

Figure 7. Request/receive observed location information from ISP. -------------------- 6

Figure 8. Combiner. ---------------------------------------------------------------------------- 6

Figure 9. Looking at the world through AR [31]. ------------------------------------------ 7

Figure 10. The conceptual diagram of Mobile based AR. -------------------------------- 8

Figure 11. System architecture. --------------------------------------------------------------- 8

Figure 12. Room number model. ------------------------------------------------------------- 9

Figure 13. Link generator. --------------------------------------------------------------------- 9

Figure 14. Running result. -------------------------------------------------------------------- 10

Figure 15. Recognition rate according to different angle of view.---------------------- 11

Figure 16. Recognition rate according to different distance. ---------------------------- 11

Figure 17. The ambiguous problem. -------------------------------------------------------- 12

Figure 18. The motion blur problem. ------------------------------------------------------- 12

Figure 19. GUI and Non-AR function. ----------------------------------------------------- 15

Figure 20. Usability Survey for future evaluation. ---------------------------------------- 15


22

Appendix 3: List of tables

Table 1. Recognition rate according to different angle of view. ................................ 10

Table 2. Recognition rate according to different distance. ........................................ 11

mobile room schedule viewer using augmented reality

Documents