MOBILE PHONE AUGMENTED REALITY BUSINESS CARD

TEO TZONG REN

'I'bis prroject is submitted in partial fülfilmcnt of the requirements für a Kachelur oof. Science with honours

(('ogniti\e Sciences)

Faculty of ('ognitivc Sciences and I luman Development l! NIVI: RSI"fl MALAYSIA SARAWAK

(2010)

ACKNOWLEGEMENT

First and fbremost I offer my sincerest gratitude to my supervisor, Dr. Edmund Ng Giap Weng, who has supported me throughout my project with his patience and knowledge whilst allowing me the room to work in my own way. It would have been next to impossible to complete this project without his help and guidance. I would also like to thank Md. Abdullah Al Jubair, who assisted me in developing the project's application. Next, I would like to thank my parents fir

supporting me throughout all my studies at University. Lastly, I offer my regards and blessings to all of those who had supported me in any respect throughout this

project's completion.

III

TABLE OF CONTENTS

Acknowledgement Table of Contents List of Figures List of Tables Abstract Ahstruk

Chapter I- INTRODU('TION

1.0 Overview 1.1 Background 1.2 Problem Statement 1.3 Objective

1.3.1 General Objective 1.3.2 Specific Objectives

1.4 Importance of'Research 1.5 Project Scope 1.6 Values ot'Study 1.7 Significance of'Study 1.8 Structure of Project 1.9 Conclusion

Chapter 2- LI'T'ERATURE REVIEW

Page

iii iv vii x xi xii

i i 3 4 4 4 4 5 5 6 6 7

2.0 Overview K 2.1 Introduction of'Augmented Reality K 2.2 Introduction of'Mobile Phone Augmented Reality IO

2.2.1 S mhian S60 II 2.2

.2 S60Augmented Reality The Map Tracking II

2.2.3 History of'Mobile Augmented Reality 13 2.3 Introduction of'Business Card 15

2.3.1 Most Typical of Business Card Formats 16 2.3.2 Current Research on Augmented Business Card 17

2.4 Mobile 3D Map 18 2.5 Conclusion 19

IV

Chapter 3- METHODOLOGY

3.0 Overview 20 3.1 Application Development Process Model 20

3.1.1 Reuse-Based Model 21 3.2 Application Specifications 23

3.2.1 Hardware Requirements 23 3.2.2 Software Requirements 23

3.2.2.1 ARToolKitPlus S60 23 3.2.2.2 OpenGL ES 25 3.2.2.3 Carbide. c++ 26 3.2.2.4 S60 Platform SDKs for Symbian OS 26

3.3 System Requirements 27 3.4 Application Flow 28 3.5 Conclusion 28

Chapter 4- DESIGN AND DEVELOPMENT

4.0 Overview 29 4.1 Application Architecture 30 4.2 Application Development 31

4.2.1 Application Development's Obstacles 32 4.3 Coding Development 32

4.3.1 Map of'UNIMAS 32 4.3.2 The Building of Faculty of ('ognitive Sciences

and Human Development 34 4.3.3 Navigator Board 36 4.3.4 Keypad Functions 37 4.3.5 Rain FIlcct 39 4.3.6 Play Sound 40

4.4 Conclusion 40

Chapter 5- CONCLUSION

5.0 Overview 41 5.1 Discussion 41 5.2 Application Dchugging 44

5.2.1 Marker Select ion 44 5.2.2 Viewing from Ditlcrcnt Angles, Distances and

Lighting Condition 45 5.3 Strengths oFthe Application 46 5.4 Weaknesses of the Application 46 5.5 Future Recommendation 47 5.6 ('onclusion 48

V

REFERENCES 49

APPENDIX 53

vi

LIST OF FIGURES

Figure 1 The evolution and miniature of mobile AR: (a) Backpack with HMD, (b) UMPC, (c) handheld, (d) Mobile phone 2

Figure 2.1 Map Tracker 13

Figure 2.2 3D markers 13

Figure 2.3 AR-Tennis game 14

Figure 2.4 Visual Codes 14

Figure 2.5 A typical business card 15

Figure 2.6 A virtual avatar on a business card by using mobile-phone-based augmented reality 17

Figure 2.7 A 2D map, a 3D map and it photograph 18

Figure 2.8 Examples of'Mobile 3D maps 19

Figure 3.1 Reuse-Based Development Model 22

Figure 3.2 Level ofextended version for ARToolKit 24

Figure 3.3 Basic workflow of'an AR application using fiducial marker tracking 25

Figure 3.4 Flow of the application 28

Figure 4.1 Application architecture ol'Augm nted Business Card 30

vu

Figure 4.2 Code for texture loading

Figure 4.2.1 Code for texture rendering

Figure 4.2.2 Texture of UNIMAS map

Figure 4.3 Code for FCSHD building

Figure 4.3.1 Code for FCSHD building rotating

Figure 4.3.2 FCSHD textures (Top, Bottom, Front, Back, Lett and Right) and the 3D building

33

33

34

35

35

36

Figure 4.4 Code for Navigator Board 36

Figure 4.4.1 Texture of Navigator Board 37

Figure 4.5 Code fir Keypad Functions 38

Figure 4.5.1 Keypad and buttons ot'Nokia N95 38

Figure 4.6 Code tier rain ct1cct 39

Figure 4.6.1 Scrrcnshut of'raining rficct 40

Figure 4.7 Code iOr Sound etlret 40

Figure 5.1 Marker (business card) detection by mobile phone 42

Figure 5.2 Virtual neap and building displayed on mobile phone's screen 42

viii

Figure 5.3 Raining etlect with sound 43

Figure 5.4 Switch back to normal view of map by terminating the rain rendering 43

Figure 5.5 The "About" menu 43

Figure 5.6 Exit or quit from application 44

Figure 5.7 ARToolKit Marker Generator Online and the example values ofpixels in pattern file format (. pat) 45

Figure 5.8 Angle, distance and lighting condition that caused Mobile Phone (ARTooIKitPlus S60) unable to detect marker accurately 45

Ix

LIST OF TABLES

Table 1 Vary of standard size for Business Cards in several countries 16

Table 2 Hardware Requirements 23

x

ABSRTACT

MOBILE PHONE AUGMENTED REALITY BUSINESS CARD

Teo Tzong Ren

The main idea for this project is to introduce the technology of Mobile Phone Augmented Reality (AR) and develop it into an application for business card. AR technology is rather easier to introduce to the public by using Mobile Phone (Symbian OS) as compared to any other devices. Thus, this project bring up the basic idea of Mobile Phone AR technology and its application on a physical Business Card. Using a typical business card imprinted with a marker that assists in the marker tracking and location identification process, an AR 3D virtual map will be the information displayed on the Mobile Phone upon application execution. Furthermore, lower-end graphic is the challenge ofthis project in order to robust the capability of Mobile Phone tbllowed by enhances efficiency rendering process in real time with AR technology.

ri

ABSTRAK

MOBILE PHONE AUGMENTED REALITY BUSINESS CARD

Teo Tong Ren

7icjuun utunw projek ini uduluh untuk meny, erkenalkan tek, zulogi "Aagmcvned Reality" (AR) tek%n hi, nhit dan memhungunkun sun, aplikasi untuk kud

perniagaan. "1'eknologi AR lehih se. cuui dipe, kenulkan kepada musvurukut nelalui tele/inn hi, nhit (Svmhiun OS). Judi, pr(? jek ini menvtnnhung hiiuun asas tentctng AR tele%n himhit serta upliku. cinvu du/um Kad Perniuguun. 1)engan penggnnuun Kcrc! Pernicrguun yang hercetuk penundu . vertu menvunrhung dahlin proses penj(. jukcur tunda dun penenn, un lokasi. pela 31) , nuvu akan ditc{vung dulcnn

feldoll himhit . cehagui upuhi/u up/ikasi dijulankun. Se/ain inc, gru/ik

yang herkuuliti renduh mencpakun suluh sutu caharan da/am kujiun projek ini hugi nrc, itie. cuuiku, rnvu dengun teleJun himhit, sertu tuna ineningkijikan kelancurun proses. c meluki. c dulum muse nvuta dengun ne, >ggrncakun teknologi AR.

xii

CHAPTER I

INTRODUCTION

l. 0 Overview

This chapter discusses the background of the research, problem statements,

objectives, contribution, importance of research, scopes of the project, value and

significance of'the research as well as the structure of project.

1.1 Background

Augmented Reality (AR) technology, is a growing area that originated

from the Virtual Reality (VR) technology, yet vary from VR where the virtual

objects is superimposed upon or composite in the real world and the users are

interacting with the virtual objects in real-time (Vallino, I99S). AR require 6DOF

pose tracking ot'device such as head-mounted displays (IIMD), tangible intertiºce

object, etc, where the pose tracking must be inexpensive, work robustly and in

I

time through changing environmental conditions (Wagner & Schmalstieg, 2009).

However, the trend of AR devices had switched. This could be seen in figure I

from the first mobile augmented reality system - Backpack with I(MD (Figure

I a) that had been switched to a smaller and lower cost devices - Smart Phone

(Figure ld) (Rosenblum & Julier, 2009).

The term smart /phone was initially coined by unknown marketing

strategists to refer to a then-new class of cell phones that could facilitate data

access and processing with significant computing power (Zheng & Ni, 2006).

According toZhcng and Ni (2006). it smart phone is like it small, networked

computer in the firm ofa cell phone Which usually provides personal information

management (PIM) applications and some wireless communication capacity.

i 0000 -S,

4f iFt la) (b) (c) (d)

Figure 1: The evolution and miniature of mobile AR: (a) Backhack with I IMI),

(h) IJMPC. (c) handheld, (d) Mobile phone. (Wagner & Schmalstieg, 2009)

Wagner and Schmalstieg (2009) stated that smart phones are aiming liºr a

different market in AR compared to the more powerful and larger ultra mobile

PCs (UMPCs. Figure I b). Ile smart phones are surprisingly robust and R olproof

in applications development although its appearance is fragile. Achieving

sufficient performance fir AR applications are therefore require carefiºI choice of

algorithms and optimized code as smart phone has limited processing capabilities

compared to the PC platf6rm (Wagner & Schmalstieg, 2009). Most smart phones

come with built in camera which naturally lend itself to computer vision

approaches. The quality of AR in smart phone is therefore lower than the AR in a

PC platform as the computer vision's quality is based on camera and image sensor

characteristics (Wagner & Schmalstieg, 2009). Moreover, marketing has driven

the development of smart phone to more megapixels rather than higher video

quality (Wagner & Schmalstieg, 2009) which may increase the capability ofsmart

phone in perförming AR application.

In this project, the application of Mobile Phone AR in a typical business

card will he covered.

1.2 Problem Statement

Typical business cards come with aspect ratios of dinmensions range from

1.43 to 1.8. In United State and Canada, the size of a business card is 3.5 inch x

2.0 inch equivalent to the 1.75 aspect ratio (PrintingForLess. com, 2009).

These sizes of business card often includes the giver's name, company

affiliation (with logo) and contact information such as addresses, contact

number(s), e-mail addresses and website. 't'oday, a professional business card also

includes one or more aspects of'striking visual design such as map of'thc address

location.

The standard site of'a typical business card makes it impossible to allºICate

Inure Ill li/rlllatloll clue to the limitation of space results to the Ill lilrlllatloll included

in it business card is unclear or incomplete, hence creating confusion to its viewer.

An example of' such is the map attached in a business card that can sonletinles

confuse its viewer due to Ill furlllatloll of Its local loll Is unclear.

Hence, there is a need to find a way to solve such problem. With the task

at hand, this Project is to design and develop a business card application by using

Mobile Phone AR technology, fir enhancing the limctions of, traditional business

card. The idea of this application is to display clearer information through the

Mobile Phone AR technology within the space limitation in a physical object of

the real world.

1.3 Objective

The objective of the study is divided into two categories which is the

general objective and specific objLc: tives.

1.3.1 General Objective

The aim is to design and develop it Mobile Phone Augmented Reality

application for typical business cards.

1.3.2 Specific Objectives

The specific objectives of this project include:

" To design and develop 31) map using lower level graphics in

Svmbian OS Mobile Phone

" Implement 31) mar into it physical business card by using Mobile

Phone Augmented Reality technology.

1.4 Importance of Research

Despite the technology of Mobile Phone AR is still in an immature state.

the number oI smart phone usage. globally speaking is predicted t� rise up to KOO

million units by the year 20I2 (Rosenblum & Julier. ? OOH)). This trend tells that

the Mobile Phone AR technology is going to become it new broad area.

Applying the While Phone AR into a typical business card is beneficial

whereby it can save NO space and cost.. standard and typical site of a business

4

card has very limited space fbr displaying infbrmation, what ºmrre a map. This

problem can be solved by using a small tiny unique marker, which would be use

to help display the virtual object such as 3D map on the mobile phone screen later.

The research area does not only specify towards business card. The idea of

such application can also be applied into other materials or fields such as

magazines, leaflets, newspaper advertisements and etc.

1.5 Project Scope

The project only covers the enhancement of'business card by using Mobile

Phone AR technology. This is because the limitation exists where mobile phone

has lower capability in AR technology as compared to the PC platform. The scope

of the project is to digitize the use of typical business cards and adding more user

interface concept by using the Mobile Phone AR. This application can help in

increasing the effectiveness of information delivery through physical objects

(business card) to its viewer.

1.6 Values of Stud)

Although the use of mobile AR is currently in an unknown state, it is

believed that in time to come. it will he a powerful technology due to the device

that the technology would he run on is a smart phone (Wagner & Schmalstieg.

2009). The application that is designed in this project is an idea that can he use in

any area in the tiºture. The use of the application can save space and cost, and then

increase viewer's understanding. In addition. it shows the effectiveness use of

Mobile Phone AR ºn current mrld's uifbrmation delivery through a limited

physical space.

S

1.7 Significance of Study

The significance of the study is to expose Mobile Phone AR technology

and its application to the real world. The application introduces the use of Mobile

Phone AR technology in enhancing the function of typical business card by

merging user interface virtual 3D maps into the real environment.

1.8 Structure of Project

Chapter 1:

This chapter discusses the background of the research, problem statements,

objectives, contribution, importance of research, scopes of the project, value and

significance ofthe research as well as the structure of the project.

Chapter 2:

This chapter covers Augmented Reality (AR). Mobile Phone Augmented Reality,

Business cards and Mobile 31) maps. Apart from that, some examples of'existing

augmented business card will also be discussed. The end of'this chapter will then

cover the Mobile Phone AR in the application of'physical business card.

Chapter 3:

This chapter discusses about the application development process model,

application specifications tier K)th hardware and sollware which are going to he

used. Moreover, application flow will also he discussed.

Chapter 4:

In this chapter. the architectural design of the application developed is discussed.

More internal components of the application in the section of' application

architectural will he described.

6

Chapter 5:

The final chapter discusses the application validation, strengths and weaknesses of

the application, future recommendations and also the overall conclusion of the

project throughout the study.

1.9 Conclusion

This chapter introduced the overview of this project. The use of' Mobile

Phone AR technology is believed to he very well and much more accepted by the

public compared to PC. Besides, it is the easiest way to introduce AR technology

to the public as well. This project covers the design of business card application

fbr enhance its typical functions by Mobile Phone AR technology. The idea can

also he applied into other materials or fields like the newspaper, magazine, leaflet

and etc.

7

CHAPTER 2

LITERATURE REVIEW

2.0 Overview

This chapter covers Augmented Reality (AR), Mobile Phone Augmented

Reality, Business cards and Mobile 31) maps. Apart From that, some examples of

existing augmented business card will also he discussed. The end ot'this chapter

will then cover the Mobile Phone AR in the application of'physical business card.

2.1 Introduction of Augmented Rcalith

. augmented Reality (: AR) technology is it basic idea which superimposes

graphics, audio and other sense enhancements over real world environment in

real-time (Vallino. I99S). According to Wagner & Schmalstieg (200')). AR

require (i1)OF pose tracking device such as head-mounted displays (I IMI)).

R

tangible interface object, etc; where the pose tracking must be inexpensive, work

robustly and in real time through changing environmental conditions. AR

technology aims to enhance the user's perception and interaction with the real

world by implementing the real world with 3D virtual objects which appear to

coexist in the same space as the real world.

Vallino (1998) stated that when AR technology reaches the maturity level,

user should not ledl any conflict and discrepancies between the augmented

environment and the rules that user normally senses in the real world. Due to this,

the goal of AR technology is therelbre to create a system where user cannot

differentiate between the real world and the augmented virtual element. hence.

AR attempts to change those graphics to accommodate user's head and eyes

movements, so that the graphics would always lit the perspective.

Besides that, the need to know where the user is located in reference to his

or her surroundings is the biggest challenge faced by developers in AR research

(Bonsor, 2009). Another additional problem that exist in AR is tracking the

movement of users' eyes and heads. Bonsor (200k)) stated that a tracking system

has to recognize these movements and project the graphics related to the real-

world environment the user is seeing at any moment. Although such problem

exists, there are also some ways to increase the tracking accuracy. One of them is

using multiple GPS signals like military. An AR system with GPS receiver is able

to track user's position within an area precisely There ºs a system called real-time

kinematic (UPS being developed, where it can achieve centimetre-level accuracy

(Bonsor, 2009).

As Tong as researchers overcome the challenges that lice them, AR Will

likely penetrate every corner of our lives. The AR technology o1l6-s many

potential applications in various lields including maintenance and construction.

military, instant information as well as gaming. There are hundreds of' potential

applications for such it technology, gaming and entertainment being the most

obvious ones (Ronsor, 2009).

C)

2.2 Introduction of Mobile Phone Augmented Reality

In recent years, mobile phones had become an increasingly attractive

platform for augmented reality technology (Rosenblum & Julier, 2009). It is

predicted that the number of mobile phone will be sold in 2012 is 1.8 billion, and

800 of them are estimated to be smart phone (Rosenblum & Julier, 2009). The

term Smart Phone was first introduce by unknown marketing strategists to retcr to

a then-new class of cell phones which could facilitate data access and processing

with significant computing power (7_heng & Ni, 2006). Zheng and Ni (2006) also

stated that it smart phone is a small, networked computer in the form of .1 cell

phone which usually provides personal information management (PIM)

application and some wireless communication capacity.

Befbre the development of mobile phone AR, some research had been

clone in mobile AR to replace the cumbersome backpack plus head-mounted

display setups (Figure la) with ultra mobile P('s (tJMPC's, Figure Ih). This

evolution continues by the replacement oftJMP('s to PDAs (Figure lc) and Smart

Phone (Figure ld). According to Wagner and Schmalstieg (2009), smart phone

are aiming fir it dif1crent market in AR as compared to UMP('s. The smart

phone's AR performance is surprisingly robust and fbolproof although the

appearance ofthe devices is fragile.

Thus, in order to achieve the sufficient perliºrmance lr AR application,

choosing the right algorithms is important (Rosenhlum & Julien, 2009). This is

because smart phone has lower computing capabilities as compared to the P('

plattbrm Which it much higher end device. For instance, heavy use of template

CI+ code results in it prohibitive increase in code size (Rosenblum & Julien,

2009). Normally, most smart phone has built-in camera, and this allow the device

lends itself to the computer vision approach. Nevertheless, the quality ofcomputer

tracking is highly influenced by the camera and image sensor characteristics like

frame size. update rate, or lens distortion (Wagner & Schmalstieg, 2009).

I ()

2.2.1 Symbian S60

The S60 Platform is a software platform for mobile phones that runs on Symbian OS which is an operating system (OS) designed for mobile devices and

smart phones, with associated libraries, user interface, frameworks and reference implementation of common tools (Symbian Foundation, 2010). S60 is widely

used by most of the smart phones in the world whereby it was first created and

made open source by Nokia and contributed to the Symbian Foundation

(Symbian Foundation. 2010). According to Symbian Foundation (2010), the

Symbian occupied 47% of the smart phone platform market with 17.9 millions of handsets sold in the fourth quarter of year 2008 with an increasing figure every

year.

The reason why S60 platfbrm is preferred by most of the smart phones

company is that it consists of suite libraries and standard applications such as

telephony, PIM tools and Helix based multimedia layers (Symbian Foundation,

2010). In addition. S60 software is it multivendor standard for smart phones that

supports application development in ('+ +, Java MIUP, Python and Adobe Flash.

(Nokia. 2010).

There are ü ur major release of' S60 which are the "Series 60" (2001),

"Series 60 Second Edition" (2003), "Series 60 3"d Edition" (2005) and the most

current one. the "S60 5'r' Edition" (2008) (Nokia. 2009).

2.2.2 S60 Augmented Reality - 't'hc Map 't'rackin};

Langlot/. (2010) had creaicd it software fior Symhian phones that tracks

niaps, which are outfitted with regular grids of dots and tracked with 2,5 IX)F.

The map tracker performs six degree of freedom hose estimation from almost any

neap or other uniquely textured planar object. Small black dots are added into the

map to serve as it ret: rence point for tracking.

ii

According to Langlotz (2010), the map tracker makes use of Studiersuhe

Tracker's advance features such as the fast feature detection and poses estimation.

Also, the map tracker runs on Symbian at the frame rates of 15-30 fps for a

complete application. Besides this, the map tracker allows the map to be rotated

arbitrarily and tilting the phone to -45° as it treats maps planar as a full six degree

of freedom (Langlotz, 2010). The map tracker works by using the regular

Studierstube Tracker pipeline as a circle detector and a grid detector are adding in

it.

The steps which are perfbrmed fir every Frame by the map tracker is declared

by Langlotz (2010) as follows:

"1 hrc"sholdir>g The map trucker thresholds the image and utctomutirullv extracts dark

are as'.

" C'011 finu" %ullo titi»s,, The contour %ulluiwr searches for connected regions.

" Circle defection The c"irclc' clc'lc'clur checks all extracted cuiNuul: c (comlccYrcl regions)

using a simple rlli/zsr filling tilizurilhm.

(; r"id detection All circles are then handed over to the grid detector that tries to

recorrstricc"t the regcrlar grid of thc' clots nn the male. '1"he ,L

rid detector then

extracts the patches between the dots at a resolution of 05.05 pixels. This

high resolution for the hatches is required in order to track males with /rich frequencies (sma/l structures such as buildings or . streets at a si: e of

it /e'it mi/limeter". sl. which nould otherttict create aliasim. and hence result in bad detection quality.

" lOmhlate nuuclring The template matcher uses a(; au. c. cian downsamp/ing to reduce the patch size to .?

2. x3? pi. xelc and compares it to all halchr. c in the clataha. ce.

" /'o. cc' c'. ctilºIUholl

( oºl'c'c'Il1' c/c'rc'c'/c'c/ c'('//. c are u. cc'c1 for estimating the camera's 1ºo. cc rc'lcltitR' to the map resulting in a hl)OI"*/ºocc' tracking.

12

Figurc 2. I: Map Tracker by Langkotr (20I0).

2.2.3 History of Mobile Phone Augmented Reality

According to Wagner & Schmalstieg (2003), AR'I'oolKit has ported by

them to Window CE and created the first sell contained AR application on an oll-

the-shell embedded device. This port later evolved to become the AR'I'oolKitl'lus

library (Wagner & Schmalstieg. 2009).

In the year 2004. Mühring. Lessig. & I3imher created a tracker Ii)r mobile

phones that detects color-coded 31) marker shapes (Figure 2.1 ). 'I he system did

not take camera calibration or sub-pixel accuracy into account, thus causing its

accuracy to he very limited.

lb

i Figure 2.2: 31) markers by N16hring, l. rssig, & ßimbrr (2004).

13