Self-location Recognition Using Azimuth Invariant Features andWearable Sensors

Takayuki Katahira and Yoshio Iwai

Abstract—Self-location capability is a very useful and infor-mative attribute for wearable systems. This paper proposes amethod for identifying a user’s location from an omnidirectionalimage sensor, a GPS data source and wireless LAN data.Azimuth-invariant features are extracted from an omnidirec-tional image by integrating pixel information circumferentially,thus enabling a user to independently recognize his/her locationfrom the omnidirectional image feature, the GPS data and thewireless LAN data projected into a sub-space made from thelearning data. We show the effectiveness of our method byexperimental results in real data.

I. INTRODUCTION

In recent years, the potential for developing “wearable”computers has significantly increased because of the progressin science and technology. Typical wearable computerswhich are small and unobstructive[1] include watches[2],rings[3], and headphones[4]. The concept of “wearablecomputers” is gaining socially acceptability. Most wearablecomputers have hands-free capability and can support varioustasks. In particular, they are widely used for providingwearer’s location information in various applications. Inmedical practice, a monitoring system for patients has beendeveloped by using wearable computers[5]. Such a systemcan recognize the patient’s states and in an emergency pro-vide the patient’s location to a doctor. Thus, self-location isvery informative for supporting various tasks. Many differentsensors can be used to acquire environmental informationfor obtaining the self-location information such as rangefinders[6], image sensors[7], the GPS[8], and the wirelessLAN[9]. In this paper, we propose a method for identifyinga user’s location from an omnidirectional image sensor[10],GPS data and wireless LAN data.In self-location recognition by using image sensors, 3D

reconstruction methods are frequently used[11]. These meth-ods, however, have problems of huge computational cost andother difficulties in 3D reconstructions. To minimize theseproblems, a memory based navigation approach has beenproposed[12]. Memory based navigation keeps the trainingimages as a location database, and recognizes self-locationchanges by comparing an input image with the locationdatabase. This method need not reconstruct a 3D scene fromthe images, and can improve the location recognition rateeven in complex scenes by adding training images into thelocation database.Self-location recognition by using the GPS is widely used

in car navigation systems, airplanes and ships. The GPS can

Graduate School of Engineering Scence, Osaka Unieristy,1–3 Machikaneyama, Toyonaka, Osaka 560–8531 Japaniwai@sys.es.osaka-u.ac.jp

acquire self-location information directly from satellites, andcan recognize the location anywhere where the signal of thesatellites can be received. The identification method by usingGPS and other sensors can recognize the location even inindoor environments[8]. The main problem with the GPS,however, is that the accuracy of location information variesbecause of the number of the “visible” satellites. More GPSsatellites are required to improve the accuracy, but satellitesare costly to launch.Recently self-location recognition by using wireless LANs

has been proposed. This method uses signal strengthsand the network names of wireless LANs for locationrecognition[13]. Self-location recognition by using wirelessLANs can recognize self-locations in subways, buildings,and so on. The method can accurately recognize locations incities because there are many wireless LAN access points;however the recognition rate becomes less effective wherewireless LAN access points are sparse.As describe above, each of the sensors has advantages

and disadvantages. We therefore propose a method for iden-tifying a user’s location by using a combination of multiplesensors: the omnidirectional image sensor, the GPS, and thewireless LAN to combine the advantages of each of thesesensors. We also propose to use the azimuth invariant featureextracted from the omnidirectional images, and we show theeffectiveness of our method by experimental results with realdata.

II. AZIMUTH INVARIANT FEATURES

In this paper, we use an omnidirectional image sensor asshown in Figs. 1 and 2. The sensor consists of a downwardfacing camera and an upward facing hyperbolic mirror. Thesensor has the same optical characteristics as a commoncamera, and captures 360 degrees panoramic view at anygiven time. We can extract azimuth invariant features fromthe omnidirectional images by using these characteristics.In general, the omnidirectional images captured at the sameposition differs form each other when the sensor’s orientationchanges as shown in Fig. 3. This problem is also resolvedby using the azimuth invariant features.As shown in Fig. 4, we consider the polar coordinate

system (r,θ ) with the origin fixed at the center of the image.For any feature f (r,θ ) extracted at the position (r,θ ), we cancalculate the azimuth invariant feature X(r) by the followingintegral transform:

X(r) =∫ 2π

0f (r,θ )dθ =

∫ 2π

0f (r,θ +Δ)dθ , (1)

The 2009 IEEE/RSJ International Conference onIntelligent Robots and SystemsOctober 11-15, 2009 St. Louis, USA

978-1-4244-3804-4/09/$25.00 ©2009 IEEE 4757

Fig. 1. Hyperbolic mirror

Fig. 2. Omnidirectional imagesensor

Fig. 3. Omnidirectional images captured at the same point with differentorientations

where Δ is the perturbation of the azimuth θ . For anyΔ , the above equation is always satisfied. This integraltransform is difficult to calculate for a common camerabecause a common camera cannot capture a 360 degreespanoramic image at any one time. However, by using anomnidirectional image sensor, we can easily calculate suchan integral transform. Therefore, the azimuth invariant fea-ture is a suitable feature for omnidirectional image sensors.We can use autocorrelation or differentiation as the functionf . In the following section, we describe the self-locationrecognition system as an application of invariant features andshow experimental results.

III. SELF-LOCATION RECOGNITION SYSTEM

In this section, we describe a self-location recognitionsystem using the azimuth invariant features, the GPS data,and the wireless LAN data.

A. Overview of the system

Fig. 5 shows the process flow of the self-location recogni-tion system from capturing the wearable sensors’ positionalinformation to outputting the recognition result. We use anomnidirectional image sensor[10] for capturing image data,and use the GPSMAP60CSx (Garmin) for acquiring the GPS

Fig. 4. Polar coordinate system

Wearable sensors

Omni. image GPS Wireless LAN

Dimensionalreduction

Clustering

Calculation of cluster centers

Training data(GPS)

Training data(Omni. image)

Training data(Wireless LAN)

Input data(GPS)

Input data(Omni. image)

Input data(Wireless LAN)

Com

par

e

Com

par

e

Com

par

e

Recognition (Omni. image)

Recognition (GPS)

Recognition (Wireless LAN)

Self-locationrecogntion

Location database

Fig. 5. Process flow of the system

Omnidirectional image sensor

GPS

Wireless LAN

Fig. 6. A user wearing a prototype sensor system

data. For the wireless LAN data, we use a notebook computerto get signal strengths and network names. Fig. 6 shows auser wearing a prototype sensor system.After data acquisition, image features are extracted from

the captured omnidirectional images, and extracted imagefeatures are compressed by the KL transform. The GPSdata contain elevation, latitude, longitude and time, and thewireless LAN data contain signal strengths and networknames. After feature extraction, the training data are clus-tered by the k-mean clustering algorithms and then, eachcluster is labeled. The center of each cluster is calculatedafter the clustering. At the clustering phase, recently acquiredtraining data can be considered as positional near data, so theGPS data are clustered with corresponding time information.Finally, all labeled clusters and training data are separatelystored into a location database. Self-location recognition isindependently achieved by comparing input data with datain the location database. When the recognition results differ,the system checks the confidence of the recognition results,and then makes the final decision on any positional changes.

4758

The recognition and discrimination methods are described insection III-D.

B. Features

In this section, we describe features used in the system.We use normalized RGB features and differentiation featuresas image features and these features are considered azimuthinvariant. We also use elevation, latitude, and longitude inthe GPS data, and signal strengths and network names inthe wireless LAN data.1) The normalized RGB feature: The normalized RGB

features are autocorrelations of the RGB pixel values onthe circle C(r,θ ) as shown in Fig. 4. To avoid effects ofillumination changes, we use a color vector (R′,G′,B′)T thatnormalizes the RGB values so that its length is 1. Autocor-relation of (R′,G′,B′)T on the circle C(r,θ ) is approximatedby the Monte Carlo method and calculated by the followingequation:

X(r)RGB =∫C

⎛⎝R′(r,θ )G′(r,θ )B′(r,θ )

⎞⎠

⎛⎝R′(r,θ )G′(r,θ )B′(r,θ )

⎞⎠T

dθ

�k

∑i=0

⎛⎝R′(r,2π i/k)G′(r,π i/k)B′(r,π i/k)

⎞⎠

⎛⎝R′(r,2π i/k)G′(r,π i/k)B′(r,π i/k)

⎞⎠T

=k

∑i=0

⎛⎝R′R′i R′G′

i R′B′iG′B′i G′G′

i G′B′iB′R′i B′G′

i B′B′i

⎞⎠

=

⎛⎝RRRGB RGRGB RBRGBGRRGB GGRGB GBRGBBRRGB BGRGB BBRGB

⎞⎠ , (2)

where k is the number of samples. The autocorrelation matrixX(r) is a symmetric matrix, so X(r) is vectorized to a featurevector φ(r)RGB by using 6 components as follows:

φRGB(r) =

(RRRGB,RGRGB,RBRGB,GGRGB,GBRGB,BBRGB)T . (3)

For a single omnidirectional image, the normalized RGBfeature ΦRGB is generated by varying the radius from r1 torn as follows:

ΦRGB = (φRGB(r1),φRGB(r2), · · · ,φRGB(rn))T . (4)

2) The spatial differentiation feature: The spatial dif-ferentiation feature is calculated in the same way as thenormalized RGB feature by using differential values insteadof pixel values. The differential value of the RGB pixel iscalculated as follows:

X(r)di f =∫C

⎛⎜⎝

∂R(r,θ)∂ r

∂G(r,θ)∂ r

∂B(r,θ)∂ r

⎞⎟⎠

⎛⎜⎝

∂R(r,θ)∂ r

∂G(r,θ)∂ r

∂B(r,θ)∂ r

⎞⎟⎠T

dθ . (5)

The above calculation is performed by a Monte Carlo ap-proximation and finally, by varying the radius from r1 to rn,we get:

Φdi f = (φdi f (r1),φdi f (r2), · · · ,φdi f (rn))T , (6)

where φdi f (ri) is a spatial differentiation feature vector onthe circle with radius ri.

3) Data compression: The number of dimensions of theazimuth invariant features extracted from an omnidirectionalimage is still high, and it takes considerable computationtime to recognize the self-location position. Therefore, theazimuth invariant features are compressed by using theKL transform. By means of the transformation matrix Qcalculated by the KL transform, all azimuth invariant featuresare compressed as follows:

Ψx = QTΦx,(x= RGB,di f ). (7)

We use the compressed vectors ΦRGB, and Φdi f as the imagefeatures extracted from an omnidirectional image, and storethem in the location database. The distance function in theimage feature space, which is used in the clustering andclassification processes, is defined as follows:

dimg(Φi,Φ j) =M

∑k=0

∣∣ψi,k−φ j,k∣∣ , (8)

where M is the number of dimensions of the compressedfeatures, and satisfies M < N, i.e. L1−norm is used for themetric in the image feature space.

4) The GPS feature: We use the elevation, latitude, longi-tude and time outputs. The units of latitude and longitude arein degrees, and the unit of elevation is in meters. The GPStime is the atomic time scale implemented by the atomicclocks. The GPS time was zero at 0000 hours on 6th January1980 and since it is not perturbed by leap seconds, GPS isnow ahead of UTC by 15 seconds. We, therefore, compensatefor the time difference in advance. The number of dimensionsof the GPS data is only 4, so we need not compress it. Thedistance function between two points Xgps,Ygps is definedas follows:

dgps(Xgps,Ygps) = ||Xgps−Ygps||=

√Δ12+Δ22+(xele− yele)2,

Δ1 = AΔα cosxlon,

Δ2 = AΔβ ,

Δα = ylat − xlat,

Δβ = ylon− xlon, (9)

where xlat ,xlon,xele,A are latitude, longitude, elevation, andequatorial radius, respectively.

5) The wireless LAN feature: We use the signal strengthsand network names of the wireless LAN as the wireless LANfeatures. Network names are expressed by character strings,and signal strength is expressed by percentage. The distancefunction between two wireless LAN features Xlan,Ylan is

4759

Are three classification results

all equal ?recognition result

Are two of three classification results

all equal ?

Does GPS data exist?

Does wireless LAN indicate indoor?

Majority recognition result

GPS recognition result

Wireless LAN recognition result

Image feature recognition result

YES

YES

YES

YES

NO

NO

NO

NO

Fig. 7. Decision Flow

defined by the following equation:

dlan(Xlan,Ylan) =n

∑i=0

m

∑j=0

δ (xname,i,yname, j)g(xsig,i,ysig, j),(10)

δ (x,y) ={1 x= y,0 otherwise

,

g(x,y) =

⎧⎨⎩0 x · y= 0,x/y x� y∩ x · y �= 0,y/x otherwise

,

where xname,i,xsig,i are i-th network name and signal strengthof the wireless LAN, respectively. n,m are the numbers ofreceived signals.

C. Clustering

For self-location recognition, we label clusters of trainingdata, and the minimum area of location recognition corre-sponds to these clusters. Input data taken by wearable sensorsare classified into the clusters, which provide the self-locationinformation. We use the k-mean algorithm for clustering, andthe distance functions described above are used for the k-mean algorithm. We denote Wj( j = 1,2, · · · ,c) as a clusterobtained by the algorithm.

D. The Classification Method

In the proposed system, self-location recognition is inde-pendently performed for each feature: the azimuth invariantfeature, the GPS feature and the wireless LAN feature. Whenthree classification results are same, the system outputs thesame result, but when the classification results are different,the final decision is made by the procedure shown in Fig. 7.We describe each classification method in the followingsections.1) Classification with the azimuth invariant feature:

Classification with the azimuth invariant feature is performedby the k-nearest neighbor method which uses image featurescompressed by the KL transform. The decision rule isexpressed by the following equation:

W = maxj=1,··· ,c

#Wj →Ximg ∈W, (11)

1: Faculty of Engineering Science (South side)

2: Faculty of Engineering Science (West side)

3: Computer Center

4: Athletic Field (Front)

5: Athletic Field

6: Library

7: Main Street

8: Main Street and Auditorium

9: Auditorium

10: Auditorium and Gate

11: Gate

12: Restaurant

13: Student Hall

14: Faculty of Law and Economics

15: Faculty of Science

Indoor: Faculty of Engineering Science

Fig. 8. Experimental area (outdoor)

where Ximg is an input image feature, #Wj is the numberof samples among the k-th nearest neighbors in the locationdatabase. W is the most frequent class label among the knearest samples, and an input image feature Ximg is assignedto the class W .2) Classification with the GPS feature: Classification

with the GPS feature is performed by the nearest-neighbormethod. The distance between the input GPS feature Xgpsand the center C(Wj) of the cluster Wj in the locationdatabase is calculated by (9). The decision rule is expressedby the following equation:

W = arg minj=1,··· ,c

dgps(C(Wj)−Xgps) ⇒Xgps ∈W. (12)

3) Classification with the wireless LAN feature: Classifi-cation with the wireless LAN feature is performed by thek-nearest neighbor method. The distance between the inputwireless LAN feature Xlan and the cluster center C(Wj) inthe location database is calculated by (10). The decision ruleis expressed by the following equation:

W = maxj=1,··· ,c

#Wj → Xlan ∈W. (13)

W is the most frequent class label among the k nearestsamples, and an input image feature Xlan is assigned to theclass W .

IV. EXPERIMENTS

We conducted self-location recognition experiments toshow the effectiveness of our method. Omnidirectional videosequences were captured by a walking person, and a numberof selected still images per second. The still images andthe corresponding GPS and wireless LAN data were usedas training data. The training data were captured on 24thSeptember, 2008 at our University, and the verificationdata were captured on 4th October, 2008. 500 items oftraining data and associated verification data were recorded.25 clusters of training data are shown in Figs. 8 and 9.Numbers 1 to 15 are outdoor scenes, and Numbers 16 to25 are indoor scenes. We also compared the recognitionaccuracy of the normalized RGB feature with that of thespatial differentiation feature.

4760

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 total R

ecog

nitio

n ra

te (

%)

Normalized RGB feature only GPS Wireless LAN Proposed Method

Fig. 11. Recognition rates of the proposed system using normalized RGB features and other sensor data

Area No.

TABLE I

CONFUSION MATRIX OF THE PROPOSED SYSTEM USING NORMALIZED RGB FEATURES

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 total

Rec

ogni

tion

rate

(%

)

Spatial differentiation feature only GPS Wireless LAN Proposed Method

Fig. 12. Recognition rates of the proposed system using spatial differentiation features and other sensor data

Fig. 9. Experimental area (indoor)

A. Effects of feature dimension

We compressed the azimuth invariant features, becauseof the high dimensionality of the feature vector. The highdimensionality results in a sparseness of the feature space,

FRR (Normalized RGB)

FRR (Spatial Differentiation)

FAR (Normalized RGB)

FAR (Spatial Differentiation)

The number of dimensions

Rec

og

nit

ion

rat

e (%

)

Fig. 10. Recognition rate by image features

and affects the accuracy of the clustering and the recognitionrate. Therefore, we conducted an experiment to investigatethe accuracy change against the change of the dimensionality

4761

Area No.

TABLE II

CONFUSION MATRIX OF THE PROPOSED SYSTEM USING SPATIAL DIFFERENTIATION FEATURES

of the image features. The recognition in this experiment isperformed by the image features only. The experimental re-sult is shown in Fig. 10. According to the result, recognitionrates of both features increase when the dimensionality offeatures is low, but when the number of dimensions of featureis greater than 10, the recognition rate does not increase anddeteriorates. Therefore, we decided to use 10 dimensionalimage features in all future experiments on balancing usefulresults versus computation cost.

B. Self-location recognition

We have conducted self-location recognition experimentsusing two systems: one uses the normalized RGB featuresand the other uses spatial differentiation features. The recog-nition rates of both systems are shown in Figs. 11 and 12, andthe confusion matrices are shown in Tables I and II. The i-throw and the j-th column element expresses the percentagethat the i-th area datum is recognized as the j-th area.From Figs. 11 and 12, the proposed system achieves a

high accuracy rate by using multiple sensors throughout.Comparing the results using normalized RGB features, withthe results using spatial differentiation features, the for-mer system has better performance than the latter system.Normalized RGB features can express the characteristicsof places from the omnidirectional images irrespective ofillumination changes. The recognition performance of indoorscenes is not good using the azimuth invariant features,because of the lack of variety in indoor scenes. The azimuthinvariant feature is a feature extracted by integral transform,so slight changes of indoor scenes cannot affect the trans-form. Therefore, in order to recognize indoor scenes moreaccurately, we need to extract image features sensitive tolocal image changes, but this is a trade-off problem that isaffected by the noise.

V. CONCLUSION

This paper proposes a method for self-location recognitionusing an omnidirectional image sensor and other wearablesensors. The proposed method uses azimuth invariant fea-tures extracted from the omnidirectional images, the GPS

data and the wireless signals from the wearable sensors.The results demonstrate that through wearable sensors themethod can recognize self-location indoors and outdoorswith a high accuracy rate. In future work, we will improvethe recognition method so that it is more accurate and robustagainst noisy data.

REFERENCES

[1] K. Tsukada, and M. Yasumura, “ActiveBelt: Belt-type Wearable TactileDisplay for Directional Navigation,” Proc. of UbiComp2004, pp. 384–399, 2004.

[2] N. Kern, S. Antifakos, B. Schiele, and A. Schwaninger, “A Modelfor Human Interruptability: Experimental Evaluation and AutomaticEstimation from Wearable Sensors,” Proc. of the Eighth IEEE Intl.Symp. on Wearable Computers, Vol. 1, pp. 158–176, 2004.

[3] M.Fukumoto, and Y.Tonomura, “Body Coupled FingeRing: WirelessWearable Keyboard,” Proc. of the ACM Conference on Human Factorsin Computing Systems, pp. 147–154, 1997.

[4] K. Hirota and M. Hirose, “An Implementation of Wearable AuditoryInterface,” Proc. of MOVIC 2002, pp. 570–575, 2002.

[5] P. Lukowicz et al., “AMON: a wearable medical computer for highrisk patients,” Proc. Sixth International Symposium on WearableComputers, Seattle, Washington, pp. 133–134, 2002.

[6] H. Surmann, A. Nuchter, and J. Hertzberg, “An autonomous mobilerobot with a 3D laser range finder for 3D exploration and digitalizationof indoor environments,” Robotics and Autonomous Systems, Vol. 45,pp. 181-198, 2003.

[7] C. Zhou, Y.Wei, and T. Tan, “Mobile Robot Self-Localization Based onGlobal Visual Appearance Features,” Proc. IEEE Int. Conf. Robotics& Automation, pp. 1271–1276, 2003.

[8] H.-S. LEE, et al., “Pedestrian Tracking Using GPS, Pedometer andMagnetic Compass,” Trans. of IEICE B, Vol. J84-B, No. 12, pp. 2254–2263, 2001.

[9] T. Ishihara, N. Nisio, “A Positioning System Exclusively UtilizingGPS and Wireless Network Detecter,” IPSJ Tech. Report, No. 2004-UBI-6-14, pp. 91–96, 2004.

[10] K. Yamazawa, Y. Yagi, M. Yachida, “Omnidirectional image sensor —HyperOmni Vision—,” Proc. 3rd Int. Conf. on Automation Technology,Vol. 5, pp. 127–132, 1994.

[11] K. T. Simasarian, T. J. Olson, and N. Nandhakumar, “View-InvariantRegions and Mobile Robot Self-Localization,” IEEE Trans. onRobotics and Automation, Vol. 12, No. 5, pp. 810–816, 1996.

[12] Y. Yagi, K. Imai, K. Tsuji, M. Yachida, “Iconic Memory-BasedOmnidirectional Route Panorama Navigation,” IEEE Trans. on PatternAnalysis and Machine Intelligence, Vol. 27, No. 1, pp. 78–87, 2005.

[13] J. Rekimoto, A. Shionozaki, T. Sueyoshi, T. Miyaki, “PlaceEngine:A WiFi location platform based on realworld-folksonomy,” InternetConference 2006, pp. 95–104, 2006.

4762