facilitating e cient object tracking in large-scale ...qsheng/papers/compj2011.pdffacilitating e...

Facilitating E�cient Object Tracking

in Large-Scale Traceability Networks

Yanbo Wu, Quan Z. Sheng, and Damith C. Ranasinghe

School of Computer ScienceThe University of Adelaide, Adelaide, SA 5005, Australia

Email: {yanbo, qsheng, damith}@cs.adelaide.edu.au

With recent advances in technologies such as radio-frequency identi�cation(RFID) and new standards such as the electronic product code (EPC), large-scale traceability is emerging as a key di�erentiator in a wide range of enterpriseapplications (e.g., counterfeit prevention, product recalls, and pilferage reduction).Such traceability applications often need to access data collected by individualenterprises in a distributed environment. Traditional centralized approaches (e.g.,data warehousing) are not feasible for these applications due to their uniquecharacteristics such as large volume of data and sovereignty of the participants. Inthis paper, we describe an approach that enables applications to share traceabilitydata across independent enterprises in a pure Peer-to-Peer (P2P) fashion. Dataare stored in local repositories of participants and indexed in the network basedon structured P2P overlays. In particular, we present a generic approach fore�ciently indexing and locating individual objects in large, distributed traceablenetworks, most notably, in the emerging environment of the Internet of Things.The results from extensive experiments show that our approach scales well in bothdata volume and network size. A real-world returnable assets management systemis also developed using the proposed techniques to demonstrate its feasibility.

Keywords: Internet of Things; Traceable Networks; Radio Frequency Identi�cation;Peer-to-Peer Systems; Scalability

1. INTRODUCTION

Recent advances in RFID (radio-frequency identi�ca-tion) technologies, wireless sensors, and Web serviceshave led to the emergence of the �Internet of Things�,a global network where everyday objects such as build-ings, sidewalks, and commodities are identi�able, read-able, addressable, and even controllable via the Inter-net [1, 2, 3]. Such a network can provide real-time in-formation on the movements of objects. It is essentiallya �data-on-network� system, where RFID tags containan unambiguous ID and other data pertaining to theobjects can be stored and accessed over the Internet.This makes automatic tracking and tracing possiblein large-scale applications (e.g., supply chain manage-ment across companies and geographies, targeted prod-uct recalls and anti-counterfeiting). Traceability ap-plications analyze automatically recorded identi�cationevents to discover the current location of an individ-ual item. They can also retrieve historical information,such as previous locations, transportation time betweenlocations, and time spent in storage.Central to realizing these traceability applications

is the management of data, particularly on how toshare data that are typically collected by individualenterprises. An obvious solution is to publish all data

collected within each organization to a central datawarehouse. Unfortunately, this approach has severalsevere drawbacks. Firstly, object movement and relateddata are valuable information that companies may bevery reluctant to put their data in a shared centralwarehouse. Secondly, such an approach has very limitedscalability and is not feasible for the environment of theInternet of Things where the amount of data collectedcould be enormous (e.g., from hundreds of thousands ofRFID-tagged objects in a supply chain) [4, 5, 6, 7].

An alternative would be a peer data management

system (PDMS) that allows data to be stored in lo-cal repositories at each organization and makes thoserepositories accessible in a Peer-to-Peer (P2P) fash-ion [8, 9]. A representative e�ort in this direction isIBM's Theseos [10] that enables traceability applica-tions to process complex queries across organizations ina completely distributed setting. Theseos relies on anovel traceability data model that eliminates any datadependencies between organizations, which serves as aglobal schema that allows the formulation of a querywithout knowledge on how the data is stored, where itis located, and how a tracking query is executed [4]. Toimprove the performance of traceability query process-ing, each organization is required to maintain the infor-mation on the movement path of an object. With this

The Computer Journal, Vol. ??, No. ??, ????

2 Y. Wu et al.

information, it is possible to minimize the number ofnodes to be visited without �ooding queries to all nodesin the network. Unfortunately, to obtain this informa-tion, Theseos requires high synchronization with otherenterprise data (e.g., billing or accounting information).This is impractical for many applications where such en-terprise data may not be conveniently available.

In this paper, we propose a generic approach fore�cient tracing and tracking of objects in large-scale, distributed environments such as the Internetof Things. In particular, by analyzing a wide rangeof traceability applications, we abstract the conceptof traceable networks and a new model for movingobjects in discrete spaces. Our approach is builton top of the DHT (Distributed Hash Table) basedoverlay network [11]. We index an object and itslatest state at a deterministic node called gateway

node. Every time the object moves from a source

node to a destination node, the gateway node updatesobject status at the source and the destination nodes ofits movement, thereby establishing the information onmovement path of the object. Since gateway nodes arerandomly chosen in an anonymous way, the sovereigntyof the participating organizations is not a�ected. Toreduce the indexing overhead from large volume datain traceability applications, we further propose anenhanced group-based indexing mechanism. Instead ofindexing each object individually, we group the objectsthat arrive at a node within a time frame by the pre�xof their hashed ids and index these groups. We proposealgorithms to determine the optimal pre�x length byconsidering both indexing cost and load balancing.Extensive experiments have been conducted to show theviability and scalability of our proposed techniques.

The remainder of the paper is organized as follows.Section 2 brie�y introduces traceable networks anda moving object model. Section 3 describes a P2Papproach for e�cient processing of traceability queriesin a DHT-based overlay network. Section 4 presentsan enhanced indexing approach dealing with massivevolumes of data in large-scale traceability applications.Section 5 introduces algorithms for processing queriesbased on the underlying P2P architecture framework.Section 6 reports the results of the evaluation ofthe proposed techniques. Section 7 demonstrates areal-world application developed using the techniquesproposed in this article. Finally, Section 8 gives anoverview of the related work and Section 9 providessome concluding remarks.

2. PRELIMINARIES

In this section, we will brie�y introduce traceablenetworks and a new framework for modeling movingobjects.

ManufacturerDistributorRetailer

Darwin

PerthAdelaide

MelbourneSydney

Cairns

Alice SpringBrisbane

FIGURE 1. An Example of Supply Chain Networks

2.1. Traceable Networks

Traceable networks in�uence our daily life to a greatextent [12]. A supply chain in which goods aretransferred from the manufacturers, through a chain ofdistributors to retail stores and �nally to consumers isan example of a traceable network. This network maybe across several states, countries or even continents.Figure 1 illustrates a typical pharmaceutical supplychain network. Pharmaceuticals are critical to people'slives and an important capability is the ability to tracean individual bottle of medicine to its source throughsupply networks in the event of a problem such astampering. Other examples include hospital equipmentmanagement systems, returnable asset managementsystems and patient monitoring applications. Thesenetworks share some common characteristics. Weabstract them and introduce the generic traceablenetwork architecture.Figure 2 gives a high level description of traceable

networks. There are three main entities in such anetwork: node, receptor, and object. Each node governsa number of receptors (e.g., RFID readers) that are usedto capture the objects. Receptors are deployed at �xedlocations (e.g., entry of a warehouse or hospital room).Nodes are logical and abstracted to represent partnersin the network. For example, in a supply chain network,a node may be a distribution center or a retail store,and the receptors are RFID readers deployed at theentrances and exits. Objects are goods attached withRFID tags.Physically, objects move along a certain trajectory

(e.g., a pallet moves in a supply chain). We callthis the physical object �ow. After the objects arecaptured, their movements are digitalized and thecaptured information is called the information �ow.Receptors are the medium to transform the physicalobject �ow into the information �ow. Nodes store the


Facilitating Efficient Object Tracking in Large-Scale Traceability Networks 3

NODES

RECEPTORS

OBJECTS

Internet

Past Present Future

��

��

��

��

� ��

��

FIGURE 2. A High Level Description of TraceableNetworks

information �ow segments which occur inside their ownterritory. Nodes are connected (e.g., via the Internet) sothat they can exchange information with other nodes inorder to get the whole picture of an object's trajectory.It should be noted that we assume in this paper that

the data captured by receptors (e.g., RFID readings) isalready cleansed. Sensor data cleansing is an importantresearch topic, which has attracted lots of e�orts fromresearchers in the last few years. Interested readers arereferred to [13, 14, 15].Within such networks, two important problems

should be solved to enable e�cient tracing: i) howinformation �ow should be modeled and stored (i.e., thedata model), and ii) how information is shared betweenorganizations. The data model a�ects the performanceof information persistence and retrieval. Solutions toinformation sharing a�ect information completeness,tracing e�ciency as well the privacy of individuals andcorporations and eventually the successful adoption ofthe overall solution.

2.2. A Model for Moving Objects in Discrete

Space

In existing models for moving objects, both time andspace domains are continuous. This is necessarybecause the query result domain is also continuous.However, in many traceability applications (e.g., RFID-enabled supply chains), objects are only captured at�xed locations (signi�ed by nodes and receptors). Thequeries refer to a �nite set of �xed locations instead ofa point or region in an in�nite space. The traditionalcontinuous models can be adopted in these applications.However, they are unnecessarily heavy because of thede�nition and maintenance of various spatial elements.We therefore propose a new model that can representmoving objects and their attributes in an economicway, namely MOODS (a Model for mOving Objects inDiscrete Space).

L(o, t) : O× T 7→ N (1)

Equation 1 shows the signature of the model. Givenan object o in the object set O and a point of time tin the time domain T, the locating function L derivesthe location where the object o was/is/will be at t. Ifo is not in the system, the result of L is nil (indicating�nowhere�).In our model, the time domain T is continuous

because receptors are working continuously to capturemoving objects. The time-parameterized queries maytake any moment as an input. However, as wementioned above, the space domain is discrete. Insteadof a continuous in�nite domain, MOODS de�nesthe space domain as a �nite set of nodes N ={n1, n2, ..., nm}. The set is dynamic since new nodescan join and existing nodes may leave the network. Theobject domain is the same as the existing models, i.e.O = {o1, o2, ..., on}.The function L essentially locates an object. Another

important function is to �nd the trace of an object.Similar to L, we de�ne the trace function TR as:

TR(o, tstart, tend) : O× T× T 7→ P (2)

P : {(nk1, nk2

, ..., nkp, . . . , nkl

)|nkp∈ N, 0 < l ≤ m}

(3)Given an object o and a time range (tstart to tend), TR

�nds the trace of o during that time frame. P denotesthe domain of path. A path is a sorted list of nodes (canbe empty) in N. The sorting is done by the order of thenodes visited by object o (i.e., by time).MOODS can be concretely de�ned in a distributed

database. The schema at each node is de�nedas record:{object_id, start_time, end_time,

from, to}. The primary key for this schema isobject_id and start_time. We use this compositekey as the primary key, instead of object_id, becausean object may leave a node, and then come back later.Using object_id as the primary key can only recordthe latest appearance of the object. Finally, elementsfrom and to de�ne the information �ow segment. Themoving path of an object is de�ned as a distributedlinked list.

2.3. The PeerTrack Project

The work presented in this article is part of PeerTrack1,a research project that aims at developing noveltechniques enabling applications to share traceabilitydata e�ciently and e�ectively across independententerprises in large-scale traceable networks. InPeerTrack, traceability applications can share dataacross independent organizations in a P2P fashion.This is realized by a novel data model that introduces

1http://www.cs.adelaide.edu.au/peertrack/


4 Y. Wu et al.

n1 n2 n3 n4

n7

o1 o1 o1

M1. o1.latest = n4

M2. o1.to = n4

M3. o1.from = n3

o1.from=nil

o1.to=n2

o1.from=n1

o1.to=n3

o1.from=n2

o1.to=nil

o1.from=n2

o1.to=n4

o1.from=nil

o1.to=nil

o1.from=n3

o1.to=nil

o1.latest=n3

o1.latest=n4object movement

message

FIGURE 3. An Example of IOP Acquisition

properties to indicate the departure and the arrivalinformation of objects. We call the properties theinformation of object path (IOP) [16]. With IOP, eachnode maintains segments of objects' moving paths anduses this information to expedite P2P queries in thenetwork. Our IOP is similar to receivedFrom andsendTo in IBM's Theseos [4, 10]. However, as wediscussed in Section 1, Theseos does not o�er anygeneric approach on how to obtain such information intraceable networks, which is the focus of this paper.

3. A P2P APPROACH FOR TRACEABLE

NETWORKS

In this section, we will introduce a P2P approach thatcan process traceability queries in an e�ective ande�cient way.The main idea of our approach is to index an object

and its latest location in the DHT network, whilekeeping the IOP information at the nodes where theobject has been observed. Instead of a centralized indexserver, we store the index for an object at the nodewhich is the result of a DHT lookup for the object's id.IOP contains information about the source (i.e., wherethe object comes from) and destination (i.e., where theobject leaves to) nodes, thus it can be used to quicklyanswer queries to obtain the trace of an object (tracequeries as de�ned in Equation 2).The place where the index is stored is determined

solely by the id of the object thanks to DHT'sdeterminism, so that from anywhere in the network,the object can be located by its id2. The abstract datastructure is essentially a double linked list distributedin the DHT network. The head of the list is the nodewhere the latest location is indexed, which we call agateway node. When the object moves to a new node,the gateway node is updated and the list is expanded.In our work, we adopt Chord [17] as the overlay for

2In order to take this advantage, we hash the object's raw idusing the SHA-1 function. Thus the ids of the objects and thenodes are in the same space.

its adaptiveness as nodes join and leave. Figure 3illustrates our idea.The node n7 is the gateway node for object o1. It

stores the latest location of o1. In this case, before o1arrives at node n4, it was captured at node n3. After itarrives and is captured by a receptor governed by n4, n4sends a message (M1) to n7 by the P2P lookup interface,telling n7 that o1 has arrived at n4. Upon receivingthis message, n7 updates its index and sends messagesto n3 and n4. The message to n3 (M2) indicates thato1 arrives at n4, so n3 �nds the record for o1 whichdoes not have end_time, and updates its IOP by settingo1.to

3 as n4 and o1. end_time as the last time thisobject was seen in o3 (cached by a temporal table).In this way, if o1 comes back to n3 again, this will betreated as an isolated record. The message to n4 (M3)indicates that o1 was from n3, so n4 updates its IOP bysetting the record for o1 without o1.from as n3. Nowwe have all columns for o1 at n3 acquired (from at n3has been set when o1 arrived at n3), and all columnsexcept end_time and to at n4 are set. The latter twocolumns will be set when o1 leaves n4 and arrives at adi�erent node.In this way, the IOP is established and the index is

updated. Since IOP is essentially a distributed doublelinked list sorted by time, the correct sequencing ofrecords is always guaranteed. To answer queries abouta speci�c object, the �rst step is to �nd out its latestlocation by querying the corresponding gateway node.We can then trace back the list and ask each node alongthe list for the desired information.In our approach, IOP is stored at nodes in a

distributed way. An alternative way is to store it only atgateway nodes. However, by doing this, we will not beable to answer range queries (e.g., �how many objectsmoved from n1 to n2 during last month?�) becauseobjects traveling together may not be indexed at thesame gateway node, due to the uniformity of the hashfunction. Storing IOP at gateway nodes is thereforenot a good idea in the sense that we have to �ood thenetwork to answer range queries.We do not require the nodes along the routing

path from the original node to the gateway node tostore the index to the latest location of an object,although doing so increases the availability of indexesand statistically decreases the time to route queriesor indexes to gateway nodes. However, our approachcan still achieve similar advantages without this extrastorage cost. The reason is that IOP information isstored along the moving trace of objects. Whenever aquery is routed to a node that keeps the IOP of thecorresponding object, the processing of the query cancommence without routing to the gateway node. Thisis illustrated in Figure 4.Suppose that an object o1 moved from node n1 to

3For convenience of discussion, �The to column of the recordfor o1� is simpli�ed as o1.to.



n1 n2 n3 n4

n7

o1 is now

indexed at n7

o1 o1 o1

n6 n5

query : trace o1

FIGURE 4. An Example of Query Processing

n4 via n2 and n3. Object o1 and its latest location n4are indexed at node n7 which is the gateway node foro1. When a trace query of object o1 is performed onnode n6, the query is �rst routed to the gateway noden7 (via nodes n5 and n2) and then dispatched to n4 forprocessing. Clearly, this is a waste since one node alongthe routing path keeps the information of the object (inour case, node n2). The routing can simply be stoppedat this node, which will begin the query processing.A problem of this naive IOP acquisition method

is the indexing cost. Whenever an object arrivesat a node, the system has to issue 3 messages (1message from the node and 2 messages from the object'scorresponding gateway node) in order to establish theIOP links. This is clearly an expensive approach forlarge-scale applications where order of magnitude isoften thousand, and objects often move in groups.In order to solve this problem while still having theIOP links established, we next introduce a novel IOPacquisition algorithm.It is also worth mentioning that our architecture

is built on top of the existing DHT networks andwe do not explicitly deal with the dynamicity of theunderlying networks. One reason is that Chord adaptswell with the dynamic networks. When new peer joins,only a small portion of nodes will migrate their data.Similarly, when a peer leaves, it will migrate its data toanother peer [17]. The data migration can be expensivewhen its amount is huge. However, since the nodes inthe traceable networks are reliable servers, they havetheir own replication policies and the o�ine time isnormally very short. We do not maintain replicationfor neither data or index to save storage cost.

4. THE IOP ACQUISITION ALGORITHM

Before we introduce the algorithm, we �rst summarizethe symbols used in the algorithm in Table 1.In a large traceable network, the data volume might

Symbol Description

Nn Number of nodes in the traceable networkL Length of object/node ID in bitsLp Length of pre�x in bitsLmin The minimum value of Lp

Tinterval Time interval of data capturingTmax The maximum value of Tinterval

No Number of objects captured within Tinterval

Nmax The maximum value of No

TABLE 1. Symbols

0000 0000 0000 00010000 0000 0000 00020000 0000 0000 00030000 0000 0000 00040000 0000 0000 00050000 0000 0000 0006

0001 0000 0000 00010001 0000 0000 00020001 0000 0000 0003

group prefix = 0000

group prefix = 0001

h ('0000')

DHT

Lookup

DHT(h('0000'))

h ('0001')

DHT(h('0001'))

P2P Network

Group ID Generation Gateway Lookup

FIGURE 5. An Example of Grouping and GatewaySelection

be very high. Indexing each individual object will causeenormous number of messages to �ood the network.An obvious solution to solve this problem is to classifythe objects arriving at a node within a small periodof time into di�erent groups. In particular, objects'raw ids can be hashed using the secure hash algorithmSHA-1 and grouped by the pre�xes of their hashedids. Figure 5 illustrates how data are grouped and howthe gateway nodes are chosen. In general, we de�ne aglobal pre�x length (4 in this example) and group theobjects according the pre�x of their ids. The objects areindexed in groups to the gateway node which is chosenaccording to the pre�xes by the DHT network.This approach can signi�cantly reduce the bandwidth

cost. For example, if 1024 objects arrived at a node nand we choose a pre�x length of 4, there are at most24=16 pre�xes. As a consequence, instead of indexingall these 1024 objects, we simply classify them into atmost 16 groups by pre�xes (with pre�xes as the groupids), and index the groups. With proper compressionmethods, the bandwidth cost can be further reduced.Based on this concept, we introduce our enhanced

IOP acquisition algorithm in the following sections.


6 Y. Wu et al.

4.1. Timing of Grouping

A group function is invoked periodically at timeintervals of Tinterval or triggered by other conditions(to be discussed later) to divide the objects capturedduring this time frame into groups. Two objects belongto the same group when their ids have Lp pre�x bits incommon.Tinterval, is the width of windows of the object

stream. We take the objects in the same window forgrouping and indexing at one cycle. This parameter isintroduced to restrict the size of the indexing message.Unfortunately, a �xed value of Tinterval will causeproblems when the object stream is unstable. Forexample, if an object stream suddenly has an extremelyhigh volume (e.g., more products enter the warehouse inone cycle), the number of objects in the same windowwill be very high thus the size of indexing messagesbecomes larger.In our work, we exploit an adaptive scheme to

determine the width of windows. In particular,we de�ne a maximum time interval, Tmax, and themaximum number of objects received at each cycle,Nmax. After Tmax has passed, or when the number ofobjects received has exceeded Nmax, the current cycleends and the grouping process is started on the receivedobjects during this cycle. Meanwhile, a new cycle isstarted. In order to limit the message size sent forindexing the objects, we believe an upper bound onthe number of received objects in each cycle (Nmax)is necessary. With a small Tmax, it is possible to enablereal time indexing so that the objects will not be delayedtoo long for indexing when volumes become very low.The value of Tmax can be determined by the system'stimeliness restriction and is con�gurable. Similarly, thevalue of Nmax can be con�gured.

4.2. Grouping Algorithm

Lp is the key to determine the total number of groups inthe system and should be chosen wisely. If it is too big,the number of groups is high and in the worst case, it isclose to the number of objects. As a result, the numberof messages will not be signi�cantly decreased. On theother hand, if it is too small, only a small portion of thenodes in the network will be responsible for indexing,thus the work load is not well balanced. Essentially,it is important to �nd an optimal value of Lp thatcan guarantee that almost every node in the networkhas the opportunity to index at least one pre�x (i.e.,group). We denote the probability that a node has atleast one group to index as p. Because pre�xes aredistributed uniformly at random over the nodes by thehash function, according to probability theory, we have:

p = 1−(Nn − 1

Nn

)m

(4)

Where m is the number of groups to distribute, and

it is a function of the length of pre�x: m = 2Lp .The optimal value of Lp should yield the value of p

as close as possible to 1.And if m is Nn, we have:

limNn→∞

p = limNn→∞

(1−

(Nn − 1

Nn

)Nn)

= 1− limNn→∞

1(1 + 1

−Nn

)(−Nn)

= 1− 1

e= 0.6321 < 1

(5)

Obviously, when m is Nn, the probability that all mnodes are used for indexing is only 0.6321, which is notlarge enough for good load balancing.When m is Nn log2Nn, we have:

limNn→∞

p = limNn→∞

(1−

(Nn − 1

Nn

)Nn log2Nn)

= 1− limNn→∞

((Nn − 1

Nn

)Nn)log2Nn

(6)

It is easy to prove that the limit value of Equation 6is 1.Thus we can determine the Lp value:

Lp > log2 (Nn log2Nn) = log2Nn + log2 log2Nn (7)

According to Equation 7, as long as Lp is chosenlarger than log2Nn + log2 log2Nn, we have a highprobability to give each node in the network a chanceto be responsible for a group. In our system, wechoose dlog2Nn +log2 log2Nne for Lp. As shown in ourexperiments (see Section 6), the value we have chosen isgood enough to maintain adequate performance whileensuring good load balancing.As new nodes join and existing nodes leave, Nn

is dynamic. As a result, there is no precise wayto calculate this value. However, there are existingalgorithms available to estimate the value of Nn.Interested readers are referred to [18] for details.According to Equation 7, Lp increases much slower thanNn. Therefore there is no need to recalculate Lp everytime the network changes. Instead, Lp can be adjustedat a relatively long interval.During the bootstrap of the network, however, Nn

should be re-calculated more frequently to quicklydetect a change of Lp because Nn starts from a smallvalue and changes frequently. According to Equation 7,Lp will also begin with small values and keep changing,thus at the beginning, the number of groups is smalland objects are indexed nearly individually. To solvethis problem, we introduce Lmin, a minimum value ofLp. The value of Lmin is also con�gurable and appliesto all nodes in the network.



A triangle for indexing objects with prefix '000'.

The current length of prefix is 3.

a) A tree extended for the triangle in a).

The two nodes indexing '0000' and '0001' become parents for another

four nodes indexing the four indices indicated in the lowest layer.

b)

0000 0001

000

000

0000 0001

00000 00001 00010 00011

FIGURE 6. An Example of Data Triangle

4.3. Algorithms for Indexing and Index Persis-

tence

After the groups are built, the node that capturedobjects will send indexing messages to the gateway nodefor each group. The gateway node is determined bythe hash value of the group id. For example, objectsbelonging to the group �00� will be indexed in thenode hash(�00�) (Note that the parameter for the hashfunction is a string, instead of an integer, because�00� and �000� are di�erent pre�xes). The indexingmessage contains the list of objects in the group �00�.The gateway node stores the indexing information foreach object individually. We apply gzip compressionon the message to reduce the size of the message.Obviously, compression over a list of objects is muchmore economic than compression of individual objects.The IOP update process is almost the same as that inSection 3 except instead of sending one message for eachobject, we send one message for each group of objectswhich are from the same node. However, we cannotsimply store and lookup the index at the gateway node.There are two issues we need to address because of theintroduction of the grouping scheme.Firstly, changes of Lp cause grouping inconsistencies.

For example, at node n1, Lp was 2 and we groupedobjects 0000 and 0001 into the same group 00. Theindex was stored at gateway node hash(�00�). Beforethese objects arrived at node n2, new nodes joined thatcaused Lp to become 3. After objects 0000 and 0001arrived at node n2, they are grouped into group 000 andthe corresponding gateway node becomes hash(�000�).At this moment, the node hash(�000�) does not haveprevious information about these two objects and itmay assume that node n2 is the �rst node for thetrace of these objects, which in fact should be n1. Asimilar situation also happens when the size of thenetwork decreases. This causes the IOP informationto be updated incorrectly.Secondly, when new nodes join the network, the

increase of number of nodes does not cause Lp toincrease, there are always at least Nn − 2Lp nodes idle.The load is then not well balanced.To solve the above two problems caused by Lp,

we introduce a distributed data structure called Data

Triangle. A data triangle consists of three nodes in thenetwork. The roles of them are not symmetric. Figure 6(a) shows an example of the triangle. The gateway nodeis responsible for indexing objects whose ids are pre�xedby �000� (current pre�x length is 3). The two childnodes are responsible for indexing �0000� and �0001�.But the child nodes do not index groups directly, insteadthey are secondary storage for their parent. The parentdelegates part of the indexing data according to the nextbit in an object id after the pre�x to the child nodesrespectively, thereby keeping the workload between thethree balanced. Essentially, data triangle is a simpli�edversion of Distributed Pre�x Hash Tree [19].Suppose to increase the current Lp by 1, we need

∆Nn new nodes to join. We have:

dlog2((Nn + ∆Nn)log2(Nn + ∆Nn))e−dlog2 (Nnlog2Nn)e = 1

(8)

Removing the ceilings, we have:

log2((Nn + ∆Nn)log2(Nn + ∆Nn))

− log2 (Nnlog2Nn) < 2(9)

It is easy to verify that ∆Nn is less than 3Nn. When∆Nn = Nn or ∆Nn = 2Nn, the inequality holds, whilewhen ∆Nn = 3Nn, it does not. Since the equation onthe left is a monotonically increasing function of ∆Nn,we can conclude that the value of ∆Nn satisfying theequality in Equation 8 must be between 2Nn and 3Nn.In the worst case, there are ∆Nn nodes idle. For

a traceable network whose global pre�x length is Lp,we use extra 2Lp+1 logical nodes (the child nodes inthe data triangle) for indexing. The 2Lp+1 + 2Lp =3Nn log2Nn logical nodes are distributed among atmost ∆Nn + Nn < 4Nn physical nodes. By similarinference to Equation 7, there is a high probability thatall physical nodes have at least one group to index.With the changes to the network (nodes join and

leave), the child nodes may have their children.Eventually, the indexing will be distributed in a treestructure (Figure 6 (b)). However, the tree is notnecessary. From the analysis above, we can see that thedata triangle is good enough to maintain a well balancedworkload. Maintaining a tree introduces longer delayfor looking up.Trees also cost more for processing indexes. For all

new objects, unless we fully traverse the tree, it isimpossible to determine whether they are new or havehistorical information stored somewhere in the tree. Inthis case, each object will be looked up in each level ofthe tree, so for No objects and a tree of height h, thetotal number of DHT lookup operations is h∗No, whichclearly is expensive.Fortunately, we do not maintain the whole tree all

the time. Instead, once the network size changes and


8 Y. Wu et al.

causes the pre�x length to change, we start a splitting-

merging process. If the pre�x length increases, then thechild nodes in the triangle become parent nodes in newtriangles. The data stored in the old parent will all bedelegated into the two new parent nodes which are itschild nodes. Similarly, if the pre�x length decreases,the parent node in the triangle becomes a child nodeof another node that is indexing a shorter pre�x, theparent node's two child nodes migrate the data they areindexing to the parent node. Thus eventually we alwaysmaintain only triangles, instead of trees. To look up anobject which does not exist locally, we only need to askthe parent and its two children. Taking Figure 6 as anexample, after the pre�x length increases from 3 to 4,the node indexing pre�x �000� will split all its data into�0000� and �0001� according the fourth bit of their ids.Figure 7 shows our indexing algorithm.It is not necessary to start the splitting-merging

process as soon as the pre�x length changes. Insteadit can be done during the system's free time. This willnot cause problems because as we can see in Figure 7, wealways try to traverse a tree. This is not only because wedo not do immediate splitting-merging, but also becausethe detection of pre�x length changes at di�erent nodesis not synchronized, thus it is possible that at some time(usually after a pre�x change), there are trees existingin the system.The index algorithm (see Figure 7) �rst tries to

update the index stored locally (line 1) and gets a listof objects whose index is stored either in ascents ordescents by a set di�erence operation (line 2). Thenwe refresh the local indexing records by searching upand down in the tree (lines 4 and 5) for the objectsin the di�erence set. After the index of all objectsare downloaded to the local storage, the index will begrouped by source nodes and sent back to child nodes(not shown in the algorithm).The two refresh procedures are used to retrieve

the indexes by traversing the tree up and down,respectively. Before sending the fetching request to therespective nodes, the object list is �ltered by the pre�x(the �lter function in line 4 of refresh_from_ascentand lines 2, 7 of refresh_from_descent) for pruning.The address of the parent and children can be cachedto save the cost of DHT lookup.For the delegate procedure, the system �rst �nds

the objects not having been stored (line 1). Then itchecks whether it is necessary to delegate some recordsto the two child nodes (line 2). There can be di�erentstrategies to determine this. For example, whether thelocal storage for this pre�x exceeds a certain amount,or whether there have been a number of records olderthan a pre-con�gured time. If a delegation is required,we select the earliest α ∗ objects.count (0 < α ≤ 1)objects indexed at this gateway and delegate them tothe two child nodes. The delegation is similar to theFIFO cache replacement policy. This is based on theobservation that the latest records are more likely to

Algorithm : IndexInput: pre�x p with length Lp and the set of

objects belong to the group hash(p)Output: None (the algorithm only updates indexinginformation)1: update the index for objects local.objects ∩ objects2: objects′ ← local.objects− objects

// get the set of objects which are not stored locally3: if objects′ is not empty4: refresh_from_ascent(objects′, p)5: refresh_from_descent(objects′, p)6: end if

7: delegate(objects)

refresh_from_ascentInput: pre�x p with length Lp and the set of

objects belong to the group hash(p)Output: None (the algorithm refresh local index withascents)1: i← 12: do

3: p′ ← p.substring(1,Lp − i)// get substring of p from the 1st character to the

(Lp − 1)st one4: objects′ ← filter(objects, p′)5: if objects′ = Φ break

6: update index with information from gateway node forp′

the gateway node for p′ also updates its index7: i← i+ 18: while there exists gateway node for pre�x p′ and i ≤Lp − Lmin

refresh_from_descentInput: pre�x p with length Lp and the set of

objects belong to the group hash(p)Output: None (this algorithm refresh local index withdescents)1: objects′ ← filter(objects, p + `0')2: if objects′ is not empty3: update index with information from gateway node for

p + `0'the gateway node for p+ ‘0′ also updates its index

4: refresh_from_descent(objects, p + `0')5: end if

6: objects′ ← filter(objects, p + `1')7: if objects′ is not empty8: update index with information from gateway node for

p + `1'the gateway node for p+ ‘1′ also updates its index

9: refresh_from_descent(objects, p + `1')10: end if

delegateInput: pre�x p with length Lp and the set of

objects belong to the group hash(p)Output: None (the algorithm only updates indexinginformation)1: objects′ ← objects not stored in any node in the tree2: if delegation is required3: select α ∗ objects′.count earliest local indexing records

as objects′′

4: delegate objects′′ to the two children according to theLp bit of their ids

FIGURE 7. Algorithm for Indexing a Group of Objects



be read and updated in the near future. Here α is aglobal con�guration to control the number of objectsto be delegated. It should be noted that the splitting-merging process is trivial and we have not included thedetails of the algorithm.

4.3.1. Tradeo� between Indexing Performance and

Load Balancing

Lp is the key factor in both indexing performanceand load balancing. It is hard to satisfy these tworequirements at the same time.On the one hand, Lp determines the number of groups

(which is 2Lp). When Lp is small, the number ofgroups is small. Consequently, the bandwidth usagefor indexing is small. On the other hand, according toEquation 4, smaller number of groups causes workloadto be less well balanced. Equation 7 gives the minimumvalue of Lp which guarantees almost all nodes areassigned indexes to maintain. In theory, it is theoptimal choice. We will prove this by experiments inSection 6.In practical applications, further more factors a�ect

the choice of Lp. For example, in a network where thevolume of data is high (e.g., supply chain networks)and indexing performance is more important, it isreasonable to decrease Lp to save more bandwidth costs.In a network where number of nodes is high, and thevolume of data is relatively low (e.g., tra�c monitoringnetworks), larger Lp balances the workload distributionbetter.

4.4. Index Lookup Algorithm

The object lookup algorithm is straightforward. We�rst try to �nd the object in the gateway node forthe pre�x of current length. If it cannot be found, weconduct a bidirectional linear search starting from thegateway node in the tree. It should be noted that thesearch is directed by the id of the object, so the searchdoes not need to traverse the whole tree.

4.5. Algorithm Analysis

The indexing process consists of three phases: namelygrouping, routing and index-persisting. The groupingphase simply scans the No new objects. Its complexityis Θ(No).Chord routing takes O(log2Nn) hops with high

probability. To route the objects in 2Lp groups will takeO(2Lp log2Nn) hops in total. When objects are routedindividually, it takes O(No log2Nn) hops. Since thenumber of received object No can be very large, while2Lp = Nn log2Nn is relatively small, our grouping indexalgorithm can signi�cantly decrease the P2P routingcost.However, the index-persisting phase is complicated

because of the Data Triangle structure and the possibletrees extended from it. In the best scenario, all the

A B C

O O

GatewayNode

t1: o is read at node A

indexed, IOP updated

t2: o is missed at node B

t3: o is read at node B

indexed, IOP updated

……

t1: o is at node A

t2: o is at node A

t3: o is at node C

……

FIGURE 8. Example of Missing Reading

indexing information for new objects are stored in thegateway node, then the persisting will only involve locallookup and storage. The complexity is O(No) in themeasure of database operations.In the worst scenario, all the new objects have their

history indexes stored in either ascent nodes or descentnodes, or all the objects are new to the whole networkthus there is no historical information in the tree (thenthe tree has to be fully traversed). Index persisting mayinvolve both refreshing (i.e., refresh_from_ascentand refresh_from_descent) and delegation. Supposethat the tree is complete binary and its height4 is h, andall the historical indexing information is stored either inthe deepest leaves or in the root. Therefore, it requiresat most h DHT lookup to obtain the information. Sothe complexity of the worst case of the index persistingis O(h∗No). However, as the tree will be split or mergedwhen pre�x length changes, its height is well controlled.In most cases h is 1 (only children) or 2 (parent andchildren). The two trees in Figure 6 illustrates thesetwo situations respectively. Then the index persistingphase can be done in constant time.

4.6. Managing Uncertainties

Due to limitations of the RFID hardware anddynamic network environments, the recorded data donot perfectly map the real movements of objects.Uncertainty is a very broad topic and it is not asingle layer problem. For example, research thataddresses uncertainties spans from data storage anddata modeling to query processing and knowledgeengineering [12]. In this section, we brie�y discuss thetypical uncertainty problems in PeerTrack and how theyare handled.

4.6.1. Hardware Faults

A reader may miss identifying an object or a temporalmalfunction of a device may cause a systematic error in

4The height of a tree is de�ned as the number of edges fromthe root to the deepest leaf.


10 Y. Wu et al.

A B

O

Secondary

Gateway

Node

Primary

Gateway

Node

h1(o) h2(o)

Object movement Index

FIGURE 9. Replication of Index

events generated at a node. A missed tag read results inno data collected since data, such as the identi�er storedon the tag, is not captured by the reader. Figure 8shows an example of this scenario. When object oarrives at node B, it fails to be read by the reader.Consequently, the record at the corresponding gatewaynode is outdated. When object o arrives at node C, therecord is corrected. A missing reading is a fatal andunrecoverable problem to all RFID systems. In thisexample, object o never appears at node B. PeerTrackhandles this situation and the outdated record at thegateway node is corrected as soon as the object movesto a new node.

4.6.2. Network Faults

A node may leave the network for various reasons,such as network or power outage, system failure etc.When this occurs, all data stored at that node becomeunavailable. A general approach for data recovery inP2P system is replication. In PeerTrack, we replicatethe RFID data and its index in di�erent ways.The index is replicated by using a secondary hash

function when choosing gateway node. This is depictedin Figure 9. In addition to the primary hash function(h1), a secondary hash function (h2) is used to choosea secondary gateway node. When the primary gatewaynode leaves the network, the system upgrades thesecondary one to primary and use it to maintain theindex. The approach is �exible in the sense that if athird replication is necessary, the only change is to adda third hash function.The IOP information is replicated at a node chosen by

a uni�ed hash function h′, i.e., the node that keeps thereplication of node A, A′, is chosen by h′(A). Figure 10shows an example of using IOP replications to answertracing queries. When B is o�ine, B′ will be queried.As a result, the routing path of the query becomesGateway → B′ → A, instead of Gateway → B → A.

A B

O

A' B'

IOP of node A is

replicated at h'(A) = A'

Gateway

Node

B is offlinex

IOP of node B is

replicated at h'(B) = B'

o.from = A

o.from = A

o is at node B now

Query: get o's history

Object movement Replication

Data Query Routing Path

FIGURE 10. Replication of IOP

5. QUERY PROCESSING

In this section, we introduce the algorithms to processtwo kinds of tracing queries, namely item level andrange queries.

5.1. Item Level Queries

To perform the trace function TR (see Section 2.2),we need to �nd the gateway node of the given object.From the gateway node, we can discover the nodewhere the object is seen for the last time and simplytraverse back using the IOP information. In addition,as mentioned in Section 3, if any node along the routefrom the query requesting node to the gateway nodehas the information of the object, the trace query canbe processed from this node by traversing backward andforward using the IOP. In this way, we do not have to�nd the gateway node of the object. With TR solved, itbecomes straightforward to process other kinds of itemlevel queries, include but not limited to:

• Dwell time query. For example, �How long did o1stay at node n1?�.

• Arrival time query. For example, �When did o1arrive at node n1?�.

• Source node query. For example, �Where did o1come from before it arrived at node n1?�.

5.2. Range Queries

Range queries are used to �nd out the aggregatedinformation for a given criteria. For example, �Howmany objects moved from node n1 to node n2 duringlast month?�. In our architecture, with the help of IOP,this query can be processed at either node n1 or n2,despite the fact that it involves two nodes. Suppose thequery is being processed at n1:



SELECT COUNT(*) FROM RECORDS

WHERE TO = n2 AND

END_TIME BETWEEN `2011-04-01' AND `2011-04-30'

This query is segment-oriented. A more complexrange query can be path-oriented. For example, �Howmany objects moved along the path n1 → n2 →... → nk during last month?�. With our solution, thisquery can also be e�ciently answered. First, the nodeinitiating the query �nds the nodes involved in thequery via Chord lookup interface. Then it asks the nodeni (i from 2 to k) to answer the query: �Which objectsmoved from node ni−1 during last month?�. This querycan be easily answered. Finally, the results from thek − 1 nodes are intersected to get the result.Other range queries regarding either path or segment

can be handled in a similar way.

5.3. Algorithm Analysis

5.3.1. Item Level Queries

There are two cases for query processing according towhere the L function (see Section 2.2) is answered,which will be discussed separately in the following.

Gateway. If there is no relevant node along therouting path from the node where the query is issuedto the gateway node, then the locate is answered by thegateway node. The routing process takes O(log2Nn)hops. At the gateway node, there is a high probabilitythat the lookup algorithm in takes O(1) hops.

Intermediate Node. If during the routing, a nodealong the routing path has the information for thequeried object, the routing will be terminated andthe intermediate node will start to process the query.Although the routing cost remains O(log2Nn), theconstant coe�cient is reduced.After the node n where the query can be answered

is found, the complexity of query processing is solelybased on the type of the query. For example, L takesO(l) (l is the length of the object's lifetime trajectory)DHT lookups in the worst case (when the object was atthe end or start of the trace for the given time) and O(1)for the best case (when the object was at n for the giventime). However, for a trace query TR that requires thelifetime trajectory to be found, it will always take O(l).

5.3.2. Range Queries

For range queries that only involve segments, the costis solely based on the query processing method of localdatabase. There is no network cost, except when thequery is initiated from a third party node (This rarelyhappens), a Chord lookup is used to �rst �nd the nodeinvolved which cost log2Nn hops.For range queries that involve paths, the length of

paths determines the cost. Suppose the length of a pathinvolved in a query is l, l ∗ log2Nn hops are used to �ndall the involved nodes. All the other calculations are

FIGURE 11. Scalability of Indexing on Data Volume

done locally in either memory or DBMS, which can beignored compared to the cost of P2P lookup.

6. EXPERIMENTS

Our experiments focused on three aspects. Firstly, weevaluated the performance of the indexing algorithmsand compared the scalability of the individual indexingapproach and the enhanced group indexing approach.Secondly, we studied the performance of queryprocessing using the techniques proposed in this paperand compared it with that of centralized architecture.Finally, we examined the e�ect of the pre�x length(Lp), on the proposed group indexing algorithm, onload balancing and indexing performance of traceablenetworks.We used the open source P2P simulator OverSim [20]

in the experiments. In our tests, the maximum numberof nodes is 512 and the maximum number of objectsat each node is 5,000, corresponding to the limit ofour experimental environment. All experiments wereconducted on an Intel Core 2 Quad 2.4GHz system with4GB of RAM.

6.1. Indexing

The indexing cost, measured by the total volume ofmessages transferred over the network, is consideredto study the feasibility of the proposed group indexingalgorithm and related data structures. According tothe analysis in Section 4.5, the indexing cost dependson the size of the network and height of the pre�x tree.We �rst conducted the test with a network of 512 nodesand generated a speci�c number of objects at each node.We ran the test 10 times and for the ith (1 ≤ i ≤ 10)time, the number of objects generated at each node is500∗i. To simulate the movement of objects, 10% of thelocal objects at each node were moved along a trace of10 nodes. We measured both the group and individualindexing algorithms. The result is depicted in Figure 11.


12 Y. Wu et al.

FIGURE 12. Scalability of Indexing on Network Size

In order to see whether the objects moving in group orindividually would a�ect the performance, we measuredthe performance of group indexing algorithm for the twocases: i) objects moved to other nodes individually, ii)objects moved to other nodes in groups.From Figure 11 we can see that, when the data

volume is not high (e.g., 500), the group indexingalgorithm does not show much of an advantage. This isbecause that when the number of objects is small, mostof the groups contain only one or two objects. Thenumber of groups (2Lp) is close to the number of theobjects (No). Thus the group indexing algorithm costsalmost the same as the individual indexing algorithm.However, with increasing data volume, the indexing costof the group indexing algorithm increases much slowerthan the individual indexing algorithm, because data isgrouped and compressed before sent to gateway node.It is also clear that when the objects move in groups,

the indexing cost is further reduced. Even when thereare fewer objects, the performance is much better. Thisis because that when objects move in groups, it is highlypossible that most objects fall into the same window(Section 4.1), so in that window, the number of objectsis high.We also studied the performance of the indexing

algorithms under di�erent network sizes. In theexperiment, the number of objects generated at eachnode is 5,000 and the network size varies from 64,128, 256, to 512. The result in Figure 12 showsthat with the increase in network size, the indexingcost for the individual indexing algorithm increaseslinearly while the group indexing algorithm shows a sub-linear pattern. The reason is that when the networksize increases, Lp increases as well, which leads to anincrease in the number of groups. However this increaseis slower than the increase of the number of nodes itself.Generally, the performance degrades when the ratio ofdata volume to network size becomes small.However in reality, particularly large-scale applica-

tions, there are much more objects than nodes. Clearly,these applications can take advantage of the bene�ts

Query

Q1 Where has object oi been?

Q2 Where is object oi?

Q3 How many objects moved from node ni to node nj?

Q4 How many objects moved along the path n1 → n2 →... → nk during last month?

TABLE 2. Testing Queries

brought by our proposed group indexing approach.Figure 12 also shows that the indexing costs less when

the objects move in groups, because in this case, objectsare more likely to fall into the same capturing window,so that they can be grouped by the grouping algorithm.

6.2. Query Processing

In this experiment, we studied the performance of ouroverall approach for query processing under di�erentnetwork sizes and di�erent data volumes. We tested theperformance of query processing using a set of queries,including both item level and range ones, shown inTable 2.We also compared the performance of the P2P

architecture and a centralized approach. For the P2Papproach, we added 5ms (typical network latency of T1line) as the network latency for each network query. Forthe latter, we used the model proposed in [21] to buildthe same data in a centralized MySQL database.Figure 13 shows the query processing time in

networks of di�erent sizes. For each query in Table 2,we ran it 100 times with di�erent randomly chosenobjects/nodes. The results presented in this article arethe average of the 100 tests. All the queries are initiatedfrom a node involved in them because this is a typicalcase.From the �gure we can see that the query processing

time increases slowly for the P2P approach, butincreases sharply for the centralized one. Since the dataat each node in di�erent settings (number of nodes inthe network) is of the same amount, the local processingtime does not change. Then, the query time of Q1, Q2and Q4 in P2P network is proportional to the logarithmof the size of the network. For Q3 (Figure 13 (c)), thecost for Q3 almost stays constant for P2P approach.This is because that there is no P2P lookup involved.However, the cost for centralized approach increaseswith the number of nodes.For the centralized approach, because all the data

is stored in the same database, when the size of thenetwork increases, the amount of the data in the centraldatabase increases too. The time for querying isrelevant (ultra-linear because of the join queries used) tothe size of the database, which is proportional to the sizeof the network when the number of objects generatedat each node is �xed.



(a) (b)

(c) (d)

FIGURE 13. Query Processing Performance on Network Size

(a) (b)

(c) (d)

FIGURE 14. Query Processing Performance on Data Volume

We can also see that when there are less nodes,centralized approach outperforms P2P one. However,when the number of nodes and records in the networkreaches certain amount, which is typical in large scale

RFID networks, it will run more slowly than the P2Papproach.

Figure 14 shows the query processing time fordi�erent data volumes (the manner in which we


14 Y. Wu et al.

generate di�erent volumes of data is identical to thatin Section 6.1). Similar result was obtained. Theprocessing time for Q1, Q2 and Q4 increases much moreslowly in P2P approach than in centralized one. Theprocessing time for Q3 in P2P approach increases too(Figure 14 (c)). This is because the volume of data ateach node increases.The high scalability shown in the experimental results

is due to the IOP information in our design.

6.3. The E�ect of Lp

As discussed in Section 4.3, the choice of Lp signi�cantlya�ects the performance and load balance of traceablenetworks. In this experiment, we studied the di�erentschemes of Lp and tested their corresponding loadbalancing capabilities. Figure 15 (a) shows the result forthe three di�erent schemes (i.e., Scheme 1: Lp=log2Nn,Scheme 2: Lp=log2Nn + log2 log2Nn, and Scheme 3:Lp=2 log2Nn). Scheme 2 is the one that we havechosen for our system (which is also the scheme usedin all other experiments). We studied Scheme 1 andScheme 3 in this experiment as they are asymptoticallysmaller/bigger than Scheme 2, respectively.We illustrate the load balance by showing the load

percentage (i.e., the number of objects handled by agiven set of nodes divided by the total number ofobjects) for a given node percentage (i.e., the numberof nodes in the set divided by the total number ofnodes). A well balanced scheme should yield a linearrelationship between the load percentage and the nodepercentage (where y = x, i.e., diagonal), meaning thateach node receives the same number of objects to index.The farther the curve is away from the diagonal, theworse it is.As we can see from Figure 15 (a), when Scheme 1 was

chosen as the length of pre�x Lp, the load is not wellbalanced. The curve is far away from the diagonal andshows some saltations. Scheme 3 performs best amongthe three because it is very close to the diagonal, whichimplies that the load is well balanced. However, Scheme3 makes Lp too long and the number of groups becomestoo big, leading to less objects in each group. Thissigni�cantly a�ects the indexing cost (see Figure 15(b)). Overall, from our study, Scheme 2 provides agood choice for Lp, with which the work load is alsowell balanced.We also tested the performance of the three schemes

with di�erent network sizes at �xed data volume (5,000objects at each node). Figure 15 (b) shows that amongthe three, Scheme 1 is the most e�cient one and Scheme3 is the worst (its indexing cost is the square of thenumber of nodes). However, there is a tradeo� betweenindexing performance and load balancing. To improvethe former, Lp should be smaller, which leads to poorload balancing. To improve the latter, Lp should bebigger to ensure that all nodes have groups to beresponsible for, which leads to a higher indexing cost.

(a)

(b)

FIGURE 15. The E�ect of Pre�x Length

Scheme 2 shows acceptable results on both indexingperformance and load balancing. In reality, we canchoose di�erent schemes for di�erent scenarios. Forexample, if the performance of an application is veryimportant, Scheme 1 will be a good choice.

7. AN EXAMPLE APPLICATION

The PeerTrack architecture can be adopted in manyapplications. In this section, we introduce an RFID-based asset management system, AMS, which wassuccessfully developed and deployed in a local companythat provides linen services for over 200 customersacross South Australia. RFID tags are a�xedto trolleys, which are moved among the companywarehouses and its customers to deliver clean linenor collect the one to be cleaned. AMS aims torealize a fully automated tracking and tracing servicewith the capability of monitoring and controlling thecompany's logistical operations in real-time. On the topof the PeerTrack architecture, we integrated an eventprocessing module and a rule engine. The companymanagers get noti�ed via email or SMS based on RFIDevents and pre-de�ned rules. Figure 16 shows the wholearchitecture, including the Tracking Engine and theRule Engine, as well as the the PeerTrack Platform.



FIGURE 16. The System Architecture

FIGURE 17. AMS Monitoring Software

It should be noted that the architecture can be easilyadapted into many other applications such as supplychain management and tra�c control system.Figure 17 shows a screenshot of AMS. According to

the remote events or local RFID readings through theunderlying PeerTrack system, AMS displays real-timeinformation about the status of the company assets(�Real-time Order/Objects Status Monitoring� panel).AMS features a Tracking Engine (�Order/ObjectsSearch Tool�) which is implemented by using thetracking algorithms proposed in this article. Themanagers can easily track the current status of a speci�casset or its complete history.

8. RELATED WORK

Successfully tracing objects in a distributed environ-ment is not a single layer problem. In this section, weoverview the relevant techniques to our work.The �rst step of modeling moving objects is to

abstract the basic elements such as time, region andvelocity. [22] introduces a data model called MOST(Moving Objects Spatio-Temporal) for databases withdynamic attributes, i.e., attributes that changecontinuously as a function of time. A languagecalled Future Temporal Logic (FTL) has been designedto support queries for dynamic attributes. Thiswork models the moving objects and their attributesin a generic way that can be easily adapted intovarious applications in di�erent domains. [23] proposesa similar model with a method to represent thecontinuous attributes such as time and space discretely.Later works mostly use the same or similar idea.

With the elements modeled, it is possible to answerbasic queries such as location of an object at a certaintime. In order to answer more complicated queries, suchas an aggregate query �how many objects are in regionR now?� and a future state query �where will the objectO be after one hour?�, various complex and domain-speci�c models have been developed. For example,in [24], adaptive multi-dimension histogram (AMH) isused in answering aggregate queries about past, presentand future. It can estimate the number of objects thatwill satisfy some spatial condition for a near future time.It does not require the knowledge of velocity vectors,but uses an exponential smoothing based stochasticapproach. Since this kind of work focuses on aggregatequeries, it does not address the single-instance queries.In [25], the moving objects are associated with fourvariables (starting time, starting location, destination,initial velocity) in order to predict their future locations.There are some other works focusing on variousproblems of modeling and querying moving objects byemploying di�erent techniques. However, they sharesome common characteristics. Firstly, space and timeare modeled as continuous attributes, although theunderlying representation in physical storage can bediscrete. Secondly, most of the works de�ne region orsimilar concepts that cover a �nite area in a continuousin�nite domain in order to answer range queries.Index is often used to quickly answer range queries

for a speci�c attribute. For example, if the spaceis divided into cells and objects are indexed by cell,it is easy to answer queries such as �how many orwhich objects are in cell c?�. However, dividingthe space into �xed-size cells does not work well indynamic environments because the boundary of thespace must be decided in advance. Early works,including R-Tree, R∗-tree, TR-tree, TB-tree, TPR-tree, TPR∗ R-tree and some other similar trees [26],dynamically decide which point or region to index andoptimize the indexing process in di�erent ways. [27]considers streaming data that require frequent updatesand proposes an e�cient B+-Tree based indexingmethod which represents the moving-object locationsas timestamped vectors. SINA (Scalable INcrementalhash-based Algorithm) [28] supports concurrent andcontinuous spatial-temporal queries by abstracting the


16 Y. Wu et al.

continuous queries as a spatial join between a set ofmoving objects and a set of moving queries. [29]indexes the predicted trajectories to quickly predictfuture locations of moving objects. Many recent workssuch as [30, 31, 32, 33, 34] focus on other speci�cproblems and provide corresponding solutions. Theseindex methods all operate in centralized environments.

In recent years, some generic peer-based databasemanagement systems [8] have been proposed to supportlarge volume data and complicated queries. PIER [35]presents a distributed query engine based on a P2P-based overlay network.However, PIER requires everydata tuple to be shared in the network, which brings twosigni�cant issues. Firstly, the amount of messages thatare sent for indexing are proportional to the numberof data tuples. It is problematic when the approachis used in large databases with millions or billions oftuples. Secondly, the data is made public in PIER thusthere is no privacy among participants. PISCES [36]identi�es a subset of data tuples to index based onsome criteria (e.g., query frequency, update frequency,indexing cost). This signi�cantly saves the cost ofindexing and storage. However, because its objectiveis to improve e�ciency of range queries, PISCES doesnot work well with single instance query. If the columnin the search criteria that is used to search for anindividual object is not indexed, the query has tobe �ooded into the whole network. Although thesegeneric solutions lays the foundation for peer-basedspatial databases, they are not dedicated to e�cientlymanage spatial data and answer spatial queries. [37]presents an analytical model to predict the cost of queryalgorithms based on query location, query size and themoving objects' distribution so that the �nal schemechosen to perform the query is optimal. [38] proposesa distributed hash technique to answer range queries,which scales well under certain assumption about thequery distributions. [39] introduces a middlewaredesign based on distributed R-tree and Quadtree tosupport both range and kNN (k Nearest Neighbors)queries.

The work presented in [40] extends Quadtree indexinto a P2P network to answer range queries e�cientlywhile keeping the load on the nodes in the networkwell balanced. [41] provides a linear yet distributedstructure that facilitates multiple search paths to bemixed together by sharing links. These distributedindex methods are all based on spatial elements suchas point, link or region thus they work well for rangequeries but not single-instance queries. In [42], theauthors proposed a generic model to deal with the eventmatching problem of content-based publish/subscribesystems over structured P2P overlays. This model isuseful in a distributed RFID system if it is event-driveninstead of query-driven.

9. CONCLUSION

Recent advances in technologies such as radio-frequencyidenti�cation (RFID) make automatic tracking andtracing possible in a wide range of applications.Unfortunately, realizing traceability applications inlarge-scale, distributed environments such as theemerging Internet of Things presents signi�cantchallenges due to their unique characteristics such aslarge volume of data and sovereignty of the participants.In this paper, we have presented a generic approach thatenables applications to share traceability data acrossindependent enterprises in a P2P fashion. We builtour approach on top of a DHT-based overlay network.Objects are indexed at deterministic gateway nodesthat are responsible for updating objects' status atthe source and destination nodes for their movements.In this way, the IOP (information of object path)properties of objects are established, which are criticalto support e�cient processing of traceability queries.To reduce the indexing overhead from massive volumesof data in large-scale traceability applications, wefurther proposed an enhanced group-based indexingapproach. Extensive experiments showed the viabilityand scalability of our approach.We view our work presented in this paper as a �rst

step towards e�cient tracking and tracing of objectsin the Internet of Things. Our ongoing work includesfurther performance study of the proposed techniques.Another important direction for future work is toadd capabilities for predicting future status of objects.This typically involves overcoming uncertainty issuesintroduced by traceability applications (e.g., incompleteand noisy data) using statistical and probabilistictechniques.

ACKNOWLEDGEMENTS

This research has been supported by the AustralianResearch Council Discovery Grant DP0878917 andLinkage Grant LP100200114.

REFERENCES

[1] Gershenfeld, N., Krikorian, R., and Cohen, D. (2004)The Internet of Things. Scienti�c American, 291, 76�81.

[2] Ranasinghe, D., Sheng, Q. Z., and Zeadally, S.(2010) Unique Radio Innovation for the 21st Century:Building Scalable and Global RFID Networks. Springer,Berlin, Germany.

[3] Welbourne, E., Battle, L., Cole, G., Gould, K., Rector,K., Raymer, S., Balazinska, M., and Borriello, G.(2009) Building the Internet of Things Using RFID:The RFID Ecosystem Experience. IEEE InternetComputing, 13, 48�55.

[4] Agrawal, R., Cheung, A., Kailing, K., and Schönauer,S. (2006) Towards Traceability Across Sovereign, Dis-tributed RFID Databases. Proceedings of IDEAS'06,



India, pp. 174�184. IEEE Computer Society, LosAlamitos, CA.

[5] Franklin et. al, M. J. (2005) Design Considerations forHigh Fan-in Systems: The HiFi Approach. Proceedingsof CIDR'05, Asilomar, USA, pp. 290�304.

[6] Sheng, Q. Z., Li, X., and Zeadally, S. (2008) EnablingNext-Generation RFID Applications: Solutions andChallenges. IEEE Computer, 41, 21�28.

[7] Wang, F., Liu, S., and Liu, P. (2009) Complex RFIDEvent Processing. The VLDB Journal, 18, 913�931.

[8] Hose, K., Roth, A., Zeitz, A., Sattler, K., andNaumann, F. (2008) A Research Agenda for QueryProcessing in Large-Scale Peer Data ManagementSystems. Information Systems, 33, 597�610.

[9] Wu, S., Jiang, S., Ooi, B. C., and Tan, K.-L. (2009)Distributed Online Aggregations. PVLDB, 2, 443�454.

[10] Cheung, A., Kailing, K., and Schönauer, S. (2007)Theseos: A Query Engine for Traceability AcrossSovereign, Distributed RFID Databases. Proceedingsof ICDE'07, Turkey, pp. 1495�1496. IEEE ComputerSociety, Los Alamitos, CA.

[11] Balakrishnan, H., Kaashoek, M., Karger, D., Morris,R., and Stoica, I. (2003) Looking up Data in P2PSystems. Communications of the ACM, 46, 43�48.

[12] Wu, Y., Ranasinghe, D. C., Sheng, Q. Z., Zeadally, S.,and Yu, J. (2011) RFID Enabled Traceability Networks:A Survey. Distribured and Parallel Databases, 29, 397�443.

[13] Chen, H., Ku, W., Wang, H., and Sun, M.(2010) Leveraging Spatio-Temporal Redundancy forRFID Data Cleansing. Proceedings of SIGMOD'10,Indianapolis, USA, pp. 51�62. ACM, New York, USA.

[14] Gonzalez, H., Han, J., and Shen, X. (2007) Cost-Conscious Cleaning of Massive RFID Data Sets.Proceedings of ICDE'07, Istanbul, Turkey, April, pp.1268�1272. IEEE Computer Society, Los Alamitos, CA.

[15] Je�ery, S., Garofalakis, M., and Franklin, M.(2006) Adaptive Cleaning for RFID Data Streams.Proceedings of VLDB'06, Seoul, Korea, September.VLDB Endowment.

[16] Sheng, Q. Z., Wu, Y., and Ranasinghe, D. (2010) En-abling Scalable RFID Traceability Networks. Proceed-ings of AINA'10, Perth, Australia, April, pp. 1061�1068. IEEE Computer Society, Los Alamitos, CA.

[17] Stoica et. al, I. (2001) Chord: A Scalable Peer-to-PeerLookup Service for Internet Applications. Proceedingsof SIGCOMM'01, San Diego, USA. ACM, New York,USA.

[18] Jelasity, M. and Montresor, A. (2004) Epidemic-StyleProactive Aggregation in Large Overlay Networks.Proceedings of ICDCS'04, Tokyo, Japan, pp. 102�109.IEEE Computer Society, Washington DC, USA.

[19] Ramabhadran, S., Ratnasamy, S., Hellerstein, J. M.,and Shenker, S. (2004) Brief Announcement: Pre�xHash Tree. Proceedings of PODC'04, St. John's,Canada, pp. 368�368. ACM, New York, USA.

[20] Baumgart, I., Heep, B., and Krause, S. (2007) OverSim:A Flexible Overlay Network Simulation Framework.Proceedings of GI'07, Anchorage, USA, pp. 79 �84.IEEE Computer Society, Los Alamitos, CA.

[21] Wang, F. and Liu, P. (2005) Temporal Management ofRFID Data. Proceedings of VLDB'05, Trondheim, Nor-way, September, pp. 1128�1139. VLDB Endowment.

[22] Sistla, A., Wolfson, O., Chamberlain, S., and Dao,S. (1997) Modeling and Querying Moving Objects.Proceedings of ICDE'97, Pennsylvania, USA, pp. 422�432. IEEE Computer Society, Washington, DC, USA.

[23] Erwig, M., Güting, R. H., Schneider, M., and Vazir-giannis, M. (1998) Abstract and Discrete Modeling ofSpatio-Temporal Data Types. Proceedings of GIS'98,Washington, D.C., United States, pp. 131�136. ACM,New York USA.

[24] Sun, J., Papadias, D., Tao, Y., and Liu, B. (2004)Querying about the Past, the Present, and the Future inSpatio-Temporal Databases. Proceedings of ICDE'04,Boston, USA, pp. 202�211. IEEE Computer Society,Washington, DC, USA.

[25] Chon, H., Agrawal, D., and Abbadi, A. (2001) Storageand Retrieval of Moving Objects. Proceedings ofMDM'01, Hong Kong, China, pp. 173�184. Springer-Verlag, London, UK.

[26] Mokbel, M. F., Ghanem, T. M., and Aref, W. G.(2003) Spatio-Temporal Access Methods. IEEE DataEngineering Bulletin, 26, 40�49.

[27] Jensen, C. S., Lin, D., and Ooi, B. (2004) Query andUpdate E�cient B+-tree Based Indexing of MovingObjects. Proceedings of VLDB'04, Toronto, Canada,pp. 768�779. VLDB Endowment.

[28] Mokbel, M. F., Xiong, X., and Aref, W. G. (2004)SINA: Scalable Incremental Processing of ContinuousQueries in Spatio-Temporal Databases. Proceedings ofSIGMOD'04, Paris, France, pp. 623�634. ACM, NewYork, USA.

[29] Patel, J. M., Chen, Y., and Chakka, V. (2004)STRIPES: An E�cient Index for Predicted Trajecto-ries. Proceedings of SIGMOD'04, Paris, France, pp.635�646. ACM, New York, USA.

[30] Botea, V., Mallett, D., Nascimento, M. A., and Sander,J. (2008) PIST: An E�cient and Practical IndexingTechnique for Historical Spatio-Temporal Point Data.Geoinformatica, 12, 143�168.

[31] Gao, Y., Chen, G., Li, Q., Zheng, B., and Li, C.(2008) Processing Mutual Nearest Neighbor Queries forMoving Object Trajectories. Proceedings of MDM'08,Beijing, China, pp. 2176�2195. Elsevier Science Inc.,New York, USA.

[32] Lange, R., Dürr, F., and Rothermel, K. (2008) ScalableProcessing of Trajectory-based Queries in Space-partitioned Moving Objects Databases. Proceedings ofGIS'08, Irvine, USA, pp. 31:1�31:10. ACM, New York,USA.

[33] Tzoumas, K., Yiu, M., and Jensen, C. S. (2009)Workload-aware Indexing of Continuously MovingObjects. PVLDB, 2, 1186�1197.

[34] Zhang, M., Chen, S., Jensen, C. S., Ooi, B.,and Zhang, Z. (2009) E�ectively Indexing UncertainMoving Objects for Predictive Queries. PVLDB, 2,1198�1209.

[35] Huebsch, R., Hellerstein, J. M., Lanham, N., Loo,B. T., Shenker, S., and Stoica, I. (2003) Querying theinternet with PIER. Proceedings of VLDB'03, pp. 321�332. VLDB Endowment.

[36] Wu, S., Li, J., Ooi, B. C., and Tan, K.-L. (2008) Just-in-time Query Retrieval over Partially Indexed Data onStructured P2P Overlays. Proceedings of SIGMOD'08,


18 Y. Wu et al.

Vancouver, Canada, pp. 279�290. ACM, New YorkUSA.

[37] Meka, A. and Singh, A. (2005) DIST: A DistributedSpatio-Temporal Index Structure for Sensor Networks.Proceedings of CIKM'05, Bremen, Germany, pp. 139�146. ACM, New York, USA.

[38] Li, X., Kim, Y., Govindan, R., and Hong, W. (2003)Multi-dimensional Range Queries in Sensor Networks.Proceedings of SenSys'03, Los Angeles, USA, pp. 63�75.ACM, New York, USA.

[39] Zimmermann, R., Ku, W.-S., and Chu, W.-C. E�cientQuery Routing in Distributed Spatial Databases.Proceedings of GIS'04, Washington DC, USA, pp. 176�183. ACM, New York, USA.

[40] Tanin, E., Harwood, A., and Samet, H. (2007) Using aDistributed Quadtree Index in Peer-to-Peer Networks.The VLDB Journal, 16, 165�178.

[41] Lee, W. and Zheng, B. (2005) DSI: A Fully DistributedSpatial Index for Location-Based Wireless BroadcastServices. Proceedings of ICDCS'05, USA, pp. 349�358.IEEE Computer Society, Los Alamitos, CA.

[42] Zhang, S., Wang, J., Shen, R., and Xu, J. (2010) To-wards Building E�cient Content-Based Publish/Sub-scribe Systems over Structured P2P Overlays. Proceed-ings of ICPP'10, pp. 258�266. IEEE Computer Society,Washington, DC, USA.


facilitating e cient object tracking in large-scale ...qsheng/papers/compj2011.pdffacilitating e...

Documents