Ad Hoc Networks 19 (2014) 28–42

Contents lists available at ScienceDirect

Ad Hoc Networks

journal homepage: www.elsevier .com/locate /adhoc

QoS-aware distributed adaptive cooperative routing in wirelesssensor networks

http://dx.doi.org/10.1016/j.adhoc.2014.02.0021570-8705/� 2014 Elsevier B.V. All rights reserved.

⇑ Corresponding author. Tel.: +82 10 3409 4112; fax: +82 31 204 9082.E-mail address: cshong@khu.ac.kr (C.S. Hong).

Md. Abdur Razzaque a, Mohammad Helal Uddin Ahmed b, Choong Seon Hong c,⇑, Sungwon Lee c

a Department of Computer Science and Engineering, University of Dhaka, Bangladeshb Department of Management Information Systems, University of Dhaka, Bangladeshc Department of Computer Engineering, Kyung Hee University, Republic of Korea

a r t i c l e i n f o

Article history:Received 1 April 2013Received in revised form 7 December 2013Accepted 4 February 2014Available online 24 February 2014

Keywords:Quality-of-serviceCooperative routingWireless sensor networksEnergy efficiencyLexicographic optimizationReinforcement learning

a b s t r a c t

In this paper, we investigate a cooperative routing problem in time-varying WirelessSensor Networks (WSNs) targeting the achievement of quality-of-service guarantees indelay and reliability domains. We develop a distributed adaptive cooperative routing pro-tocol, called DACR, that exploits cooperative communication on top of delay- and energy-aware end-to-end routes and optimizes the trade-off between the reliability and delaythrough Lexicographic Optimization at each hop. We employ a lightweight reinforcementlearning method to update the routing nodes with knowledge of expected performancesthat could be provided by the candidate relay nodes, helping to determine the optimalrelay with the least overhead. The decision of selecting a transmission mode (i.e., director relayed transmission) at each hop is taken adaptively so that the reliability is maxi-mized. The performances of our DACR have been evaluated through ns-2 simulations fora wide range of link failure rates and data traffic generation rates and the results show thatthe DACR outperforms a number of state-of-the-art protocols.

� 2014 Elsevier B.V. All rights reserved.

1. Introduction

In a number of mission-critical applications of WirelessSensor Networks (WSNs), such as battlefield surveillance,disaster response, wildlife monitoring, radioactive radia-tion monitoring, and so on, the sensed data packets haveto meet certain quality-of-service (QoS) levels in multipledomains (e.g., end-to-end delay, reliability, i.e., packetdelivery ratio, network resources, and so on) [1,2]. Forexample, in radioactive radiation monitoring application,the sensed data packets carrying radioactive leakage detec-tion information need to be delivered to the control centerwithin a predefined limited time while maintaining acertain level of packet delivery ratio for reliable event

perception. However, due to time-varying wireless chan-nel, dynamic network topology, and severe constraints onenergy and computation power of tiny sensor nodes,achieving these QoS requirements in WSNs is a challengingproblem.

In this paper, we exploit cooperative communications[3] to investigate QoS provisioning for mission-criticalapplications of WSNs, in the delay and reliability domains.The applicability of cooperative communication in re-source-constrained WSNs is advantageous for the follow-ing reasons: (i) it exploits the spatial diversity gain inmultiuser wireless systems to combat the effects of chan-nel fading, (ii) it does not necessitate multiple antennasat each node; and (iii) it reduces energy consumptionwhile improving network performance [4–6]. Cooperativerouting, a routing method that uses cooperative communi-cation, is effective for multihop WSNs because it involvesmore nodes in carrying data packets toward the

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 29

destination sink, thus increasing the energy-distributionamong the nodes [7,8]. The cooperation mechanism isthe key to the performance of cooperative routing systems;however, it is challenging to find the optimal cooperativepolicies, e.g., when to cooperate, how to cooperate andwith whom to cooperate, in a dynamic wireless networkenvironment.

Although there has been significant effort to studycooperative routing systems, little work has been doneon QoS provisioning in wireless networks exploiting suchsystems, especially in the context of achieving multiobjec-tive QoS services, e.g., end-to-end (e2e) delay, reliability,network lifetime, and so on. REER [9] investigated theproblem of reliable data delivery from many sensors tothe destination sink by exploiting advantages of geo-graphic and cooperative routing. A trade-off analysis be-tween reliability and energy-efficiency was also carriedout in that study. Backpressure-based control algorithmsin dynamic cooperative policies have been developed in[10] for delay-limited mobile ad hoc networks to achievethe time average reliability and energy-efficiency. How-ever, computing the optimal stationary, randomized policyexplicitly can be challenging and often impractical inWSNs, since it requires advance knowledge of arrival andchannel probabilities. A multi-agent reinforcementlearning based cooperative communication mechanism(MRL-CC) has been proposed for QoS provisioning in delayand reliability domains for WSNs in [11]. Each MRL-CCcandidate relay node maintains its Q-values and those ofits cooperative partners. When a packet is received by agroup of cooperative nodes, each node compares its ownQ-value with that of others; and the node that determinesit has the highest Q-value is elected to forwarding the datapacket to the adjacent cooperative nodes toward the sink.Thus, the reactive relay selection mechanism of MRL-CCnot only increases the overhead, but also fails to utilizethe appropriate network channel conditions, since theQ-value alone does not reflect the exact qualities (e.g.,delay, packet delivery ratio, energy-level) of the availablerelays. An extended version of this work is also presentedin [12] by the same authors with similar contributions.

In this paper, we present a distributed adaptive cooper-ative routing (DACR) protocol to achieve the QoS require-ments in the reliability and delay domains under thenode energy level constraint. In DACR, an AODV [13]-baseddelay- and energy-aware e2e route from a source to thedestination sink is created first, on which the data packetsare transmitted using either direct transmission or relayedtransmission mode. At each hop, the source of the linkadaptively chooses the transmission mode that maximizesthe per-hop reliability, given that the delay and energyconstraints are maintained. The optimal relay selection cri-teria of DACR are locally available at each routing node.These criteria are the expected delay and reliability valuesthat can be offered by a candidate relay node, if it is se-lected, and its residual energy level. Each routing nodeperiodically learns the aforementioned criteria for all thecandidate relays using a lightweight reinforcement learn-ing (RL) method [14]. It then executes a lexicographic opti-mization (LO) [17] algorithm (with two functions) todetermine the best relay proactively among the feasible

candidates. What follows, we summarize the contributionsof this work.

� We propose an adaptive cooperative routing protocol,which can be executed in a distributed manner withoutrequiring global channel state information (CSI) at eachrelay or at a central controller in the network, therebyreducing the required cooperation overhead.� We show that our hop-by-hop dynamic decisions about

cooperation effectively optimize the QoS performancesin the delay and reliability domains.� We show that proactive relay selection is more efficient

than reactive one in terms of trading-off the QoS perfor-mance improvements and operation overhead.� We also show that our energy-aware route discovery

and relay selection yield excellent energy-distributionamong the network nodes, thereby increasing the net-work lifetime.

The rest of this paper is organized as follows. In Section2, we describe the limitations of existing cooperative rout-ing algorithms in meeting QoS requirements of WSN appli-cations. In Section 3, we present the system model and theassumptions made in this work. The DACR architecture indescribed in detail in Section 4, and performance evalua-tions using Network Simulator-2 [18] are explained in Sec-tion 5. Finally, we conclude the paper in Section 6 and offerinsights for further works.

2. Related works

A large number of prior studies on cooperative commu-nication have investigated the problems of minimizingsum power [6,8,19], minimizing outage probability[20,21], meeting target throughput or SNR constraint[22–24] and references therein. It has been well demon-strated that the cooperative communication is effectivein combating the multiple fading effects in wireless net-works, and improving the network performance in termsof energy-efficiency, adaptivity, outage probability andnetwork throughput. For example, in minimum powercooperative routing (MPCR) system [6], the cooperativeroutes are constructed based on the Bellman–Ford shortestpath algorithm in terms of power usage and it has beenproved, through analysis, that MPCR algorithm can haveup to 37.64% power saving in random networks comparedto traditional routing systems.

Recently, many researchers have focused on the advan-tages of cooperative routing systems. [25] proposes twointerference-aware routing schemes for CDMA ad hocnetworks, which enforce cooperation among nodes todetermine the interference-minimized high throughputroutes. Improvement in throughput of up to 60% has beenobserved compared to the classic minimum energy routingapproach. Also, an interference-aware performance metricbased on the effective data rate is formulated and evalu-ated in [26]. In [7], a throughput optimal distributed coop-erative routing scheme is developed by constructing acontention graph based on virtual nodes and virtual links.An enhanced relay selection metric for cooperative

30 M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42

communication is formulated in [27] by exploiting channelstate information (CSI) and the residual energy of sensornodes. However, QoS provisioning issues in the delay orreliability domains have not been explored previously.

Refs. [9–12] presents several proposals for QoS-awarecooperative routing systems. In [9,10], cooperative routingalgorithms are developed to enhance the reliability in en-ergy-constrained networks. The cooperative relay selectionis based on the offered reliability of the candidate nodes.The performances have been analyzed to examine thetrade-off between the reliability and energy-efficiency.However, they have not consider the e2e delay minimiza-tion problem. In [11,12], the QoS-aware cooperative rout-ing algorithms are developed in both the delay andreliability domains, wherein it is assumed that the Q-valueof a node represents the quality of its packet forwarding interms of delay and packet loss ratio. Each candidate coop-erative node maintains its Q-values and those of its coop-erative partners, and when a packet is received by agroup of cooperative nodes, each node compares its ownQ-value with those of other nodes. The node that deter-mines it has the highest Q-value is elected as the best relayand forward the packet to the adjacent node toward thesink. Thus, the reactive relay selection mechanism of[11,12] not only increases the overhead but also fails to uti-lize the appropriate network channel conditions, since theQ-value does not reflect the exact qualities (e.g., delay,packet delivery ratio, energy-level) of the available relays.

Out of the existing systems, MRL-CC [11] is the mostsimilar to our approach. However, there are a few funda-mental differences between the two. First, MRL-CC oper-ates on a multihop mesh cooperative structure for datadissemination in WSNs, whereas in DACR, an e2e delay-and energy-aware path between the source node and thedestination sink is created first on which the cooperativerouting works. Second, MRL-CC uses the node withthe highest Q-value as the relay node, but in our DACR,the problem of the best relay selection is converted intoa Lexicographic Optimization (LO) problem that selectsthe node offering higher link reliability and lower delayunder the minimum residual node-energy constraint.Third, MRL-CC’s relay selection is reactive, i.e., after thesource’s transmission of a data packet, the node havingthe highest Q-value is elected as the best relay node;whereas DACR employs a proactive approach, the sourcenode finds the best relay by executing LO functions beforetransmitting the data packet, reducing the overhead.Finally, at each hop of a path, MRL-CC routers exploit coop-erative transmissions, whereas, DACR routers decide adap-tively whether to use direct or cooperative transmissionsbased on the link conditions.

1 This is a reasonable assumption for cooperative communications [8].

3. System model and assumptions

Consider a wireless sensor network consisting of a ran-domly distributed large set of sensor nodes where eachnode has a single omnidirectional antenna. The delay-and reliability-constrained sensing data packets flow frommany sensors to a destination sink node (placed anywherein the network) in multihop fashion. For each data packet,

the application layer restricts a predefined e2e reliabilityrequirement (Ke2e

rel ) and delay-deadline (Ke2edel ), where, Ke2e

rel

is the ratio of the total number of received packets by thesink to the number of sent packets by the sensor nodes,i.e., it defines the required minimum percentage of packetsthat should be delivered to the sink for reliable event per-ception; and, Ke2e

del is the time limit within which the packetmust be reached at the destination sink, otherwise the con-tent information of the packet will not be useful for thecontrol center.

Furthermore, the wireless channel of the network ismodeled as follows. The links between any two sensornodes in the network is subject to narrowband Rayleighfading, propagation path-loss, and additive white Gaussiannoise (AWGN). The channel fading for different links is as-sumed to be statistically mutually independent; thisassumption is reasonable since the nodes are usually spa-tially separated. In such an environment, the system re-quires an adaptive communication protocol that canmeet up the above QoS requirements through cooperativespatial diversity.

We assume that, at each routing node of a multihoppath from the source to the destination, neighbor nodescooperating in sending the information to its next-hopnode can precisely time their transmitted signal to achieveperfect phase synchronization at the receiver.1 Ratherselecting multiple relays at each routing hop, we use sin-gle-relay cooperative transmission, which has been evalu-ated as the most effective strategy considering theachievable network performance and the correspondingprotocol operation overhead [8,11,12]. Under this setting,the information is routed to the next-hop node in a sequenceof transmission slots, where each transmission slot corre-sponds to one use of the wireless channel. In each transmis-sion slot, either a node is selected to broadcast theinformation to a group of nodes, or a selected relay node thathave already received the information cooperate to transmitthat information to the destination node, which is known ascooperative or relayed-transmission (RTx) [28]. Once a node isselected as relay, it gives higher priority to relay packetsfrom other nodes than transmitting its own packet. Cooper-ation results in additional spatial diversity by introducingthis artificial multipath through the relay link, which can,in turn, enhance the transmission reliability against wirelesschannel impairments such as deep-fading. At any hop of thepath, a router may also opt for direct transmission (DTx) to itsnext-hop node if cooperation is not necessary.

The nodes are assumed to transmit over a single com-munication channel using any CSMA/CA-based mediumaccess control protocol. The receiver uses Maximal RatioCombining (MRC) [3] technique to decode the signalsreceived from the source and the relay nodes. We alsoassume that the nodes use fixed transmission power andthey work in half-duplex mode, i.e., any node cannot trans-mit and receive simultaneously.

We also emphasize that since the sensor nodes havelimited computation power and are battery-operated, thesolution methodologies should be light-weighted,

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 31

requiring less complex computations, exchanging mini-mum number of messages and thus consuming lessenergy.

Fig. 2. Multi-hop cooperative routing path in WSNs.

4. The DACR architecture

The QoS-aware cooperative routing problem can beviewed as a multistage decision making problem, wherethe decision at each stage is to pick the transmission modem 2 fDTx;RTxg as well as the best relay node, if necessary.The objective is to send the information to the destinationsatisfying the QoS requirements. In this paper, we convertQoS-aware routing decision into an optimization problemunder our model and solve it by linear optimization.Fig. 1 shows the architectural components of our distrib-uted adaptive cooperative routing protocol (DACR) and inthe following subsections, we describe their design issuesin detail. We first present an AODV [13]-based ad hoc rout-ing algorithm DEAR that creates an e2e path from source todestination avoiding energy-critical nodes and minimizingthe e2e delay. The DEAR also helps to identify sets of relaynodes Rs for the source and all intermediate nodes Is onthe path, as shown in Fig. 2. We then present algorithmsfor optimal relay selection and adaptive transmissionmode selection.

4.1. Delay- and energy-aware routing

Since energy is the most important constraint in a wire-less sensor network, each sensor node has to manage itsenergy intelligently in addition to follow the rules regard-ing QoS provisioning of the underlying applications. Arouting layer protocol along with cooperative communica-tion option might be much effective to enhance this energymanagement. In our protocol, whether a node would workas a router of a multihop route from source to destinationor as a relay for any source will be decided by it dynami-cally based on its current residual energy level. If the nodehas very low energy (i.e., less than certain threshold), itwill defer any responsibility (either as a router or as arelay) even the performance that it can provide is betterthan other neighbor nodes.

In this section, we develop a delay- and energy-awarerouting protocol (DEAR) that creates an e2e path fromsource to destination sink. The design principle of DEAR

Fig. 1. The components of our DACR protocol.

is based on AODV with some modifications. Like in AODV,when a route is required, a DEAR source node broadcasts aroute request (RREQ) packet. However, unlike in AODV, aDEAR source node tags a RREQ packet with a timestampvalue T and a minimum node energy threshold value Eth.We replace the AODV RREQ fields ‘hopcount’ and ‘reserved’with T and Eth, respectively; and thus incur no extra over-head or modifications to the basic RREQ packet structure. Aneighbor node rebroadcasts this RREQ packet upon recep-tion, only if the residual energy of the node is higher thanthe specified threshold value Eth. Also, the neighbor nodededucts the sojourn period of the RREQ packet from thetimestamp T before rebroadcasting the packet. Thus, likein AODV, multiple copies of the RREQ packet eventuallyreach the destination sink. The sink then returns a route re-ply (RREP) packet toward the source node for the RREQpacket with highest T value. Thus, a delay-minimized e2epath P is established consisting of non energy-criticalnodes.

Note that T has a large initial value inserted by sourcenodes in the RREQ packets and is predefined by the sinknode. All source nodes use the same T value when broad-casting RREQ packets. The update policy of T value at theintermediate nodes does not require any clock synchroni-zation and works as follows. As a RREQ packet travels to-ward the sink, intermediate nodes update T ¼ T � Tsojourn,where Tsojourn is the sojourn period of the RREQ packet atan intermediate node. We measure the sojourn period ateach node I and tag the packet with the updated T valueand rebroadcast it. For this, when a node I receives the lastbit of a RREQ packet, its MAC layer keeps record Tarrival forthe packet. The node I might need some time to process thepacket and capture the channel before rebroadcasting it.The MAC layer calculates Tsojourn just before it actuallytransmits the packet to the physical link as follows,Tsojourn ¼ Tdeparture þ TtransDelay � Tarrival, where Tdeparture is thetime at which node I transmits the first bit of packet tophysical link and TtransDelay is the transmission delay of thepacket which can be computed using the transmission rateand packet length. Thus, any node I þ 1 receiving this RREQgets the correct measurement of the updated T value with-out requiring any clock synchronization among the nodes.Finally, the sink node can easily determine the minimumdelay path P by observing T values of the received RREQpackets.

Furthermore, unlike in AODV, DEAR route error (RERR)packets are generated by any node I on the path whenany one of the following two conditions holds true: (a)the forwarding link from I to I þ 1 fails or (b) the residual

32 M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42

energy of I falls below the threshold value. The second con-dition guarantees that an intermediate node will not dieout of energy for carrying others data traffic. Note here thatthe amount of energy consumption by the intermediatenode I for transmitting one or two RERR messages is trivialand thus the mechanism does not cause battery of thenode die fast. In what follows, we describe how the DEARrouting packets are exploited by nodes on the routing pathfor developing a set of candidate relay nodes for each ofthem.

4.2. Cooperative relay-selection

For each hop on the cooperative routing path, determin-ing the set of candidate relay-nodes (R) between two adja-cent routers and selecting the optimal one from amongthem is challenging. We address the first part of the prob-lem by exploiting the routing packets of our DEAR protocolas follows. Once the route from the source to the destina-tion is discovered, for two adjacent routers along the route,e.g., I and I þ 1, as shown in Fig. 2, any node can determinethat it is a relaying candidate for them if it has received theRREQ packet broadcasted by I, and the RREP packet repliedby I þ 1, and has not been selected by I as the next hop rou-ter in the route discovery procedure. Then, each R 2 R

relays the RREP packet (received from I þ 1) to node Iand thus I comes to learn the set of candidate relay-nodesR. In what follows, we present solution to the second partof the problem, i.e., selecting the most optimal relay fromR.

4.2.1. Optimal relay-selection criterionThe variation in channel quality affects the reliability

and delay performances associated with the direct andcooperative transmission options. Also, the advantages ofrelay cooperation often depend on sufficiently reliableinteruser channels. For example, in the decode-and-for-ward (DF) scheme, a node relays the message from thesource only if it decodes the message reliably. Similarly,in the amplify-and-forward (AF) scheme, the quality ofthe relayed signal is limited by the quality of the source-relay link since both the signal and noise are amplified atrelays. Therefore, relays should be adopted only if thesource-relay channel is sufficiently reliable. Our secondconcern is the minimization of delay incurred by coopera-tive transmission since the packets transmitted via relaynodes might experience additional delays. This observationleads to the selective relaying (SR) cooperation schemewhere relays are selected adaptively to retransmit thesource message only if the quality of the transmission overthe interuser channel meets a certain criterion. In thiswork, we define the reliability rI;Iþ1 of a link between twonodes I and I þ 1 as the ratio of the number of packets re-ceived by I þ 1 to the number of packets sent by I over aperiod of time t. Since reliability is a multiplicative metric,the link reliability can be expressed as follows

rI;Iþ1 ¼rDTx

I;Iþ1 Direct Tx:

rRTxI;R;Iþ1 ¼ rI;R � rR;Iþ1 Relayed Tx:

8><>:

ð1Þ

Note here that transmission on the link ðI;RÞ has noacknowledgment and thus it would be expensive for I tolearn the link reliability rI;R. When the next hop nodeI þ 1 receives the retransmitted signal from the relay nodeR, it tries to extract the packet information from receivedsignals (from I and R) using Maximal Ratio Combining(MRC) method [3]. If node I þ 1 can correctly extract theinformation, it sends back an acknowledgment to I. Thus,the sender node I learns that the data packet has been cor-rectly received by its next hop destination I þ 1 and cancalculate the link reliability for relayed transmission,rRTx

I;Iþ1, based on the number of acknowledgment packets re-ceived and the number of data packets sent by it duringany measurement interval t. For example, assume thatthe sender node I sends 100 packets, out of which 80 pack-ets are correctly decoded by the relay node R and only 60packets are successfully extracted by the next hop nodeI þ 1. Now, if the sender node I receives 60 acknowledg-ments from I þ 1 during a measurement interval t, thenthe link reliability measured by I will be rRTx

I;Iþ1 ¼60=100 ¼ 0:6. Therefore, Eq. (1) can be rewritten as follows

rI;Iþ1 ¼rDTx

I;Iþ1 Direct Tx:

rRTxI;R;Iþ1 ¼ rRTx

I;Iþ1 Relayed Tx:

8><>:

ð2Þ

Our second criterion aims to select a relay node thatprovides with faster communication between its sourceand destination, i.e., the problem of finding the relay hav-ing lower relay-over delay. The relay-over delay tI;R;Iþ1 fora packet is measured by the time it requires to successfullytransfer a packet from I to I þ 1 through R. This delay quan-tifies the quality of the link ðR; I þ 1Þ as well as the conges-tion level of the selected relay R. Let tTX and tACK , during anupdate interval t, are the time points at which the first datapacket has been transmitted and the acknowledgment forthe last data packet has been received, respectively, thenthe average relay-over delay per packet can be calculatedas

tmI;Iþ1 ¼

tACK � tTX

Nsuc; m 2 fDTx;RTxg; ð3Þ

where Nsuc is the number of packets successfully transmit-ted. Separate measurements are carried out for direct- andrelayed-transmissions, m 2 fDTx;RTxg.

We exploit reinforcement learning (RL) method [14],which provides a framework in which an agent can learncontrol policies in dynamic environment based on experi-ences and rewards, as shown in Fig. 3, to enable a sendernode to acquire knowledge on the above parameters. Here,the wireless channel characteristics and data traffic flowsare corresponding to the dynamic environment and sensornodes are the agents of the RL system. Our RL system is amodel-free learning technique, which can address para-metric optimization to maximize the long-term award[15].

We define the set of states S, the set of actions A and theset of rewards Rwd of our RL system as follows:

(a) State : S ¼ fsg; s 2 fI; R; I þ 1g,

Fig. 3. The reinforcement learning system.

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 33

(b) Action : A ¼ fag; a 2 faDTx; aRTx;noneg, where, theexecution of aDTx or aRTx means that either direct-trans-mission or relayed-transmission, respectively, isselected by the agent. The selection policy will bedetailed in Algorithm 1 in Section 4.3. The third optionnone will be chosen by the agent only when no node inthe neighborhood environment is able to meet up thedelay and/or reliability requirement of a given packet,as described at the end of Section 4.2.2.(c) Reward Function: The reward is obtained when theagent executes an action. For each pair of consecutiverouters, once a relay node or direct transmission isselected for delivering data packets (i.e., the actionis executed), the performance of the transmissions ismeasured (during t periods of time) in terms of reliabil-ity and delay according to Eqs. (2) and (3), respectively,and the system gives reward to the sender of the com-munication as follows

Rwd ¼Km

relðtÞ ¼ ð1� aÞKmrelðt � 1Þ þ arm

I;Iþ1ðtÞ

KmdelðtÞ ¼ ð1� aÞKm

delðt � 1Þ þ atmI;Iþ1ðtÞ;

8<:

ð4Þ

where, KmrelðtÞ and Km

delðtÞ are the updated knowledge on thereliability and delay, respectively, for direct- and relayed-transmission modes, m 2 fDTx;RTxg. Rewards are givencorresponding to the actions taken, i.e., if relayed transmis-sion is chosen, the knowledge is updated for the selectedrelay node R, denoted as KRTx

rel ðt;RÞ and KRTxdel ðt;RÞ. Note here

that the exponential weighted moving average (EWMA)formula, with weight factor a, has been used to updatethe knowledge, in which the weighting for older data val-ues decreases exponentially, giving much more impor-tance to recent observations. The rationale of usingEWMA in updating knowledge of a node regarding perfor-mance of its transmission environment is that expectedbehavior of the wireless links heavily depends on the his-torical behaviors and that behavior is well captured bythe EWMA.

Furthermore, note that the above RL system provideswith the most updated knowledge on expected perfor-mances of neighborhood nodes without using any compli-cated prediction techniques, or explicitly frequentupdating and maintaining of precise network state infor-mation. In what follows, we discuss on the method ofselecting the most optimal relay based on the knowledgeacquired from the RL system.

4.2.2. Optimal relay-selection algorithmOnce the sender node learns the knowledge of its trans-

mission environment, i.e., the performances of the relayedtransmissions for different candidate relay nodes R 2 R, theproblem of determining the optimal relay node boils downto formulating an equation that optimizes the trade-off be-tween the reliability and delay. More elaborately, our goalis to select a relay node that provides with comparativelyhigher reliability and reduced delay toward the destina-tion. Note that while the second criterion looks for alightly-loaded relay, the first one attempts to choose a re-lay that can ensure a high quality linkage between thesource and the destination. A weighted linear combinationof the objective parameters may suffice to solve the prob-lem [2]. However, the key problem of this technique is toset the optimal weight values associated with the parame-ters and wrong values might lead to produce a combinedmetric that fails to guarantee the QoS requirements [16].

Our proposed relay-selection algorithm uses a multiob-jective Lexicographic Optimization (LO) approach [17] tomanage this trade-off. In LO, the objective functions are ar-ranged according to their absolute importance and themost important objective function is minimized first sub-ject to the original constraints. If this problem has a uniquesolution, it will solve the whole multiobjective optimiza-tion problem. Otherwise, the second most importantobjective function is maximized (or minimized). Now, inaddition to the original constraints, a new constraint isadded. This new constraint is imparted to guarantee thatthe most important objective function preserves its opti-mal value. If this problem has a unique solution, it solvesthe original problem; otherwise, the process goes on asabove.

In solution to our problem, there are only two objectivefunctions of which the first one, the maximization of thereliability, is the most important. Let the objectivefunctions be arranged according to the lexicographic order,with the most important function being, f1ðRÞ ¼KRTx

rel ðt;RÞ; 8R 2 R and the least important one beingf2ðRÞ ¼ 1

KRTxdel ðt;RÞ

; 8R 2 R. Thus, we write the lexicographic

problem for any node I 2 P on the path P as follows:

lex maximize f1ðRÞ; f 2ðRÞ ð5Þsubject to R 2 R: ð6Þ

The above LO problem can be divided into two separateproblems with different constraint sets. The first problemis formulated as

maximize f1ðRÞ ð7Þsubject to R 2 R ð8ÞKRTx

rel ðt;RÞP Kminrel ; 8R 2 R ð9Þ

ERres P Eth; 8R 2 R; ð10Þ

and its solution R�1 and f �1 ¼ ðR�1Þ is obtained; here, ER

res is theresidual energy level of node R and Kmin

rel is the minimumthreshold link-reliability level that must be provided bythe selected relay node. The measurement for Kmin

rel hasbeen carried out in Appendix section. The constraint inEq. (10) restricts the system to select a relay node having

34 M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42

residual energy below the threshold level even if it satisfiesthe other criteria well. This restriction allows relay nodesnot to exhaust their minimum energy levels for carryingothers data traffic; and thus, it avoids creating routingholes, ensures better sensing coverage and balanced en-ergy consumption among the relay nodes at each hop,which in turn helps to improve the network lifetime.

Our second problem can be formulated as follows

maximize f2ðRÞ ð11Þsubject to R 2 R ð12Þ

KRTxrel ðt;RÞP Kmin

rel ; 8R 2 R ð13ÞER

res P Eth; 8R 2 R ð14ÞKRTx

del ðt;RÞ 6 Kmaxdel ; 8R 2 R ð15Þ

f1ðRÞ ¼ f �1 ; ð16Þ

and the solution of this problem is R�2 and f �2 ¼ ðR�2Þ; here,

Kmaxdel is the maximum allowable link-delay if packets are

retransmitted through R and its value is analyzed inAppendix section. If R�2 does not produce a unique solution,we break the tie by selecting the one that provides withhigher reliability. With the proposed scheme, actually thelink with the best quality in terms of reliability and delay,is used. Such a strategy makes our system robust to linkdynamics.

In solution to the above optimization problems (7) and(11), in worst case, there may arise a situation that neithera routing node can find a single relay in its vicinity, whichcan meet up the minimum threshold link-reliability levelKmin

rel and/or the maximum threshold link-delay level Kmaxdel

for a certain packet nor the direct link could meet up therequirement. In such cases, we intentionally drop the pack-et assuming that its on-time delivery probability to thesink gets highly reduced due to poor neighborhood envi-ronments. A similar packet dropping policy was alsoadopted in MMSPEED [1] and DARA [2]; it has the follow-ing advantages: (a) it decreases the network traffic loadthereby allowing other viable packets to reach the destina-tion in time and (b) it decreases the energy wastage forcarrying useless packets.

Note that the nice property of LO is its simplicity. Theadded computational expense of solving multiple optimi-zation problems at each instant is not significant since sim-ple linear equations are solved out. Therefore, such alightweight but effective method is suitable for resource-constrained wireless sensor networks. Even though LOhas the drawback of neglecting less important objectivefunctions when the most important one produces the un-ique solution, in our case, such situations should occur lessfrequently due to the high density of the network nodes.Moreover, LO exploits only local information to make re-lay-selection decisions. The absence of a global informa-tion exchange scheme reduces the networks setup andupdating costs and alleviates the possibility of incorrectinformation at the nodes as changes in system topologyoccur.

4.3. Adaptive transmission mode-selection algorithm

Our decision on whether to use direct or cooperativetransmission from any source or intermediate node to itsnext-hop destination is adaptive to the conditions of thelinks I-to-I þ 1; I-to-R and R-to-I þ 1. The adaptation ismade on the basis that there is no need for any relay to for-ward the source’s information if the direct link betweenthe source and the destination is of high quality. A sourceor an intermediate node makes its decision for direct andrelay-based transmission mode m 2 fDTx;RTxg whichyields the higher reliability.

Algorithm 1. Transmission mode selection algorithm forany node on a routing path

1: Initialization: Initialize Kmrel which is the knowledge

about the reliability2: repeat3: if KDTx

rel > KRTxrel then

4: if RANDOMðÞ 6 � then5: The node irrationally chooses relay-

based transmission (i.e., m ¼ RTx)6: else7: The node rationally chooses direct

transmission (i.e., m ¼ DTx)8: end if9: else10: if RANDOMðÞ > � then11: The node irrationally chooses direct

transmission (i.e., m ¼ DTx)12: else13: The node rationally chooses relay-based

transmission (i.e., m ¼ RTx)14: end if15: end if16.17: During the measurement interval t, the node

computes instantaneous reliability rdir and rrel usingEq. (2)

18: The node updates knowledge Kmrel using

reinforcement learning method as of Section 4.2.119: until the path is broken or the connection is

terminated

In the transmission mode selection algorithm (as shownin Algorithm 1), each node on the routing path maintainsknowledge about the reliability, given decision on modem made before. The knowledge Km

rel is defined as the expo-nentially weighted moving average of the perceived reli-ability of a sensor node, which is used by the node tomake the current decision. Once the current decision onmode selection is made, the resulting reliability is usedto update the knowledge Km

rel, as discussed in Section4.2.1. To avoid local optimal decisions (e.g., due to the lackof complete network information), the routing node maymake an irrational decision with very small probabilitybounded by �, known as �-greedy method [29], to explore

Table 1Simulation setting.

Basic specificationNetwork area size 2000 m � 2000 mDeployment type RandomNetwork architecture Homogeneous and flatNumber of nodes 2000Sink location (1000,1000)Initial node energy 10 JBuffer size 50Radio range 100 mSensing radius 52 mLink layer trans. rate 512 KbpsTransmit power 7:214e�3 WRcv. signal threshold 3:65209e�10 WLink failure rate, f 0.3MAC IEEE 802.11 DCFSimulation time 200 s

Sensed traffic specificationApplication type Event-drivenSources in one event �15 NodesPacket size 64 ByteTraffic type CBR (3 pps)

DACR specification

Ke2erel

0.8

Ke2edel

500 ms

Measurement interval, t 2 sEWMA weight factor, a .2Eth 5 J (initial value)� 0.1

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 35

reliability resulting from alternative decisions. InAlgorithm 1, the RANDOMðÞ function generates a fractionin between 0 and 1, i.e., 0 < RANDOMðÞ < 1. Therefore, withthe probability 1� �, the relay candidate which is expectedto be able to make the most contributions will be selectedas the optimal relay; and with the probability of �, directtransmission will be executed.

4.4. Discussion

Thanks to the hop-by-hop reliability- and delay-drivendata delivery options with adaptive cooperative routing,we can claim that once a packet reaches its destination,it is likely that the packet meets its e2e deadline. However,not all packets are guaranteed to reach their destinationsince we are compromising the e2e reachability (i.e., prob-ability of reaching) by intentionally dropping packets toguarantee the on-time packet delivery. Similarly, to assurea certain level of reachability, we compensate for choosinga transmission that could provide with minimum delay;rather, we maximize the link reliability at each hop so thatthe total reachability is increased.

Setting an appropriate value of Eth for nodes in the net-work is an important performance tuning parameter, espe-cially for achieving uniform energy-distribution. Accordingto the LO functions, the higher the value is, the smaller isthe set of viable candidate relay nodes at each hop, whichin turn reduces the probability of choosing relay node withthe best performance. Conversely, if the value of Eth is verylow, the energy dissipation rates of nodes might varygreatly and shorten the network lifetime. In order totrade-off the above facts, we allow the sink to control Eth

value of sensor nodes as follows, it starts with the 80% ofthe initial energy level and steps down further by 20% aftera certain period of interval and so on.

The nice performances of our DACR protocol do notcome out of cost. The operation overhead of DACR includesthe RREQ and RREP messages exchanged during the e2eroute discovery and computation for acquiring knowledgeon expected delay and reliability performances of theneighbor nodes. However, since our reinforcement learn-ing-based knowledge update method does not requireany additional messages to exchange, its overhead doesnot scale much. As the dominant source of energy con-sumption is the sensor radio module, our DACR incurs lessoverhead than that of MRL-CC, where a large number ofmessages need to be exchanged for constructing meshcooperative structure.

The main limitation of this paper is related to a lack ofsufficient understanding about the dynamics of severalestimation tuning parameters. For example, the measure-ment interval t, EWMA weight factor a and the value of �were determined through numerous simulation experi-ments. If we could build an analytical model for them,we would be able to dynamically select the optimal valuesto adapt to different situations. We left it for future work.

5. Performance evaluation

We have evaluated the performance of DACR using sim-ulation experiments conducted on NS-2 [18], which

supports the simulation of multihop wireless networkscomplete with physical, data link, and MAC layer models.We have compared the results with those of the minimumpower cooperative routing – MPCR [6], REER [9], andMRL-CC [11] systems. The setting of the simulationenvironment parameters is listed in Table 1. Consideringthe applicability of DACR in practical mission-criticalapplications such as radioactive radiation monitoring, bat-tlefield surveillance, forest monitoring, etc., we have takena large network area of 2000� 2000 sqm, where 2000homogeneous sensor nodes are deployed randomly, pro-ducing node density q ¼ 0:0005. Such a deployment ispractical in the sense that it ensures the sensing coveragewith sufficient redundancy. More specifically, it gives thenumber of source nodes ¼ q� PI � R2

s � 15 [30] withinan event radius Rs ¼ 52 m. Furthermore, the link layer datatransmission rate, transmission range, sensing radius, buf-fer size and initial node energy that we have considered inour simulation are quite reasonable for available sensormotes [31].

We have used event-driven data collection approachsince it is more suitable for mission-critical applications.Two separate bursts of data traffic from four randomlychosen events, listed in Table 2, are considered in the per-formance studies. In each burst, the event durations are setto 30 s (e.g., 10 � 40 s) with triggering intervals of 10 s.Thus the events of a burst are time-overlapped, i.e., thenetwork has to carry data traffics generated from 2 or 3events simultaneously. This traffic pattern helps us tostudy the performance behavior of the protocols in dynam-ically varying traffic load environment. Furthermore, foreach data point in the graphs, we take the average ofresults from 20 simulation runs, executed with different

Table 2Events and bursts description.

Event 1 Event 2 Event 3 Event 4

Burst 1 10—40 20—50 30—60 40—70Burst 2 100—130 110—140 120—150 130—160

36 M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42

random seeds. Thus, the variations in the obtained resultsmainly occur due to the randomness of the event locationsand network topology.

5.1. Performance metrics

We define the following metrics for performancecomparisons:

� Average end-to-end delay – of a single packet is mea-sured as the time difference between when the packetis received at the sink and its generation at the sourcenode. Delays experienced by individual data packetsare averaged over the total number of packets receivedby the sink. Lower value is correspond to the betterperformance.� On-time packet delivery ratio (i.e., reliability) – is the ratio

of the total number of packets received by the sinkwithin the delay-deadline to the total number of pack-ets generated by all the source nodes in the network.Higher value is correspond to better transmissionefficiency.� Energy expenditure per successful packet delivery – is

measured as the ratio of total amount of energy dissi-pated by all source and forwarder nodes of the networkfor transmission, reception and overhearing to the totalnumber of packets received by the sink. Therefore, if thesame amount of total energy is dissipated by any twoprotocols then the one that has higher reliability per-forms better.� Protocol operation overhead – is expressed as the num-

ber of control messages per successful data delivery,i.e., it can be measured as the ratio of the number ofcontrol messages transmitted to the number of datapackets delivered to the sink before the expiratio ofthe network lifetime.� Integrated performance – We plot graphs for integrated

performances of the protocols in terms of reliability,delay and energy using the following relationship: Inte-grated performance = Reliability/(Energy ⁄ Delay).Higher value is correspond to the better performance.� Network lifetime – Since a node participates in transmis-

sions from other nodes in addition to its own transmis-sions in a cooperative scenario, its lifetime depends onthe energy consumed in both cases. Hence, we definethe lifetime of a node in terms of the energy consump-tion per unit time. Thus, the lifetime of a node I with ini-tial energy E0 can be expressed as LifeTimeðIÞ ¼ E0

eI,

where, eI denotes the energy consumption per unit timefor node I. We assume that all nodes have the same ini-tial battery life. We can, therefore, define the networklifetime as the minimum lifetime of all nodes as fol-lows: LifeTimeðnetworkÞ ¼minI2NLifeTimeðIÞ, where, Nstands for all nodes in the network.

5.2. Results

5.2.1. Impacts of the link failure rateIn this section, we discuss the impacts of different chan-

nel conditions on the performances of the protocols. In thispaper, we express the channel condition in terms of thelink failure rate f and we vary the value of f from 0.05 to0.5 by the step size of 0.05 keeping all other parametersin Tables 1 and 2 fixed.

The simulation results, as shown in Fig. 4, indicate thatthe performances of the protocols decrease as the link fail-ure rate f increases in terms of all the performance metrics,as expected theoretically. However, the rate of perfor-mance degradation varies greatly between protocols. Forexample, in Fig. 4(a), even at lower values of f ð< :20Þ theaverage e2e packet delays of our DACR and MRL-CC proto-cols are similar; the gap between them increases sharply asf increases; and, we observe an improvement in the perfor-mance of DACR over MRL-CC as high as 37.6%. Our in-depth look into the simulation trace file reveals that whenf increases, the quality of network environment measure-ment through node’s Q-value in MRL-CC is insufficient toantagonize the high link failure rates. In DACR, relay-overdelays of packets are observed during t period of timeand knowledge on expected delay of a packet for differentpotential relays (as well as direct transmission) are up-dated through RL method exploiting EWMA formula.Therefore, more accurate measurement on network envi-ronment is captured by our DACR, thus helping to selectthe optimal mode of transmission and relay node (if re-quired) and contributing to achieve faster e2e packet deliv-ery. Moreover, the data delivery over the delay- andenergy-aware routing (DEAR) path in DACR gives it addedadvantage in reducing delay compared to multihop meshcooperative structure in MRL-CC. Also, the average e2epacket delays experienced by REER and MPCR protocolsare incomparable to our DACR since they optimize mainlyreliability and power, respectively.

Comparing graphs of Fig. 4(a) and (b) we can state thatthe cooperative routing systems are more robust to han-dling link failures in terms of reliability performanceopposing to delay performance. More than 80% packetsreach at the destination in time for all the protocols whenf 6 0:3, as shown in Fig. 4(b). The reliability performanceof REER protocol is similar to that of our DACR until freaches to 0.30. Beyond that, the performance of REER re-duces at higher rate and as much as only 10.4% degrada-tion compared to DACR is observed. This niceperformance of REER comes from the fact that it is basedon geographic routing (i.e., uses location informationthrough GPS) and it can handle the dead-end routingproblem efficiently. However, it experiences a significantamount of packet drops due to collisions since it doesnot consider the traffic load of neighborhood nodes whileselecting relays and next-hop nodes. Thus, its perfor-mance is affected, especially, at higher values of f. Onthe other hand, DACR’s adaptive decisions on transmissionmode and relay selections at each hop maximizes the per-hop reliability and makes it more robust to link failures,achieving more than 80% on-time packet delivery evenat 50% link failures.

Fig. 4. Performance comparisons for varying link failure rates with data traffic load = 3 pps. (a) Average end-to-end delay. (b) Reliability. (c) Energyexpenditure. (d) Integrated performance. (e) Protocol overhead. (f) Network lifetime.

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 37

The energy expenditure per successful packet deliveryin MPCR is the lowest among the studied protocols till freaches to 0.30, as shown in Fig. 4(c). This is due to MPCR’spolicy of minimum power routing and relay selection pro-cedures. However, it experiences a large number of packetdrops due to collisions and buffer overflow at intermediaryhops when f is larger, thus increasing the per packet energyexpenditure. Nevertheless, as expected theoretically, thesimulation trace file content reveals that our DACR proto-col can’t save total energy consumption of the networknodes compared to REER and MRL-CC. But, since itachieves higher delivery ratio, its per packet energy expen-diture does not increase much.

Fig. 4(d) shows the integrated performances of thestudied protocols in terms of delay, reliability and energyexpenditure. We see that the overall performance of ourDACR protocol is much higher than the others, as high as70.56% performance improvement over MRL-CC is ob-served. This is because DACR uses energy-aware route dis-covery and relay selection and optimizes the delay andreliability at each hop through adaptive decisions onselecting transmission mode and most optimal relay node.The results also prove that proactive approach of relayselection is more effective than reactive one, which is em-ployed by MRL-CC.

Every QoS provisioning scheme has to exchange addi-tional control packets (in addition to those for the basicrouting mechanism) in order to update the nodes withthe current neighborhood information necessary to pro-vide better QoS services, incurring extra overhead. Asshown in Fig. 4(e), the operation overhead of MPCR proto-col is the highest and does not increase much with f. This

result is caused by periodic HELLO packet broadcasting ofMPCR nodes and executing Bellman-Ford shortest pathalgorithm for finding the relay that could provide withminimum power e2e path. An interesting result is foundfor MRL-CC; its overhead increases exponentially whenthe value of f crosses 0.30 and we reveal the fact that thenumber of control messages in MRL-CC suddenly increasesto very high value for constructing the multihop meshcooperative structure. The e2e route discovery proceduresin REER and DACR have the similarity and both of themrely only on single hop neighborhood information for relayselection, thus showing almost same operation overhead.

Finally, network lifetime is an important performancemetric for resource-constrained wireless sensor networks.As shown in Fig. 4(f), the network lifetime of our DACR pro-tocol is much longer than the other protocols and startingfrom 12.5% to as much as 155.6% improvement over MRL-CC is observed for increasing values of f. The rationale be-hind achieving this result is that DACR takes the residualenergy levels of nodes into consideration while creatinge2e route and selecting the relays at each hop and thus dis-tributes the energy-load more uniformly over the nodes ofthe network. Whereas in MRL-CC, neither the mesh coop-erative structure formation nor the relay selection methodis energy-aware; rather, the mere Q-value based relayselection might put extra loads on a certain node that per-forms better, thereby causing earlier death of the node andreducing the network lifetime drastically.

5.2.2. Impacts of data traffic load in unreliable environmentIn this section, we evaluate the performances of the

studied protocols for various data packet generation rates

38 M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42

from sensors, when link failure is set to 0.30. Our DACR’spolicy of choosing a relay node that provides with lowestlink-delay from the set of nodes guaranteeing the requiredreliability level, implemented through LO functions,enables it to achieve minimum delay e2e communicationamong all the protocols, as shown in Fig. 5(a). We also ob-serve from Fig. 5(b) that, in meeting the reliability require-ment of 0.8, our DACR protocol can tolerate the data trafficload of up to 7 packets per second (pps) compared to 5 ppsfor REER, 4 pps for MRL-CC and 3 pps for MPCR. Again, thisis due to its judicious selection of transmission mode andrelay nodes optimizing the reliability and delay hop-by-hop basis. Comparing the graphs of Fig. 4(a) and (b) and5(a) and (b), we can also state that the cooperative routingsystems are more robust to dealing with worse networkconditions than with increasing traffic loads, i.e., perfor-mance degradation rate is much lower for the latter casethan the former one.

As shown in Fig. 5(c), the energy expenditure per packetin DACR increases at higher rate for increasing traffic loadscompared to that observed in Fig. 4(c). Since the number ofcollisions as well as the packet drops increase with highertraffic loads, the packet delivery ratio also decreases shar-ply and thus increasing the per packet energy expenditure.The integrated performance of our DACR protocol is stillmuch higher than the other protocols, as shown inFig. 5(d), which proves its superiority in handling variousnetwork conditions as well as data traffic loads.

The operation overhead of the protocols, except forMRL-CC, does not increase significantly with an increasein data traffic loads in the network, as shown in Fig. 5(e).

Fig. 5. Performance comparisons for varying data traffic loads with link failuexpenditure. (d) Integrated performance. (e) Protocol overhead. (f) Network life

Because the frequency of route creation (and thereby thenumber of control messages) is greatly increased whennetwork link conditions decline; it is not dependent onthe network traffic loads. However, the rate of fall of thenetwork lifetime with the increasing traffic loads is veryhigh, as shown in Fig. 5(f), compared to that in Fig. 4(f).This is because the transmission and reception of datapackets consume a lot of energy of the nodes and its irre-spective of the employed cooperative routing system.Therefore, distributing the total traffic load over the net-work nodes in such a way that their energy dissipation ratedoes not vary much from one to another is the most effec-tive way to increase the network lifetime. Since our DACRprotocol employs this policy through LO functions by reg-ulating the threshold energy level ðEthÞ dynamically, its en-ergy-distribution is capable of maintaining a high level ofuniformity among the nodes and thereby prolonging thenetwork lifetime compared to other protocols.

5.3. Impacts of �

We have also carried out performance evaluations ofour proposed DACR algorithm for various values of �, asmall probability with which a routing node selects an irra-tional cooperative node (discussed in Section 4.3). We ob-serve in Fig. 6 that the introduction of � in our algorithmgives slightly better performance (in terms of on-timepacket delivery ratio, i.e., the reliability) both for varyinglink failure rates and data traffic loads. We also notice thata smaller value of � (e.g., � ¼ 0:1) is better than larger one(e.g., � ¼ 0:2).

re rate = 30%. (a) Average end-to-end delay. (b) Reliability. (c) Energytime.

Fig. 6. Impacts of selecting irrational cooperative routing node with a small probability �. (a) Impacts of link failure rates (traffic load = 3 pps). (b) Impacts ofdata traffic load (link failure rate = 30%).

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 39

5.4. Computation cost

Finally, we carry out computation energy cost perfor-mance comparisons in between our proposed DACR andthe AODV routing algorithms for varying link failure ratesand data traffic loads. Even though the computation costis very much lower compared to the communication costs,this study helps us to understand the protocol operationoverheads more in detail.

For computation energy cost measurement, we assumethat the average computation cost per instruction is 1 lJ[32]. The Y-axis in Fig. 7 shows the total amount of compu-tation energy consumed during the whole simulation per-iod. The Fig. 7(a) shows that the proposed DACR routingsystem is more robust to the increasing link failures interms of computation cost overhead compared to the clas-sic AODV system. On the other hand, our DACR systemconsumes much computation energy for relay selection

Fig. 7. Computation energy cost comparision for varying link failure rates and dImpacts of data traffic load (link failure rate = 30%).

procedure and the amount sharply increases with thehigher data traffic loads, as shown in Fig. 7(b). Our in-depthlook into the simulation trace files reveals that the AODVexperiences huge control packet processing due to in-creased route reconstructions required as well as the num-ber of retransmissions is also increased at higher linkfailure rates. However, the cooperative relay nodes inDACR help to reduce the number of route reconstructionsand packet retransmissions, saving a significant amountof computation energy. On the contrary, the computationcost for cooperative relay selections through solving LOequations in our proposed DACR system is exponentiallyincreased at higher traffic loads. Finally, in measurementscale, the computation cost is negligible compared to thedata transmission and reception costs and thus the overallenergy cost performance of our proposed DACR system ismuch better compared to the state-of-the-art protocols,as depicted in the graphs of Figs. 4 and 5.

ata traffic loads. (a) Impacts of link failure rates (traffic load = 3 pps). (b)

40 M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42

5.5. Discussion

The simulation results show that our DACR can effi-ciently cater for the application requirements with variousnetwork conditions, i.e., different combinations of trafficloads and link failure rates. As a result, DACR can signifi-cantly improve the effective capacity of a sensor networkin terms of traffic flows meeting the reliability, delay andnetwork lifetime requirements with reduced overhead,under versatile network environments. The developedframework is expected to work with good performance ina large number of emerging WSN applications, particularlyin which both delay- and reliability-sensitive traffic flowsco-exist. For example, medical applications, forestmonitoring, radioactive monitoring and object trackingapplications, industrial process control and so on. However,it may not be suitable for applications that emphasize onlyon non-critical traffic flows, for instance, agriculturalapplications.

The rather pessimistic model of using the fixed value of� for probabilistically selecting an irrational cooperativerouting node, the measurement interval t and the EWMAweight factor a in our current work provides with the firststep research. Some dynamic selection techniques mightimprove the DACR performance. The analytical modelingto dynamically select the optimal values of the parametershas been left for future work.

Furthermore, the introduction of random explorationfactor � in solving the transmission mode selection prob-lem, in Algorithm 1, has added a random element in thefeasible region. Thus, it has become a simulation optimiza-tion problem [36], where the objective function cannot beevaluated exactly, which in turn makes it difficult to pre-dict the convergence speed of the algorithm. Adaptive ran-dom search methods for simulation optimization problem,that can maintain balance among exploration, exploitationand estimation, are more effective [36,37]. However, thedevelopment of an adaptive and almost surely convergentsearch method for our problem leads to a new research andwe keep it as a future work.

Note that, in wireless sensor networks, the nodes arecommonly set to operate at low-duty cycle for conservingenergy and thereby increasing the network lifetime, whichis typically implemented by employing duty cycle MACprotocols. In recent years, a number of cooperative MACprotocols have been developed for duty cycle enabledwireless sensor networks [33–35]. Our proposed distrib-uted adaptive cooperative routing (DACR) system nodesmay also employ duty cycling and thus could possessrobustness against the underlying sensor network patternthrough using a suitable MAC protocol. The study on theperformance improvement due to joint effort of our DACRand a suitable duty cycled cooperative MAC protocol inWSNs has been left for future works.

6. Conclusions

This paper proposes a QoS-aware distributed adaptivecooperative routing system (DACR) to achieve bothreliability and delay guaranteed data delivery in wireless

sensor networks while distributing the energy consump-tion load among many nodes and thereby increasing thenetwork lifetime. The DACR’s adaptive decision on trans-mission mode selection and judicious choice of relay nodesbased on their offered link-reliability and link-delay valuesat intermediate hops allow it to efficiently handle variousnetwork environments. The performances of our DACRhave been evaluated through extensive simulations for awide range of network link failure rates and data trafficloads; the results show that it improves the networkperformance significantly compared to state-of-the-artprotocols.

A comparative study on the performances of coopera-tive and multipath routing systems in guaranteeing QoSservices will be an important advance in the research,which we leave as a future work.

Acknowledgements

We would like to pay our highest level of gratitude andsincere thanks to the anonymous reviewers for their expertopinions and useful feedbacks, which helped a lot to enrichthe quality of the paper. This research was supported byNext-Generation Information Computing DevelopmentProgram through the National Research Foundation of Kor-ea (NRF) funded by the Ministry of Science, ICT & FuturePlanning (2010-0020728). Dr. C.S. Hong is the correspond-ing auhor.

Appendix A. Threshold values

A.1. Minimum link-reliability threshold Kminrel

In a multihop wireless sensor network, the knowledgeof per hop required minimum reliability Kmin

rel is a functionof the e2e required reliability Ke2e

rel , which is defined by theapplication, and the number of hops h between any sourceor intermediate node and the destination sink. Since thereliability is a multiplicative metric, the value of requiredper hop reliability Kmin

rel increases with the number of hopsh in order to meet the application defined reliability at thesink Ke2e

rel . Therefore, the value of Kminrel can be found from the

following equation

Ke2erel ¼ ðK

minrel Þ

h

Kminrel ¼ ðK

e2erel Þ

1=h ðA:1Þ

Consider QoS reliability requirement of 95%, if reliabil-ity of all outgoing links (using direct or cooperative trans-mission) is below 95% at an intermediate node, there is nofeasible solution to satisfy the requirement. Even a degra-dation of 5% on each link will cause a total decrease of27% on a path P with six hops. Also, as the number of hopson the path increases, the e2e reliability decreases. Usuallythe number of hops in large scale sensor networks is muchlarger than those in ad hoc networks. So, it imposes a se-vere problem on reliability. For the same P to achieve ane2e reliability of 90%, the geometric mean of reliability ofall six links on a six-link path P has to be 98%, which isvery restrictive in wireless communications. If the e2e

M.A. Razzaque et al. / Ad Hoc Networks 19 (2014) 28–42 41

reliability degrades so much that no route can meet theQoS requirement, cooperative routing seems to be an effec-tive way to enhance the e2e reliability. Note here that Ke2e

rel

is fixed for a particular packet; and thus, at each hop, wehave to choose a transmission (either direct or relayed)in such a way that the per hop reliability is maximized,ensuring high e2e reachability of the packet.

A.2. Maximum link-delay threshold Kmaxdel

The value of Kmaxdel is also determined by the e2e applica-

tion defined delay-deadline Ke2edel of a packet and the num-

ber of hops h between any node on the path and thedestination sink. Any node on the routing path can

calculate this value as follows, Kmaxdel ¼

Ke2edelh for any packet,

assuming that the sojourn period of the packet at the nextintermediate hops will also be bounded by Kmax

del . Using thesame procedure for updating lifetime of RREQ packets, dis-cussed in Section 4.1, the remaining lifetime of a data pack-et is updated by each node on the routing path, i.e., thesojourn period of the data packet ðTsojournÞ at the currentnode is subtracted from the lifetime value of the packet re-ceived from its previous hop, Ke2e

del ðcurrentÞ ¼ Ke2edel

ðpreviousÞ � Tsojourn.

References

[1] E. Felemban, C.G. Lee, E. Ekici, R. Boder, S. Vural, MMSPEED:multipath multi-speed protocol for QoS of reliability andtimeliness in wireless sensor networks, IEEE Trans. Mobile Comput.5 (6) (2006).

[2] M.A. Razzaque, M.M. Alam, M.M. Rashid, C.S. Hong, Multi-constrained QoS geographic routing for heterogeneous traffic insensor networks, IEICE Trans. Commun. E91-B (08) (2008) 2589–2601.

[3] A. Nosratinia, T.E. Hunter, A. Hedayat, Cooperative communication inwireless networks, IEEE Commun. Mag. 42 (10) (2004) 74–80.

[4] P. Herhold, E. Zimmermann, G. Fettweis, Cooperative multi-hoptransmission in wireless networks, Comput. Networks 49 (3) (2005)299–324.

[5] Y.W. Hong, W.J. Huang, F.H. Chiu, C.C.J. Kuo, Cooperativecommunications in resource-constrained wireless networks, IEEESig. Process. Mag. 24 (3) (2007) 47–57.

[6] A.H. Ibrahim, Z. Han, K.J. Ray Liu, Distributed energy-efficientcooperative routing in wireless networks, IEEE Trans. WirelessCommun. 7 (10) (2008) 3930–3941.

[7] J. Zhang, Q. Zhang, Cooperative routing in multi-source multi-destination multi-hop wireless networks, in: The 27th Conferenceon Computer Communications (IEEE INFOCOM 2008), Phoenix, AZ,USA, April 2008, pp. 2369–2377.

[8] A.K. Sadek, W. Yu, K.J. Ray Liu, On the energy efficiency ofcooperative communications in wireless sensor networks, ACMTrans. Sensor Networks 6 (1) (2009).

[9] M. Chen, T. Kwon, S. Mao, Y. Yuan, V. Leung, Reliable and energy-efficient routing protocol in dense wireless sensor networks, Int. J.Sensor Networks 4 (12) (2008) 104–117.

[10] R. Urgaonkar, M.J. Neely, Delay-limited cooperative communicationwith reliability constraints in wireless networks, in: IEEE INFOCOM2009, Rio de Janeiro, June 2009, pp. 2561–2565.

[11] X. Liang, M. Chen, Y. Xiao, I. Balasingham, V.C.M. Leung, A novelcooperative communication protocol for QoS provisioning inwireless sensor networks, in: 5th International Conference onTestbeds and Research Infrastructures for the Development ofNetworks & Communities and Workshops, 2009 (TridentCom2009), 2009, pp. 1–6.

[12] L. Xuedong, B. Ilangko, L. Victor, C.M., Cooperative communicationswith relay selection for QoS provisioning in wireless sensor

networks, in: GLOBECOM’09: Proceedings of the 28th IEEEConference on Global Telecommunications, Honolulu, Hawaii, USA,2009, pp. 3229–3236.

[13] C. Perkins, E. Belding-Royer, S. Das, Ad hoc On-Demand DistanceVector (AODV) Routing, RFC Editor, 2003.

[14] L.P. Kaelbling, M.L. Littman, A.P. Moore, Reinforcement learning: asurvey, J. Artific. Intell. Res. 4 (1996) 237–285.

[15] M. Lee, D. Marconett, X. Ye, S.J. Ben Yoo, Cognitive networkmanagement with reinforcement learning for wireless meshnetworks, in: Proceedings of the 7th IEEE international conferenceon IP operations and management (IPOM’07), April 2007, pp. 168–179.

[16] D. Djenouri, I. Balasingham, New QoS and geographical routing inwireless biomedical sensor networks, in: Proceedings of SixthInternational Conference on Broadband Communications,Networks, and Systems, 2009. BROADNETS 2009, Madrid, Spain,14–16 September 2009, pp. 1–8.

[17] A.V. Zykina, A lexicographic optimization algorithm, Autom. RemoteControl 65 (2004) 363–368.

[18] The Network Simulator – ns-2.30. <http://www.isi.edu/nsnam/ns/ns-build.html>.

[19] Y. Yuan, Z. He, M. Chen, Virtual MIMO-based cross-layer design forwireless sensor networks, IEEE Trans. Vehicul. Technol. 55 (3) (2006)856–864.

[20] B. Gui, L. Dai, L.J. Cimini, Routing strategies in multihop cooperativenetworks, IEEE Trans. Wireless Commun. 8 (2) (2009)843–855.

[21] A. Bletsas, H. Shin, M.Z. Win, Cooperative communications withoutage-optimal opportunistic relaying, IEEE Trans. WirelessCommun. 6 (9) (2007) 3450–3460.

[22] L.C. Wang, H. Chen, Partner selection schemes for cooperative multi-hop relay networks, in: Proceedings of the 4th Annual InternationalConference on Wireless Internet (WICON), Maui, Hawaii, USA,2008.

[23] W. Ge, J. Zhang, G. Xue, Cooperative geographic routing in wirelesssensor networks, in: IEEE Military Communications Conference(MILCOM 2006), Washington, DC, October 2006, pp. 1–7.

[24] Q. Chen, Q. Zhang, Z. Niu, QoS-aware cooperative and opportunisticscheduling exploiting multiuser diversity for rate-adaptive ad hocnetworks, IEEE Trans. Vehicul. Technol. 57 (2) (2008) 1113–1125.

[25] H. Mahmood, C. Comaniciu, Interference aware cooperative routingfor wireless ad hoc networks, Ad Hoc Networks 7 (1) (2009)248–263.

[26] I. Ben-Tsion, I. Kitroser, Y. Ben-Shimol, Interference-awarecooperative routing in wireless networks, in: Sixth InternationalConference on Broadband Communications, Networks, and Systems(BROADNETS 2009), 2009, pp. 1–8.

[27] N. Ahmed Ben, S. Sidi-Mohamed, G. Yacine, B. André-Luc, Enhancedrelay selection decision for cooperative communication in energyconstrained networks, in: WD’09: Proceedings of the 2nd IFIPconference on Wireless days, Paris, France, 2009, pp. 271–275.

[28] A.E. Khandani, J. Abounadi, E. Modiano, L. Zheng, Cooperative routingin static wireless networks, IEEE Trans. Commun. 55 (11) (2007).

[29] R.S. Sutton, A.G. Barto (Eds.), Reinforcement Learning: AnIntroduction, MIT Press, Cambridge, MA, USA, 1998.

[30] M.M. Rashid, M.M. Alam, M.A. Razzaque, C.S. Hong, Congestionavoidance and fair event detection in wireless sensor networks,IEICE Trans. Commun. E90-B (12) (2007) 3362–3372.

[31] C. Park, P.H. Chou, eCAM: ultra compact, high data-rate wirelesssensor node with a miniature camera, in: IEEE SenSys 2006, USA.

[32] B. Warneke, S.J. Kristofer, Smart dust: communicating with a cubic-millimeter computer, Class. Pap. Comput. Logic 1 (1) (2001)372–383.

[33] X. Wang, X. Zhang, G. Chen, Q. Zhang, Opportunistic cooperation inlow duty cycle wireless sensor networks, in: Proc. Of IEEE ICC,Workshops on Cooperative and Cognitive Mobile Networks(CocoNet), Cape Town, South Africa, May 2010.

[34] H. Jiao, M. Ingram, F. Li, A cooperative lifetime extension MACprotocol in duty cycle enabled wireless sensor networks, in: IEEEMILCOM’11.

[35] J. Lin, M.A. Ingram, SCT-MAC: a scheduling duty cycle MAC protocolfor cooperative wireless sensor network, in: IEEE ICC, 2012.

[36] A.A. Prudius, Adaptive Random Search Methods for SimulationOptimization, Ph.D. Dissertation, Georgia Inst. Technol., Atlanta,2007.

[37] J.C. Spall, Introduction to Stochastic Search and Optimization:Estimation, Simulation, and Control, Wiley, New York, 2003.

oc Networks 19 (2014) 28–42

Md. Abdur Razzaque (S’09-M’10) received

the B.S. degree in Applied Physics and Elec-tronics and M.S. degree in Computer Sciencefrom the University of Dhaka, Bangladesh in1997 and 1999, respectively. He obtained hisPhD degree in Computer Engineering fromKyung Hee University, South Korea in August,2009. He had worked as an Assistant Professorin the department of Computer Science andInformation Technology of Islamic Universityof Technology (IUT), Gazipur, Bangladesh. Heis an Assistant Professor (now on leave) in the

Department of Computer Science and Engineering, University of Dhaka,Bangladesh. He is now working as a research professor in Kyung HeeUniversity, South Korea. His research interest is in the area of modeling,

42 M.A. Razzaque et al. / Ad H

analysis and optimization of communication protocols and architecturesfor wireless networks, cooperative communications, distributed systems,etc. He is a member of editorial boards of International Journal on Internetand Distributed Computing Systems (IJIDCS) and Journal of Computingand Applications (JCA). He is a TPC member of ICOIN 2011, Malaysia. He isa regular reviewer of IEEE Transactions on Wireless Communications,Journal of Communications and Networks, Journal of Sensors, Annals ofTelecommunications, and many conferences. He is a member of IEEE, IEEEComputer Society and KIPS.

Mohammad Helal Uddin Ahmed received theB.S. and M.S. degrees in Applied Physics andElectronics from the University of Dhaka,Bangladesh in 1996 and 1998, respectively.Since 2000 he has been working as an AssistantProfessor (now in leave) in the department ofManagement Information Systems, Universityof Dhaka, Bangladesh. He is now a PhD studentof the department of Computer Engineering inKyung Hee University, South Korea. Hisresearch interest is in the area of design andanalysis of wireless networking protocols.

Choong Seon Hong received his B.S. and M.S.degrees in electronic engineering from KyungHee University, Seoul, Korea, in 1983, 1985,respectively. In 1988 he joined KT, where heworked on Broadband Networks as a memberof the technical staff. From September 1993,he joined Keio University, Japan. He receivedthe PhD degree at Keio University in March1997. He had worked for the Telecommuni-cations Network Lab, KT as a senior memberof technical staff and as a director of the net-working research team until August 1999.

Since September 1999, he has been working as a professor of the School ofElectronics and Information, Kyung Hee University. He has served as aprogram committee member and an organizing committee member for

International conferences such as SAINT, NOMS, IM, APNOMS, ICOIN,CSNM, ICUIMC, E2EMON, CCNC, ADSN, ICPP, DIM, WISA, BcN, ManFI,TINA, etc. His research interests include ad hoc networks, future Internet,wireless networks, network security, and network management. He is asenior member of the IEEE and a member of the ACM, IEICE, IPSJ, KICS,KIISE, KIPS, and OSIA.

Sungwon Lee received his B.S. and Ph.Ddegrees from Kyung Hee University, Korea. Heis a professor of the Computer EngineeringDepartment at Kyung Hee University, SouthKorea. Dr. Lee was a senior engineer of Tele-communications and Networks Division atSamsung Electronics Inc., during 1999–2008.He is an editor of the Journal of Korean Insti-tute of Information Scientists and Engineers:Computing Practices and Letters.