1

Pacifier: High-Throughput, Reliable MulticastWithout “Crying Babies” in Wireless Mesh

NetworksDimitrios Koutsonikolas,Student Member, IEEE,Y. Charlie Hu,Senior Member, IEEE,

and Chih-Chun Wang,Member, IEEE

Abstract—In contrast to unicast routing, high-throughput re-liable multicast routing in wireless mesh networks (WMNs) hasreceived little attention. There are two primary challenges tosupporting high-throughput, reliable multicast in WMNs. T hefirst is no different from unicast: wireless links are inherentlylossy due to varying channel conditions and interference. Thesecond, known as the “crying baby” problem, is unique tomulticast: the multicast source may have varying throughputto different multicast receivers, and hence trying to satisfythe reliability requirement for poorly connected receivers canpotentially result in performance degradation for the rest of thereceivers.

In this paper, we proposePacifier, a new high-throughput reli-able multicast protocol for WMNs. Pacifier seamlessly integratesfour building blocks, namely, tree-based opportunistic routing,intra-flow network coding, source rate limiting, and round-robinbatching, to support high-throughput, reliable multicast routingin WMNs, while at the same time effectively addresses the “cryingbaby” problem. Our experiments on a 22-node 802.11 WMNtestbed show thatPacifier increases the average throughput overa state-of-the-art reliable network coding-based protocol MOREby up to 144%, while at the same time it solves the “cryingbaby” problem by improving the throughput of well-connectedreceivers by up to a factor of 14.

Index Terms—reliable multicast; wireless mesh networks(WMNs); network coding.

I. I NTRODUCTION

W IRELESS mesh networks (WMNs) are increasinglybeing deployed for providing cheap, low maintenance

Internet access (e.g. [1], [2], [3]). These networks have stati-cally deployed mesh routers that are not energy constrained,and hence the main design challenge is to improve applica-tions’ performance, in particular, to provide high throughputand reliability in network access. Indeed, recent years havewitnessed numerous “exotic” protocols that aim to improvethe throughput and reliability of unicast routing. These includeopportunistic routing (OR) protocols (e.g., [4]), protocols thatexploit inter-flow (e.g., [5]) or intra-flow (e.g., [6]) networkcoding, as well as lower layer protocols (e.g., [7]).

In contrast to unicast routing, high-throughput, reliablemul-ticast routing has received relatively little attention. Reliablemulticast routing has many important applications in WMNs,such as software updates and video/audio file downloads.

An earlier version of this paper was presented at IEEE INFOCOM 2009.This work was supported in part by NSF grants CNS-0626703 andCCF-0845968.

D. Koutsonikolas, Y. C. Hu and C. C. Wang are with the School ofElectricaland Computer Engineering, Purdue University, West Lafayette, IN, 47907USA e-mail: (dkoutson@purdue.edu, ychu@purdue.edu, chihw@purdue.edu).

These applications have a strict requirement of100% PacketDelivery Ratio (PDR), since every byte of the downloadedfile has to be received byall the receivers. This requirementmakes many of the reliable multicast protocols proposed in thepast (e.g., [8], [9], [10], [11]) inappropriate, since theycannotguarantee 100% PDR. In addition, reliability for this classofapplications cannot come at the cost of significantly reducedthroughput, unlike in many military applications [12], sinceInternet users always desire fast downloads.

The fundamental challenge in achieving reliable multicastin WMNs is no different from that of reliable unicast – thatwireless links are lossy. To overcome this, researchers haveapplied classic techniques such as Automatic Repeat reQuest(ARQ), Forward Error Correction (FEC), or combinations ofthe two. The majority of the works on reliable multicast inmultihop wireless networks either are solely based on ARQ(e.g., [13], [14]) which suffer the feedback implosion problem,or combine ARQ with congestion control (e.g., [15], [10]). Arecent work [16] studied the applicability of FEC and hybridARQ-FEC techniques, borrowed from the wired Internet, toWMNs, and showed that RMDP [17], a hybrid ARQ-FECprotocol, can achieve both reliability and high throughput.

More recently, researchers have applied network coding(NC), a technique originally developed for the wireline In-ternet, to overcome the above challenge. [18] showed thatthe operation of mixing packets resembles the operation ofrateless FEC codes. Actually, NC can be viewed as a techniqueequivalent to performing hop-by-hop FEC, without the delaypenalty incurred by the decoding operations at each hop, thatwould be required by hop-by-hop FEC. In [19], the authorswent one step further and showed that the reliability gain(expressed as the expected number of transmissions) of NCover end-to-end FEC for a wireless multicast tree of heighth

with link loss ratep is in the order ofΘ(( 11−p

)h).Practical work that exploits the idea of utilizing NC for

reliable multicast is still at a preliminary stage. MORE [6]is the only practical NC-based protocol that supports high-throughput, reliable multicast. It combines NC with OR, withthe primary goal of removing the need for coordination re-quired in opportunistic routing. However, the design of MOREalso guarantees reliability, i.e., MORE is a routing protocol forreliable file transfer, for both unicast and multicast.

A second fundamental challenge in reliable multicast, whichis unique to multicast, is the “crying baby” problem as firstpointed out in [20] in the context of multicast in the Inter-

2

net. If one receiver has a particularly poor connection, thentrying to satisfy the reliability requirement for that receivermay result in performance degradation for the rest of thereceivers. This problem also raises the interesting question ofwhat is a suitable definition of overall performance metric ifmultiple receivers are allowed to achieve uneven throughput.Regardless, a major challenge in the design of high throughput,reliable multicast protocols is whether it is possible to developa protocol that improves the throughput of well-connectedreceivers without worsening the already low throughput ofpoorly connected receivers.

In this paper, we proposePacifier, a high-throughput, reli-able multicast protocol that systematically addresses theabovetwo challenges.Pacifier seamlessly integrates four buildingblocks, namely,tree-based OR, intra-flow NC, source rate lim-iting, and round-robin batching,to support high-throughput,reliable multicast routing and at the same time solve the“crying baby” problem. First,Pacifier builds an efficientmulticast tree traditionally used by multicast protocols andnaturally leverages it for opportunistic overhearing. Second,Pacifier applies intra-flow, random linear NC to overcomepacket loss over lossy links which avoids hop-by-hop feedbackand the coordination of multicast tree forwarders in packetforwarding. Third,Pacifier applies rate limiting at the source,reducing the congestion level in the network. Fourth,Pacifiersolves the “crying baby” problem by having the source sendbatches of packets in a round-robin fashion. This functionalityallows Pacifier to improve the throughput of well-connectednodes drastically and often times of poorly connected nodes.

To evaluatePacifier, we first compare its performanceagainst MORE, using extensive realistic simulations. Oursimulations use a realistic physical model, with random signalvariations due to fading, take into account the additional packetheader overhead introduced by the use of NC and OR, and areconducted over a variety of network topologies and multicastgroups. Our simulation results show thatPacifier increasesthe average throughput of multicast receivers over MOREby 171%, while it solves the “crying baby” problem, byincreasing the maximum throughput gain for well-connectedreceivers by up to 20x. Interestingly and importantly,Pacifieralso improves the throughput of the “crying babies”, i.e., thepoorly connected receivers, by up to 4.5x.

Second, sincePacifieruses the same type of NC as MORE,and has the same memory requirements at the routers, hence,like MORE, it can be easily implemented on commodityhardware. To demonstrate this, we present an application-layer implementation ofPacifier and MORE on Linux andtheir performance evaluation on a 22-node 802.11 WMNtestbed deployed in two academic buildings on the PurdueUniversity campus. Our testbed results verify the simulationresults showing thatPacifier increases the average multicastthroughput over MORE by 83-114%, while the maximumthroughput gain for well-connected receivers can be as highas14x, and the maximum throughput gain for the “crying baby”itself can be as high as 5.4x.

In summary, we make the following contributions:• We identify two challenges in supporting high-

throughput, reliable multicast in deployed WMNs, i.e.,

high loss rates [21], and the “crying baby” problem. Inparticular, to our best knowledge,this is the first work thatconsiders the well-known in the wired Internet “cryingbaby” problem in the context of reliable multicast inmultihop WMNs.1 We show that this problem, if ignored,can severely impact the performance of state-of-the-artreliable protocols.

• We proposePacifier, the first practical multicast pro-tocol that simultaneously addresses both challenges: itguarantees 100% PDR for all multicast receivers, whilesimultaneously solving the “crying baby” problem byoffering significant throughput improvements for bothwell-connected and poorly-connected receivers over astate-of-the-art protocol.

• We present the design ofPacifierwhich seamlessly inte-grates four building blocks, namely,tree-based OR, intra-flow NC, source rate limiting, and round-robin batching.While the design ofPacifier is based on the numerousprinciples and techniques developed over the past fifteenyears in the field of reliable multicast, we show the noveltechnique of NC is the key to the successful integrationof all four building blocks.

• We present extensive simulations with a realistic physicalmodel showing thatPacifieroffers significant throughputimprovements over the state-of-the-art MORE, and, un-like MORE, it solves the “crying baby” problem. Startingwith a basic version ofPacifier and adding one buildingblock at a time, we show the additional benefits of eachbuilding block.

• We present an application-layer implementation ofPaci-fier and MORE and their evaluation on a 22-node 802.11WMN testbed deployed in two academic buildings onthe Purdue University campus. Our testbed experimentsconfirm the simulation findings.

• Although an implementation at the application layerunavoidably results in some performance degradation,compared to an implementation closer to the MAC layer,it significantly simplifies debugging, deployment, anduse. This allows for fast modification and evaluation ofvarious operations that affect the performance of thisnew class of NC-based routing protocols, other thanNC itself, such as forwarding list selection, batchingscheme, and rate control, and helps to understand theirimpact on the performance of these protocols. To fa-cilitate further research of this subject, we make thesource code of our implementation publicly available athttps://engineering.purdue.edu/MESH/pacifier/.

The rest of the paper is organized as follows. We begin withan overview of related work in the field of reliable multicast(Section II) and a brief description of MORE, the state-of-the-art NC-based reliable multicast protocol (Section II-A). Wethen present the design ofPacifier and describe its variousbuilding blocks in Section III. We discuss our simulationevaluation methodology in Section IV-A and present extensive

1BMCC [10], a multicast protocol for mobile ad hoc networks, dropspackets on the path towards the worst receiver, in order to prevent that receiverfrom holding back the rest of the receivers. This solution isnot applicable infile-download applications that require 100% PDR.

3

simulation results in Section IV-B. Section V describes theimplementation and evaluation ofPacifier and MORE on awireless testbed. Finally, Section VI concludes the paper.

II. RELATED WORK

In spite of the extensive research on reliable multicast in thewired Internet, which went through the development of ARQ-based schemes (e.g., [22], [20]), to FEC schemes (e.g., [23]),to hybrid ARQ-FEC schemes (e.g., [24], [17], [25]), to ratelesscodes [26], [27], [28], [29], the majority of the work on reliablemulticast in multihop wireless networks have used the tradi-tional ARQ techniques. A survey on reliable multicast proto-cols for ad hoc networks [30] classifies them into deterministicand probabilistic ones, depending on whether data deliveryis fully reliable or not. Deterministic protocols (e.g., [13],[31], [32], [12], [33], [15]) provide deterministic guaranteesfor packet delivery ratio, but they can incur excessive highoverhead and drastically reduced throughput. On the otherhand, probabilistic protocols (e.g., [8], [9]) incur much lessoverhead compared to the former, but they do not offer harddelivery guarantees. Using rateless codes requires the source tocontinuously send packets, which can cause congestion in thebandwidth-limited wireless networks. Recently, [16] studiedthe applicability of FEC and hybrid ARQ-FEC techniques,borrowed from the wired Internet, to WMNs, and showed thatRMDP [17], a hybrid ARQ-FEC protocol, can provide bothreliability and high throughput.

Most recently, intra-flow network coding has been proposedas a whole new approach to reliable routing. NC in theoryis equivalent to hop-by-hop FEC [18], [19], and hence themaximum amount of redundancy injected from any node inthe network is determined by the lossiest link of the tree,and not by the lossiest path from the source to any receiver,unlike in end-to-end FEC. However, hop-by-hop FEC/NCalso has its practical drawbacks; it requires buffering packetsat each node for decoding/re-encoding (in case of FEC) oronly re-encoding (in case of NC). Due to the constraintson the buffer size and on packet delay, NC needs to sendpackets in batches, i.e., the source needs to wait till a batch isreceived by all receivers before proceeding to the next batch.This introduces the “crying baby” problem, where the poorlyconnected receivers slow down the completion time of well-connected receivers.

To our best knowledge, MORE is the only NC-based pro-tocol for high-throughput, reliable multicast routing (thoughit is also for unicast). Due to its significance, and since wewill comparePacifier against it in our evaluation, we presenta brief overview of MORE in Section II-A. To our knowl-edge, the only other practical NC-based multicast protocolis CodeCast [11], which exploits NC forimproving but notguaranteeingreliability in multimedia multicast applicationsin mobile ad hoc networks.

A. Overview of MORE

MORE [6] is an OR protocol for reliable file transfer.MORE is implemented as a shim between the IP and the802.11 MAC layer. In the following, we describe the mainfunctions of MORE, focusing on its multicast operation. We

briefly review its two major features: forwarding node (FN)selection and packet batching.

FN selection. MORE uses the ETX metric [34], based onloss rate measurements, to select the possible FNs. ETX isequal to the expected number of transmissions required tosuccessfully transmit a packet from the source to a destination.For each destination the source includes in the FN list thenodes whose ETX distance to that destination is shorter thanthe source’s distance. Also, for each FN the source includesa TX credit in the FN list. TheTX credit is the expectednumber of transmissions a node should make for every packetit receives from a node farther from a destination in the ETXmetric, in order to ensure that at least one node closer to thedestination will receive the packet.

The algorithm for FN selection and TXcredit calculationis run at the source. The algorithm starts by assuming thatevery node is a candidate FN for a source-destination pairand calculates the expected number of transmissions thisnode would make. It then prunes nodes that are expected toperform less than 10% of the total transmissions and assignsTX credits to the remaining ones, which form a belt of FNsthat connect the source to the destination. The algorithm isrepeated for each destination; in the end the belts formedfor each destination are merged into the final FN set. Ifan FN belongs to more than one belts, (i.e., for more thanone destination), the algorithm calculates a different expectednumber of transmissions for each of the belts it belongs to. Itsfinal TX credit is then calculated using the maximum numberof transmissions among these belts.

Batching and Coded Packet Forwarding. In MORE, thesource breaks a file into batches ofk packets. Whenever theMAC is ready to send a packet, the source creates a randomlinear combination of thek packets of the current batch andbroadcasts the encoded packet. Each packet is augmented withits code vector, the batch ID, the source and destination IPaddresses and the list of FNs for that multicast, with theirTX credits.

Packets are broadcast at the MAC layer, and hence theycan be received by all nodes in the neighborhood. Whena node hears a packet, it checks if it is in the packet’sFN list. If so, the node checks if the packet islinearlyindependentwith all the packets belonging to the same batchthat it has already received. Such packets are calledinnovativepacketsand are stored in a buffer. Non-innovative packetsare discarded. Every time a node receives a packet from anupstream node, it increments itscredit counterby its assignedTX credit included in the packet header. If itscredit counteris positive, whenever the MAC is ready to send a packet, thenode creates a linear combination of the innovative packetsithas received so far and broadcasts it. Broadcasting a packetdecrements thecredit counterby one unit.

A multicast receiver decodes a batch once it collectsk

innovative packets from that batch. It then sends an ACKback to the source along the shortest ETX path in a reliablemanner. The source keeps sending packets from the same batchuntil all receivers have decoded and acknowledged the currentbatch; it then proceeds to the next batch. Whenever a receiver

4

acknowledges the current batch, the source removes the FNsresponsible for forwarding packets only towards that receiverand recalculates the credits for the remaining FNs, using themaximum number of transmissions taken only over FN beltsto receivers that have not yet acknowledged the batch.

III. Pacifier DESIGN

The design ofPacifier addresses several weaknesses ofMORE. In particular, the belt-based forwarding in MORE canbe inefficient for multiple receivers, MORE lacks source ratelimiting which can lead to congestion in data dissemination,and MORE suffers the “crying baby” problem.

For clarity, we present the design ofPacifierin several steps.We first present a basic version ofPacifier, which consistsof several building blocks: tree-based opportunistic multicastrouting, batching and NC-based forwarding, and credit cal-culation. The basic version guarantees reliability and alreadyincreases throughput compared to MORE, but does not solvethe “crying baby” problem. We then present two optimizations:source rate limiting which reduces congestion and furtherimproves the throughput, and round-robin batching, whichsolves the “crying baby” problem.

A. Tree-based Opportunistic Routing

We argue that the use of OR in the form used in MOREis an overkill for multicast and it can lead to congestion, fortwo reasons. First, even for a single destination, congestioncan occur if too many nodes act as FNs, or if the FNs arefar from each other and they cannot overhear each other’stransmissions [35]. The situation is worsened when the numberof flows increases, since almost all nodes in the network mayend up acting as FNs. Such performance degradation wasobserved in the evaluation of MORE in [6] for many unicastflows; the situation for many hypothetical unicast flows isnot very different from a source to many multicast receivers.Second, the benefit of overhearing of broadcast transmissions,which is exploited by OR in MORE, is naturally exploitedin a fixed multicast tree, where the use of broadcast allowsnodes to receive packets not only from their parent in themulticast tree, but also from ancestors or siblings, essentiallytransforming the tree into a mesh. We note this property ofopportunistic reception of broadcast transmissions has beenpreviously exploited in the design of some of the first multicastprotocols for multihop wireless networks (e.g., ODMRP [36]),for improving the PDR.

The above observation motivates a simple multicast-treebased OR design. Specifically,Pacifier starts by building amulticast tree to connect the source to all multicast receivers.The tree is a shortest-ETX tree, constructed at the sourceby taking the union of all the shortest-ETX paths from thesource to the receivers, which in turn are based on periodicloss rate measurements.2 The multicast tree is reconstructedat the source every time some receiver completes a batch(Section III-A1) and notifies the source.

2As [6] argues, periodic link loss rate measurements and their distributionto all nodes in the network are required in all state-of-the-art routing protocols,and the overhead this process incurs is not consideredPacifier-specific.

1) Batching and Coded Forwarding:As in MORE, thesource and the FNs inPacifier use intra-flow random linearNC. The hop-by-hop nature of NC requires the source to breaka file into small batches of packets so that the packet headeroverhead, encoding/decoding time, and memory requirementsat the FNs remain low. We selected a batch size ofk = 32packets, same as in [6], [37]. For each batch, the source sendsrandom linear combinations of the packets belonging to thatbatch. The random coefficients for each linear combinationare selected from a Galois Field of size28, again same asin [6]. Intermediate FNs store all the innovative packets ofthe batch and also send random linear combinations of them.Every transmitted encoded packet is augmented with its codingvector, i.e., the random coefficients used to generate thatpacket. When a receiver receives anyk linearly independentcoded packets of a batch, it decodes the batch to obtain thek original packets. It then sends an ACK back to the sourcealong the shortest ETX path in a reliable manner.

To achieve reliability, this basic version ofPacifieruses thefollowing batch termination scheme: the source keeps trans-mitting packets from the same batch, untilall the receiversacknowledge this batch. Such a transmission scheme howeverintroduces the “crying baby” problem as the completion timeof each batch is limited by that of the worst receiver.

2) How many packets does an FN send?:Despite the useof a multicast tree for data forwarding, the use of 802.11broadcast effectively enables opportunistic routing, i.e., a nodecan opportunistically receive packets from nodes other thanits parent in the multicast tree. If a node forwards everypacket it receives, a receiver could potentially receive eachpacket originated from the source multiple times. To avoidunnecessary transmissions, we need to carefully analyzehowmany (coded) packets an FN should send upon receiving adata packet.

Since, in practice, an FN should be triggered to transmitonly when it receives a packet, we derive the number oftransmissions each FN needs to make for every packet itreceives. We define this number as the TXcredit for thatFN. Thus, inPacifier, an FN nodej keeps a credit counter.When it receives a packet from anupstreamnode (definedbelow), it increments the counter by its TXcredit. When the802.11 MAC allows the node to transmit, the node checkswhether the counter is positive. If yes, the node creates acoded packet, broadcasts it, then decrements the counter. Ifthe counter is negative, the node does not transmit. We notethat opportunistic reception of data packets is always allowed,even from downstream nodes. The credit calculation is on howmany packets to be transmitted by the FN upon receiving adata packet from an upstream node.

In the analysis, we focus on disseminating one data packetfrom the root down the multicast tree. Our analysis is basedon the simple principle that in disseminating a packet fromthe root, each FN in the multicast tree should ensure thateach of its child nodes receives the packetat least once. Notethis principle slows down a parent node to wait for the worstchild and creates the “crying baby” problem at each FN, butis consistent with the batch termination scheme of this basicversion ofPacifier.

5

We assume an FNj sends packets after receiving fromany nodes with lower ETX distance from the root to them,i.e., j’s upstream nodes. These nodes are likely to receivepackets from the root beforej.3 We also assume that wirelessreceptions at different nodes are independent, an assumptionthat is supported by prior measurements [38].

Let N be the number of FNs in the multicast tree rootedat s. Let ǫij denote the loss probability in sending a packetfrom nodei to nodej. Let zj denote the expected numberof transmissions that FNj must make in disseminating onepacket (from the root) down the multicast tree. LetC(j)denote the set of child nodes ofj in the multicast tree, andA(j) denote the set ofj’s upstream nodes.

The expected number of packets thatj receives fromancestor nodes is

∑i∈A(j) zi(1 − ǫij). Recall j’s objective

is to make sure each of its child nodes receives at leastonepacket. Since each child nodek ∈ C(j) has already overheard∑

i∈A(j) zi(1 − ǫik) from nodej’s ancestors, the amount ofpackets nodej actually needs to forward for childk is:

Ljk = min(∑

i∈A(j)

zi(1 − ǫij), 1) −∑

i∈A(j)

zi(1 − ǫik) (1)

The min operation ensures thatj does not forward the samepacket more than once, in case it receives it from more thanone FNs. Note for the source nodes, Lsk = 1 for all k ∈ C(s).

Since the expected number of times nodej has to transmita packet to ensure that its childk will receive one packet is

11−ǫjk

, the expected number of transmissions ofj for child k

to receiveLjk is:

zjk =Ljk

1 − ǫjk

(2)

Since packets are broadcast, they can be received by morethan one child nodes at a time. Hence, the expected numberof transmissions nodej has to make to ensure that each childnode hasonepacket is:

zj = maxk∈C(j)zjk (3)

zj and Ljk are inter-dependent, and can be calculatedrecursively inO(N2) operations, i.e., by traversing the FNsin the increasing order of their ETX values from the source.Since the order of FNs is well-defined, there are no loops inthe credit calculation.

For each data packet the source sends down the multicasttree (which may require multiple transmissions), FNj receives∑

i∈A(j) zi(1 − ǫij). Thus, the TXcredit of nodej is:

TX creditj =zj∑

i∈A(j) zi(1 − ǫij)(4)

B. Source Rate Limiting

Recent studies have shown the importance of adding ratecontrol to NC-based unicast routing protocols, which exploitMAC layer broadcast [37], [39], [35], [40]. However, end-to-end rate control in multicast is much more complex than in

3In contrast, MORE’s credit calculation was based on the ordering of FNsaccording to their ETX distance to the destination node. It is unclear thatnodes with larger ETX distance to the destination will receive the packetfrom the root sooner.

unicast, and there is no widely accepted solution so far. Inthe version ofPacifierpresented so far, the use of TXcreditsimplements a form of rate control at which each intermediateFN injects packets into the network. However, the source canpotentially send out all the packets in a batch unpaced.

To add rate control to the source, we exploit the broadcastnature of the wireless medium and apply a simple form ofbackpressure-based rate limiting, inspired by BMCC [10]. Thebasic idea is to have the source wait until it overhears its childnodes forward the previous packet it sent before it transmits thenext packet. Since the number of transmissions by the sourcezs has already factored in packet losses to its child nodes,the source does not need to worry about losses of individualtransmissions, i.e., it does not need to wait until all its childnodes forward each packet it sends out. In fact, it is not evensure that every of its transmissions will trigger a transmissionat each of its child nodes, as some nodes may have negativecredit counters. Instead, the source waits until it overhears atransmission fromany of its child nodes or until a timeoutbefore it sends the next packet in the batch.

The work in [10] does not discuss how to set the timeout.In [41], the authors suggested a heuristic timeout of3×Tp forthe backpressure-based unicast version of BMCC, whereTp

is the transmission time of one data packet, which depends onthe packet size and the MAC data rate. The factor of 3 is toaccount for the contention time preceding each transmission.Following the same reasoning, inPacifier, we set the timeoutto

∑j∈C(s) TX creditj × 8 × Tp. This choice for the timeout

reflects the fact that inPacifiera transmission from the sourcewill trigger on average

∑j∈C(s) TX creditj transmissions

from its child nodes, which in the worst case can be sequential,and also the fact that in multicast contention near the sourceis in general higher.

C. Solving the “Crying Baby” Problem

In MORE, the source keeps transmitting packets from thesame batch until all the receivers acknowledge that batch, asshown in Figure 1(a). This policy makes the protocol suscep-tible to the “crying baby” problem, since if the connection toone receiver is poor, that receiver can slow down the rest of thereceivers. The basic version ofPacifier we have described sofar suffers from the same problem. Note the problem wouldnot exist if the intermediate routers had unlimited memoryand hence the whole file were coded into one batch, andthere were no constraints on the delay. In the following, wedescribe a practical solution to the problem, which requiresno more memory than MORE or our basic version, i.e., FNsstill maintain only one batch at a time in their memory.

In the proposed scheme, the source iteratively sends thebatches of a file in around-robinfashion, for as many roundsas required, until it has received ACKs for all batches fromall the receivers, as shown in Figure 1(b). In detail, the sourcemaintains a counterCsi

for each batchi which is equal tothe number of remaining packets the source has to transmitfor that batch. The counter for batchi is initialized asCsi

=zs×k, wherezs is calculated in Equation (3) andk is the batchsize, and it is decremented every time a packet from batchi

is transmitted. Each intermediate FN forwards coded packets

6

k

B−1B

321

.

..

(a) Sequential batch transmis-sion in MORE. Each batch isacknowledged byall the re-ceivers before the source movesto the next batch.

k

BB−1

321

...

(b) Round-robin batch trans-mission inPacifier. The sourcemoves to the next batch everytime onereceiver acknowledgesthe current batch.

Fig. 1. Two different ways of transmittingB batches ofk original packetseach: sequential (as in MORE), and round-robin (as inPacifier ). For bettervisualization, we assume here (not true in the actual operations of theprotocols) that the same total amount of redundancy is required to be sent foreach batch.

according to its TXcredit, and only buffers packets belongingto the current batch; when it receives the first packet from anew batch, it flushes its buffer and starts buffering packetsfrom the new batch.

The source determines when to switch to work on the nextbatch as follows. It sends packets from batchi until either(1) Csi

reaches zero or (2) it receives fromone receiveracknowledging completion of this batch; it then moves to thenext batch for which there are still receivers that have notacknowledged it. When the source finishes with the last batchB, it starts the next round by going back to the first batch forwhich it has not received ACKs from all receivers. For eachsuch batch it revisits, it recalculates the multicast tree (i.e.,the FNs) and the TXcredit values for the FNs based on thereceivers that have not sent ACKs and resetsCsi

= zs × k

using the newly calculatedzs.The above batch switching policy is critical to achieving

high throughput for both well- and poorly connected receivers.On one hand, if one receiver has already acknowledged thecurrent batch before the source sends all the scheduled packetsfor that batch, not moving to the next batch at the sourcewill reduce the throughput of that receiver. On the other hand,after all the scheduled packets have been sent out, allowingthesource to move to the next batch only when it receives an ACKfrom one receiver can be inefficient, as ACKs may delay toreach the source. This policy can impact even the throughputof a well-connected receiver, if, for example, congestion in theneighborhood around the source prevents the ACK to traversethe reverse path. Its impact, though, is more severe on theworst receivers. Assume all the well-connected receivers haveacknowledged the batch in previous rounds. In the next around,for the remaining (worst) receivers, an ACK may take a longtime to reach the source, since it has to traverse several hops,and, at each hop, it competes with data packets traversingdown the tree. This could result in the source sending un-necessary packets from the current batch while waiting foran ACK to arrive. Instead, quickly moving to the next batchafter finishing the scheduled packets, without waiting for anyACK, ensures that the “network pipe” is always filled withuseful packets, and a delayed ACK will have little impacton the performance. Our evaluation in Sections IV-B, V-D

shows that switching batch wheneither of the two conditionsis satisfied results in significant throughput improvementsoverMORE for both well-connected receivers and the worst ones(i.e., the “crying babies”). We note previously [42] also noticedthis “stop-and-wait” policy (also used in MORE) can result insignificantly low throughput, in the context of unicast, as thenetwork scales.

The above round-robin batching scheme is similar to thedata carousel first introduced in Fcast [25], an FEC-basedprotocol. However, it is important to note that the use ofNC in Pacifier makes round-robin batching fundamentallymore efficient than in the FEC-based protocol, by eliminatingduplicate (i.e. non-innovative) packets. In Fcast, each batch ofk packets is encoded to producen > k packets (typical valuesfor k, n, are 32 and 255, respectively), and a receiver candecode a batch if it receives anyk out of thosen packets. Oncethe source finishes transmission of alln encoded packets fromall B batches, it starts a new round where itretransmits againthe samen encoded packetsfor each batch. This results inmany receivers receiving duplicate packets and further delaysdecoding. For example, after all receivers have received thefirst l < k packets of a batch in a round, they will receivethose samel packets in subsequent rounds. In contrast, inPacifier, the sourcesends different random combinations ofthe original k packets in every round, and in addition, everyFN also mixes the packets it receives and sends out newcombinations of them. This guarantees that, with very highprobability, a receiver will never receive a duplicate packet.

1) Adjusting TXcredit Calculation: In the basic versionof Pacifier (Section III-A2), we defined the TXcredit of anFN as the expected number of packets it has to transmit forevery packet it receives from its upstream nodes, in order toensure thatall of its child nodes will receive one packet. Thisdefinition is consistent with the batch termination scheme ofthe basic scheme, i.e., the source completes a batch when itreceives ACKs from all receivers. However, it is inconsistentwith the round-robin batching scheme, which aims to preventpoorly connected receivers from slowing down well-connectedreceivers. Hence under the round-robin batching, we adjustthedefinition of TX credit of an FN to be the expected numberof packets it has to transmit for every packet it receives fromits upstream nodes, in order to ensure thatat least oneof itschild nodes will receive one packet. To realize this change,wesimply change themax operator tomin in Equation (3). Wenote this new definition is also consistent with the policy ofmoving to the next batch whenever any receiver acknowledgesthe current batch.

2) Intricacies in TXcredit Calculation: There is a subtletyin the above adjustment to the TXcredit calculation under theround-robin batching scheme, i.e., changing themax operatorto min in Equation (3). The derivation of Equation (3) isbased on expected number of opportunistic packet receptions(based on the ETX measurements). However, in the actualdissemination of any given batchi, it is possible that the actualpacket reception is below or above the expected value. In thelater case, the best receiver will successfully receive allpacketsfor that batch, and it is the correct thing to do for the sourceto move on to the next batch. However, in the former case,

7

the best receiver could be a few packets short of receiving thewhole batchi, and hence if the source moves on to the nextbatch, even the best receiver has to wait for a whole roundbefore the source transmits again packets from batchi. Onthe other hand, if we had let the source send some additionalpackets to those predicted by Equation (3), there is a goodchance that the best receiver would have finished in the currentround; this would increase the throughput of the best receiver.The challenge here is that it is unknown beforehand whetherthe opportunistic reception in any particular batch is above orbelow the expectation, and hence those extra packets sent bythe source for a batch can potentially elongate each batch andreduce the throughput of the best receiver.

To facilitate studying the above subtlety in the TXcreditcalculation under the round-robin batching scheme, we intro-duce a tunable knob in Equation (3). Essentially, we definethe expected number of transmissions nodej makes to itschild nodes aszj = mink∈C(j)zjk +knob∗ (maxk∈C(j)zjk −

mink∈C(j)zjk). Setting knob to 1 changes the objective toensuring all child nodes receive a packet at least once, whilesettingknob to 0 changes the objective to ensuring at least onechild node receives a packet at least once. In Section IV-B5,weevaluate the impact of this knob by comparing the performanceof Pacifier under different values ofknob.

IV. SIMULATION STUDIES

We first evaluate the performance ofPacifierby comparingit against MORE using extensive simulations. The use of asimulator allowed us to evaluate the performance of the twoprotocols in large networks, using a diverse set of topologies,which are difficult to create in a testbed. We notePacifierusesthe same type of NC and has the same memory requirementsand the same fields in the packet header as MORE,4 andhence it can be easily implemented in practice. We presentan implementation study ofPacifier in the next section.

A. Evaluation Methodology

Simulation Setup. We used the Glomosim simulator [43],a widely used wireless network simulator with a detailedand accurate physical signal propagation model. Glomosimsimulations take into account the packet header overheadintroduced by each layer of the networking stack, and alsothe additional overhead introduced by MORE orPacifier. Forthe implementation of MORE, we followed the details in [6].

We simulated a network of 50 static nodes placed randomlyin a 1000m × 1000m area. The average radio propagationrange was 250m, the average sensing range was 460m, andthe channel capacity was 2Mbps. TheTwoRaypropagationmodel was used. To make the simulations realistic, we addedfading in our experiments. The Rayleigh model was used,as it is appropriate for WMN environments with many largereflectors, e.g., walls, trees, and buildings, where the senderand the receiver are not in Line-of-Sight of each other. Becauseof fading, transmission and sensing range are not fixed; theyactually vary significantly around their average values.

4Pacifier only includes the list of FN nodes in the header, sorted inincreasing ETX distance from the source. It does not requireinformationabout the edges of the tree.

TABLE IVERSIONS OFMORE AND Pacifier EVALUATED IN OUR

STUDY. ALL VERSIONS INCLUDE INTRA-FLOW NC.Name DescriptionMORE MORE [6] optimized with

scenario-specific pruning thresholdTREE Tree-based OR

TREE+RL Tree-based OR, source rate limitingTREE+RL+RRB Tree-based OR, source rate limiting,

(Pacifier) and round-robin batching

We simulated each protocol on 10 different randomly gen-erated topologies (scenarios), i.e., placement of the 50 nodes.For each scenario, we randomly generated a multicast groupconsisting of 1 source and 9 receivers. The source sent a12MB file, consisting of 1500-byte packets, transmitting atthe maximum rate allowed by the MAC (in case of MORE)or by the MAC and the backpressure-based rate limiting (incase ofPacifier). We present the result for each scenario andthe average result over all 10 scenarios.

Following the methodology in [6], [4], we implemented anETX measurement module in Glomosim which was run for 10minutes prior to the file transfer for each scenario to computepairwise delivery probabilities. During these 10 minutes,eachnode broadcasts a 1500-byte packet every second, and keepstrack of the packets it receives from its neighbors. At the endof the 10-minute duration, all the measurements are distributedto all the nodes. The source uses these measurements tocompute the forwarding lists and the transmission credits forthe two protocols. There was no overhead due to loss ratemeasurements during the file transfer.

Evaluation Metrics. We used the following metrics:

Average Throughput:The file size (in bytes) divided by thetotal time required for a receiver to collect the necessarynumber of packets for decoding, averaged over all receivers.

Total number of data packet transmissions:5 The total numberof data packets broadcast by the source and the FNs.

Source Redundancy:The total number of encoded data packetssent by the source divided by the file size. It gives an estimateof the redundancy injected in the network by the source.

Note that we did not use the PDR as a metric, since bothprotocolsguarantee100% PDR.

B. Simulation Results

We start by optimizing MORE’s pruning strategy as thedefault strategy appears to cause frequent network partition.We then proceed to evaluate the incremental performancebenefit ofPacifier’s major components, i.e., the basic version,adding source rate limiting, and adding round-robin batching.Table I summarizes the different versions of MORE andPacifier evaluated.

1) Fixing MORE’s pruning threshold:Recall from Sec-tion II-A that MORE prunes FNs that are expected to performless than 10% of the total number of transmissions. We foundusing such a pruning threshold can result in disconnection of

5The number of control packets (ACKs) is the same for both MOREandPacifier, equal toN × B, whereN is the number of receivers andB is thenumber of batches the file is broken into.

8

(a) Number of FNs (b) Throughput

Fig. 2. Number of FNs and throughput with default pruning threshold(MORE orig) and the largest pruning threshold that does not cause anydisconnection (MOREnew), for 10 different scenarios. For MOREnew, thelabels above the bars show the pruning threshold used for each scenario.

some receivers. The probability for this to happen naturallyincreases with the network size, since the larger the numberof nodes acting as FNs, the smaller the expected number oftransmissions each of them has to make. Recall also that inMORE, the source proceeds to the next batch only when allreceivers acknowledge the current batch. When a receiver isdisconnected, the source will never leave the first batch, andall the receivers will receive zero throughput.

One solution to the problem is to use a much lower pruningthreshold than 0.1. However, using a very low threshold canlead to too many FNs in dense WMNs which increases thecontention for the channel. To be fair in our evaluation and notcause performance degradation for MORE, we used the fol-lowing approach, which favors MORE, instead of a commonthreshold for all 10 scenarios: for each scenario, we repeatedthe simulation for different values of the pruning threshold α,starting with the default value of 0.1, and lowering it by 0.01until no receiver was disconnected. This last value was the onewe used as the pruning threshold in the comparison againstPacifier.

Figure 2 shows the number of FNs, and the throughput, withthe default threshold of 0.1 (MOREorig), and with the bestthreshold for each scenario (MOREnew), in each of the 10scenarios. Figure 2(a) shows that using the default thresholdresulted in a very low number of FNs; on average only 11.2FNs were used in a network of 50 nodes. However, this lownumber of FNs caused disconnection of at least one receiverand resulted in zero throughput in 8 out of 10 scenarios, asshown in Figure 2(b). For the 10 scenarios studied, the largestpruning threshold that does not cause any disconnection variesfrom 0.1 to 0.03.

In the following, we compare various versions ofPacifierto MORE new. For simplicity, we will call it MORE.

2) Impact of tree-based OR:We start the evaluation ofPacifier by examining the impact of its tree-based OR, bycomparing the basic version ofPacifier (TREE), with MORE.The only difference between the protocols is the algorithmused for selecting FNs and assigning TXcredits to them. Theresults for 10 different scenarios are shown in Figure 3.

Figure 3(a) shows TREE achieves higher throughput thanMORE in 8 out of 10 scenarios. The gain ranges from20% (Scenario 7) up to 199% (Scenario 4), with an averagethroughput gain over all 10 scenarios equal to 42%. Only intwo scenarios (2 and 3), there is a small throughput reduction

(a) Throughput (b) Number of FNs

(c) Total # of Transmissions (d) Source Redundancy

Fig. 3. Throughput, number of FNs, total number of transmissions, andsource redundancy with MORE and TREE for 10 different scenarios.

with TREE, about 16%.The higher throughput achieved by TREE compared to

MORE can be explained by the fewer FNs and lower totalnumber of transmissions in the former compared to the latter.In particular, Figure 3(b) shows that the use of a tree instead ofa union of belts results in on average 36% fewer FNs in TREEthan in MORE. Note that in some cases, TREE uses equal oreven fewer FNs compared to MORE with the default pruningthreshold (e.g., in Scenarios 2, 4, 6). However, the FN selectionalgorithm ensures that no receiver is disconnected from thesource, unlike in MORE, since there is no random, threshold-based pruning. Figure 3(c) shows the use of a tree combinedwith the new algorithm for TXcredit calculation results in onaverage 44% reduction in the total number of transmissionsin TREE, compared to MORE. Finally, Figure 3(d) showsMORE has a high source redundancy; the source sends onaverage 17 times the file size. TREE reduces the averagesource redundancy to 12. The difference in source redundancysuggests TREE is more efficient in selecting FNs and moreaccurate in calculating the TXcredit values for the FNs.

3) Impact of source rate limiting:We next evaluate theimpact of backpressure-based rate limiting at the source, asimplemented in the TREE+RL version ofPacifier. Figure 4(a)shows that the use of rate limiting at the source improves thethroughput by 5% (Scenario 6) to 94% (Scenario 1), with anaverage of 20%, compared to TREE. Figure 4(c) shows thatTREE+RL on averages reduces the source redundancy to 5.84,a 52% reduction compared to the value of 12.15 for TREE.The reduction in the source redundancy in turn reduces thetotal number of transmissions by 28% on average, as shown inFigure 4(b). We found that this reduction comes not only fromthe contribution of the source but also from the majority of theFNs. This confirms that, by pacing the source’s transmissions,the source’s children and grandchildren get better chancestosuccessfully transmit packets and make progress down the tree.

4) Solving the “crying baby” problem: The above re-sults have shown that TREE and TREE+RL already offer

9

(a) Throughput (b) Total # of Transmissions

(c) Source Redundancy

Fig. 4. Throughput, total number of transmissions, and source redundancywith MORE and TREE+RL for 10 different scenarios.

Fig. 5. Throughput with TREE+RLand TREE+RL+RRB (Pacifier) for10 different scenarios. The error barsshow throughput of the best and theworst receiver.

Fig. 6. Average throughput withPacifier as a function ofknob, over10 scenarios. The error bars showaverage max and min values over the10 scenarios.

significant throughput improvement over MORE. However,these two versions ofPacifier still suffer from the “cryingbaby” problem. We next evaluate the effectiveness of round-robin batching on solving the “crying baby” problem, bycomparing TREE+RL+RRB (the completePacifier protocol)with TREE+RL.

Figure 5 shows the average throughput achieved withTREE+RL+RRB and TREE+RL in each of the 10 scenar-ios, as well as the throughput of the best and the worstreceiver (top and bottom of error bars) in each scenariounder TREE+RL+RRB. We make three observations. First,with TREE+RL, which uses sequential batch transmission,all 9 receivers in each scenario achieve the same throughput,which is determined by the worst receiver. In contrast, withTREE+RL+RRB, well-connected receivers get much higherthroughput than the average, as shown by the large gapbetween the top of the error bars and the average in most sce-narios. Averaging over 10 scenarios, the best receiver achieves58% higher throughput than the average throughput by allreceivers. Second, allowing receivers to proceed independentlyin TREE+RL+RRB also increases the average throughput by47% on average over all 10 scenarios, compared to TREE+RL.Third, importantly, the throughput improvement for the bestreceivers comes at almost no penalty to the worst receivers.In particular, compared to with TREE+RL, the throughput of

the worst receiver with TREE+RL+RRB gets slightly worsein 3 scenarios (Scenario 7, 8, and 9 by 10%, 7%, and 3%,respectively), remains unaffected in 2 scenarios (Scenarios2 and 3), and increases by 26%-146% for the remaining 5scenarios.

5) Tuning the knob in TXcredit Calculation: Finally, westudy the intricacies in calculating TXcredit values by varyingtheknob value introduced in Section III-C2. We vary the valueof knob from 0 (the version evaluated in Section IV-B4) to2. Intuitively, asknob increases, the throughput of the bestreceiver is expected to decrease and the throughput of theworst receiver is expected to increase, since we spend moretime on each batch in every round.

Figure 6 shows the average, max, and min throughput withPacifier, as knob varies from 0 to 2. Every point is theaverage over 10 scenarios. Somewhat surprisingly, higherknob

values improve the max throughput andknob = 1 appears tomaximize the average and the min throughput.knob = 0,which is expected to optimize the performance of the bestreceiver, achieves on average the lowest max, average, andmin throughput, compared to all the otherknob values. Thisconfirms our speculation in Section III-C that settingknob = 0may not give the best result as the TXcredit calculation isfundamentally based on the expected opportunistic receptions,and a lower than expected number of receptions in any givenbatch can cause the best receiver to be a few packets short ofdecoding a batch and wait for a whole round.

An additional counter-intuitive observation from Figure 6is that the throughput does not change monotonically asthe knob increases. The reason for this behavior is that thethe TX credits assigned to FNs actually interfere in a verycomplex way. In a nutshell, increasing the TXcredit of anFN j can potentially decrease the TXcredit of its childnodes, as the grandchild nodes ofj now have more chanceof overhearingj’s transmissions. Consequently, the chanceof packet reception atj’s grand-grandchild nodes from theirupstream nodes is affected in complicated ways.6

This complex interdependence of throughput,knob values,and TX credits is shown in Figure 7. In this figure, weplot the individual throughput for each of the 9 receivers,with 3 different knob values (0. 1, and 1.4) in 3 differentscenarios (Figures 7(a), 7(c), 7(e)), and initial TXcredits ofthe FNs in the same scenarios, for the sameknob values(Figures 7(b), 7(d), 7(f)). We picked up Scenario 3, whichhad the highest max throughput for most of theknob values,Scenario 10, which had the lowest min throughput for allknob

values, and Scenario 8.In Figure 7(a), we observe that increasing theknob value

improves throughput for the three best receivers (9, 4, and 8).However, for the remaining 6 receivers, throughput increaseswhenknob changes from 0 to 1 but it reduces again when itchanges form 1 to 1.4. Finally, the average throughput is samewith knob 1 and 1.4, slightly larger than withknob 0.

In Figure 7(c), we observe that the best receiver changeswhen knob changes from 0 to 1. Withknob 0, the best

6Recall our TX credit calculation is a polynomial heuristic; optimalTX credit assignment to all FNs is an NP-hard problem.

10

(a) Receiver Throughput - Sce-nario 3.

(b) FN TX credits - Scenario 3.

(c) Receiver Throughput - Sce-nario 8.

(d) FN TX credits - Scenario 8.

(e) Receiver Throughput - Sce-nario 10.

(f) FN TX credits - Scenario10.

Fig. 7. Individual receiver throughput and FN TXcredits with 3 differentknob values in 3 scenarios. FNs are sorted in increasing ETX distance fromthe source.

receiver is receiver 9, closely followed by receiver 8 (theirthroughput differ only by 3%). But changing theknob to 1results in a large throughput improvement for receiver 8 (by46%), which now becomes the best receiver, while throughputof receiver 9 only increases by 8%. Also, when moving toknob 1.4, throughput of the best receiver increases even more,but the average throughout decreases compared to withknob

1 (although it remains higher than withknob 0).Finally, in Figure 7(e) throughput of the three best receivers

(7, 3, and 4) and of the worst receiver (6) increases as theknob

increases. However, for the remaining 5 receivers throughputonly increases whenknob changes from 0 to 1 but it decreasewhen it changes from 1 to 1.4, and the average throughputalso exhibits the same behavior.

For each of the 3 scenarios, we observe that the TXcreditsof the FNs follow very different patterns as shown in Fig-ures 7(b), 7(d), 7(f). In Scenario 3, TXcredits are in generalhigher withknob 0. In Scenario 8, there is no specific pattern– we can observe the TXcredit of the 4th FN going to 0,for largeknob values. Finally, in Scenario 10, a large fractionof the FNs maintains the same TXcredits for differentknob

values.In summary, the discussion above shows that there is no

optimal value forknob. We find settingknob = 1 in Pacifierappears to improve the max throughput while maximizing theaverage and the min throughput.

(a) Average, max, and minthroughput with each protocolfor each of the 10 scenarios.

(b) CDF of the 90 throughputmeasurements obtained witheach protocol for 10 scenarioswith 9 receivers each.

Fig. 8. Overall throughput comparison of MORE, TREE, TREE+RL, andTREE+RL+RRB (Pacifier).

6) Overall Comparison:Figure 8(a) summarizes the aver-age, maximum and minimum throughput comparison betweenMORE, TREE, TREE+RL, and TREE+RL+RRB (Pacifier),where TREE+RL+RRB used aknob value of 1. We observethat on average,Pacifier outperforms TREE+RL, TREE, andMORE by 60%, 90%, and 171%, respectively. In addition,Pacifier allows well-connected receivers to achieve muchhigher throughput, which can be up to 20x higher than withMORE (for scenario 1), and also improves throughput of theworst receiver in all 10 scenarios, compared to the other 3protocols.

Figure 8(b) depicts the same results in a different way. Itplots the CDF of the 90 throughput values obtained from10 scenarios with 9 receivers each, for the four protocols.In this figure, the CDFs for MORE, TREE, and TREE+RLhave a staircase form, since for each scenario, all 9 receiversget roughly the same throughput (equal to that of the worstreceiver) due to the “crying baby” problem. In contrast, withPacifier, receivers finish independently of each other and theCDF has a continuous form. In the median case,Pacifieroutperforms TREE+RL, TREE, and MORE by 20%, 49%,and 178%, respectively.

The benefit ofPacifierbecomes more prominent if we lookat the two ends of the CDF.Pacifiersolves the “crying baby”problem by allowing good receivers to achieve very highthroughput. The 90th percentile is 223Kbps forPacifier, 70%,higher than with TREE+RL, 77% higher than with TREE,and 159% higher than with MORE. If we look at the 10thpercentile, i.e., the worst receivers, we observe thatPacifieroutperforms TREE+RL, TREE, and MORE by 80%, 300%,and 450%, respectively. This shows again thatPacifiernot onlysolves the “crying baby” problem, it also simultaneously offersa significant improvement to the performance of the “cryingbaby” itself.

V. PROTOCOL IMPLEMENTATION AND TESTBED

EVALUATION

In this section, we describe an implementation ofPacifierona WMN testbed and present experimental results comparingPacifier and MORE.

A. Testbed description

Our testbed, Mesh@Purdue (MAP) [44], currently consistsof 22 mesh routers (small form factor desktops) deployed on

11

Fig. 9. A schematic of MAP.

two floors of two academic buildings on the Purdue Universitycampus. A schematic of the testbed is shown in Figure 9.Each router has two radios. For this study, we used oneof them: the Atheros 5212 based 802.11a/b/g wireless radiooperating in b ad hoc mode. Each radio is attached to a 2dBirubber duck omnidirectional antenna with a low loss pigtailto provide flexibility in antenna placement. Each mesh routerruns Mandrake Linux 10.1 (kernel 2.6.8-12) and the open-sourcemadwifidriver [45] is used to enable the wireless cards.IP addresses are statically assigned. The testbed deploymentenvironment is not wireless-friendly, having floor-to-ceilingoffice walls, as well as laboratories with structures that limitthe propagation of wireless signals and create multipath fading.

B. Implementation details

NC-based wireless protocols (e.g., [6], [7]) are typicallyimplemented as a shim between the IP and the MAC layer,i.e., at layer 2.5. Here, for ease of debugging, deployment,and evaluation, we implementedPacifier at the applicationlayer, using broadcast sockets, on Mandrake Linux 10.1 (ker-nel 2.6.8-12). For a fair comparison, we also implementedMORE at the application layer, following all the detailsin [6].7 Although such an implementation unavoidably resultsin some performance degradation, compared to an implemen-tation closer to the MAC layer, from crossing the kernel-userboundary, we note that this degradation is expected to besimilar for both protocols, since they use the same type ofnetwork coding, they have the same memory requirements atthe routers, and the same header fields.

Our implementation handles only synthetic traffic, i.e. datapackets are generated within the MORE orPacifier applica-tion, similarly as the implementation in [46], in which packetsare generated within Click. The layer-2.5 header of MORE orPacifier is part of the application layer packet payload. Thesource initially generatesk random payloads for the currentbatch and mixes them every time it wants to transmit a packet.It then appends the MORE orPacifierheader and delivers theresulting packet to the IP layer, which in turn delivers thepacket to the MAC for transmission. Packets are broadcastat the MAC layer, and every neighbor node can hear them.When a node receives a packet, it extracts and processes theprotocol-specific header from the payload; if the node is an FN

7The publicly available implementation of MORE [46] using the Clickmodular router from the authors of [6] currently supports only unicast.

(i.e., it finds its ID8 in the FN list in the header), it also usesthe coding coefficients (also included in the header) to checkfor linear independence. If the received packet is innovative,the rest of the payload is stored for future mixing (if the nodeis an FN) or for decoding (if the node is a multicast receiver).

1) Dealing with queue sizes:In an ideal implementation atlayer 2.5, a node running either MORE orPacifier transmitsa packet when (1)the 802.11 MAC allowsand (2)the creditcounter is positive. A layer-2.5 implementation [6] does notqueue packets in the wireless card. Instead, innovative packetsfor the current batch are stored at a buffer. A pre-codedpacket is always available awaiting for transmission. If anotherinnovative packet is received before the pre-coded packet istransmitted, the pre-coded packet is updated by multiplyingthe newly received packet with a random number and addingit to the pre-coded packet. This approach ensures that everytransmitted packet includes information from all the receivedinnovative packets, including the most recent ones.

In our application layer implementation, we cannot get anyfeedback from the MAC, and hence, we have no controlover the time a packet is transmitted. Instead, the applicationdelivers packets to the IP when only the second conditionholds and there is enough space in the socket buffer; from theIP layer, the packets are delivered to the wireless driver storedat the card’s queue for transmission at a later time.

Since we have no control over a packet, once it leaves theapplication layer, we cannot update the packets buffered atthe socket buffer or awaiting for transmission at the card’squeue, if a new innovative packet is received. This inefficiencycan have a significant impact on the performance of the twoprotocols. If a packet is queued either at the IP or at the MAClayer for a long time, it may not contain information from allthe received packets so far. Even worse, the downstream nodesmay have already received enough packets from the currentbatch, in which case the enqueued packets should not betransmitted at all. This is true in particular at the source whichmay create packets at a rate faster than the (actual) MAC’stransmission rate. To avoid this problem with application-levelimplementation, we limit the socket buffer size to one packetand the card’s queue length to three packets, so as to limit thetime from the moment a packet is created at the applicationlayer till the moment the packet is actually transmitted.

2) Dealing with end-to-end ACKs:In both protocols, amulticast receiver sends an end-to-end ACK back to thesource every time it decodes a batch. It is critical for theperformance of the protocols that these ACKs are propagatedto the source in a fast and reliable way. In particular in MORE,loss of an ACK breaks the operation of the protocol, sincethe source only moves to the next batch when all receiversacknowledge the current batch. Similarly, delayed ACKs causethroughput degradation, since the source again cannot quicklymove to the next batch. InPacifier, the first problem doesnot exist, since even if no ACK is received for batchi, thesource will eventually move to the next batch when theCsi

counter reaches zero (Section III-C). However, delayed or lost

8To reduce the header overhead, we used 1-byte IDs instead of 4-byte IPaddresses.

12

ACKs can again significantly affect performance if the sourceunnecessarily spends time on batches that have already beendecoded by all the receivers.

ACK reliability. To provide reliability, the ACKs in MOREareunicastat the MAC layer. In contrast to 802.11 broadcastmode, 802.11 unicast mode provides a reliability mechanismthrough acknowledgments and retransmissions. Unfortunately,there is an upper limit to the number of times a packet canbe retransmitted at the MAC layer. For our Atheros wirelesscards, this limit is 11. In our experiments, we found that 11retransmissions were not always enough to deliver the packetto the next hop (especially under heavy traffic). Since thisparticular card does not allow changing this limit throughiwconfig, we had to implement a simple but efficient reliabilityscheme at the application layer.

In our scheme, every node maintains an ACK cache, whereit caches every ACK it transmitted, along with some meta data(the next hop of the path towards the source, the multicastgroup, the batch acknowledged by the ACK, and its status– “ACKed” or “not ACKed”). Nodes also remember the lastACK they forwarded for each multicast group. Every time anode transmits a data packet, it piggybacks information aboutthe last ACK it received. This serves as an acknowledgment forthe ACK to the ACK’s previous hop. When the previous hopoverhears a data packet acknowledging the ACK, it marks it as“ACKed” in the ACK cache. A node retransmits an ACK when(i) it overhearsM packets from the ACK’s next hop that do notacknowledge the ACK, or (ii) it overhearsN packets from anynode other than the ACK’s next hop. We experimented withdifferent values ofM and N and finally selectedM = 10,N = 20.

Fast ACK propagation. Similar to in [6], ACKs are sentto the source over the shortest ETX path to ensure quickpropagation. In addition, in [6], ACKs are prioritized overdatatransmissions. In addition to ensuring fast ACK propagation,prioritizing ACKs over data packets is critical in our appli-cation layer implementation for one more reason. Since wehave no control over a packet once it leaves the applicationlayer, we have to guarantee that an ACK packet will never bedropped if the card’s queue is full of data packets.

To implement ACK priority over data packets in our ap-plication layer implementation, we leveraged the TOS bits(“TOS filed”) of the IP header, which can be set usingsetsockoptat the application layer, and the priority propertiesin Linux routing [47]. The basic queuing discipline in Linux,pfifo fast, is a three-band first-in, first-out queue. Each bandis txqueuelenpackets long, as configured withifconfig. In ourimplementation, we settxqueuelen = 5, as mentioned in V-B1.Packets are enqueued in the three bands based on their TOSbits. The three bands, 0, 1, 2, have different priorities, withband 0 having the highest priority and band 2 having the lowestpriority. Packets from a given band are dequeued only when allhigher priority bands are empty. By default, the TOS bits areset to0000 and packets are enqueued in band 1. For ACKs, weset them to1010. This combination corresponds to “minimumdelay + maximum reliability” (or “mr+md”) and enqueues theACKs in the highest priority band 0.

Another factor that caused significant delay to the ACKpackets and resulted in very low throughput was the ARPmessages. Since ACKs are unicast at the MAC layer, thesender of an ACK first sends an ARP request before theactual transmission of the ACK packet, in order to learn theMAC address that corresponds to the IP address of the nexthop. If no reply is received, the ARP request is retransmittedafter a timeout (the default is 1 sec). Unfortunately, both theARP requests and the ARP replies arebroadcastat the MAClayer. Since 802.11 broadcast implements no reliability mech-anism for broadcast frames, ARP messages are susceptibleto loss due to poor channel conditions or collisions (underheavy traffic). Indeed, we observed in our experiments thatsometimes ARP requests were retransmitted up to 90 times,which resulted in a 1.5 min delay, before the actual ACK wassent. To deal with this problem, before each experiment, wecached permanently at each node on the shortest ETX pathfrom a receiver to the source, the IP-MAC mapping of thenext hop, using theip command, thus completely eliminatingthe exchange of ARP messages during the experiment.

In addition to the two protocols, we also implemented anETX measurement module, same as the one we used in oursimulations (described in Section IV-A). The source code forthe two protocols and the ETX module together is over 9000lines of C code.

C. Experimental setup

In the implementation of the two protocols we used thesame parameters as in our simulation study in Section IV,i.e., the batch size wask = 32 packets, the random codingcoefficients were chosen from a Galois Field of size28, andthe knob value for Pacifier was again set to1. In all theexperiments, the bitrate of the wireless cards was set to 2Mbpsand the transmission power to 16dBm. We disabled RTS/CTSfor unicast frames as most operational networks do. With thesesettings, the length of the shortest ETX paths between differentnodes is 1-6 hops in length, and the loss rates of the links varyfrom 0% to 88%, with an average value of 29%.

We ran each protocol on 10 different scenarios (i.e., selec-tion of source and multicast group members). In each scenario,1 source and 4 receivers were randomly selected among the22 nodes of our testbed. In each scenario, we first ran the ETXmodule for 10 minutes to collect the pairwise loss rates andETX metric for each link of our testbed, and then we ran thetwo protocols, MORE andPacifier, in sequence. With bothprotocols, the source sent a 2.3MB file consisting of 1460-byte packets. Since the quality of some links of our testbedvaries substantially from day to day in a week, we repeatedthe experiments for the same 10 scenarios on 4 different days(one weekend and two weekdays), and we present separatelythe results for each day.

D. Experimental results

Figures 10(a), 10(b), 10(c), 10(d) show the average through-put achieved with MORE andPacifier in each of the 10scenarios, as well as the throughput of the best and the worstreceiver (top and bottom of error bars) in each scenario, on 4different days.

13

(a) Day 1 (b) Day 2

(c) Day 3 (d) Day 4

Fig. 10. Testbed throughput comparison of MORE andPacifier in 10 differentscenarios and 4 different days.

Similar to the simulation results, we observe thatPacifieroutperforms MORE in 9 out of 10 scenarios on all 4 days.The average throughput improvement over all 10 scenariosranges between 83% (for Day 4) and 144% (for Day 1). Thisis somewhat lower than the corresponding simulation result(171% in Figure 8(a)). The reason is that the size of ourtestbed is much smaller than the simulated networks, and pathdiversity is not as large as in the simulations.

We also observe again thatPacifiersolves the “crying baby”problem, allowing well-connected receivers in each case toachieve throughputs much higher than the average value, whilealso improving throughput of the worst receivers in almostall scenarios. Averaging over 10 scenarios for each of the4 days, the throughput of the best receiver withPacifier is244%, 302%, 239%, and 259% higher than with MORE, but,in some cases, it can be much higher, e.g., more than 8x inScenario 7, Days 2 and 3, and Scenario 8, Day4, more than10.5x in Scenario 6, Day 4, and more than 14.4x in Scenario7, Day1. Similarly, the average (over 10 scenarios) throughputof the worst receiver withPacifier is on 83%, 101%, 74%, and53% higher than with MORE on each of the 4 days, and themaximum improvement can be as high as 5.4x (higher thanin the simulation results) in Scenario 7, Day 1.

Figures 11(a)-11(d) plot the CDF of the 40 throughputvalues obtained from the 10 scenarios with 4 receivers each,for the two protocols, on each of the 4 days. Similar to Fig-ure 8(b) for the simulation results, we observe that the CDFsfor MORE exhibit a staircase form, since for each scenario,all 4 receivers get roughly the same throughput (equal to thatof the worst receiver) due to the “crying baby” problem. Incontrast, withPacifier, receivers finish independently of eachother and the CDF always has a continuous form.Pacifieroutperforms MORE on all 4 days both in the median case,by 158 - 286%, and in the two ends of the CDFs – the 90thpercentile is 85-128% higher and the 10th percentile is 128-294% higher withPacifierthan with MORE. This again shows

(a) Day 1 (b) Day 2

(c) Day 3 (d) Day 4

Fig. 11. CDFs of 40 testbed throughput measurements obtained with MOREandPacifier for 10 scenarios with 4 receivers each on 4 different days.

that Pacifier not only solves the “crying baby” problem, italso simultaneously offers a significant improvement to theperformance of the “crying baby” itself.

VI. CONCLUSION

High-throughput, reliable multicast routing has many im-portant applications in WMNs, such as software updatesand video/audio file downloads. However, designing high-throughput, reliable multicast protocols faces two challenges:the inherent lossiness of wireless links and the “crying baby”problem. In this paper, we presentedPacifier, the first practi-cal NC-based high-throughput, reliable multicast protocol forWMNs. Pacifier seamlessly integrates tree-based OR, intra-flow NC, source rate limiting, and round-robin batching, toachieve high throughput and solve the “crying baby” problem.

Our performance evaluation ofPacifier via extensive simu-lations and an implementation on a WMN testbed have shownPacifier significantly outperforms the state-of-the-art MOREfor various multicast scenarios. In particular, the experimentalresults on our 22-node WMN testbed show thatPacifierincreases the average multicast throughput over MORE by 83-114%, while the maximum throughput gain for well-connectedreceivers is as high as 14x, and the maximum throughput gainfor the “crying baby” itself is as high as 5.4x, compared toMORE.

Since the cumulative path loss rate in wireless multihopnetworks increases with the path length, multicast receiverheterogeneity is unavoidable in WMNs. In fact, the degree ofheterogeneity is expected to increase as future WMNs scalein size. Our experience with designingPacifier shows theimportance of exploiting heterogeneity, rather than ignoring it.By treating heterogeneous receivers equally, MORE penalizeswell-connected receivers, forcing them to achieve the samethroughput as the worst receiver. In contrast, by exploitingheterogeneity, and prioritizing well-connected receivers overthe “crying babies”,Pacifier manages to achieve several-foldthroughput improvement for well-connected receivers, without

14

penalizing the poorly-connected ones; on the contrary, it oftendrastically improves throughput of the worst receivers.

REFERENCES

[1] “MIT Roofnet,” http://www.pdos.lcs.mit.edu/roofnet.[2] “Seattle wireless,” http://www.seattlewireless.net.[3] “Technology For All (TFA),” http://tfa.rice.edu.[4] S. Biswas and R. Morris, “ExOR: Opportunistic multi-hoprouting for

wireless networks,” inProc of ACM SIGCOMM, 2005.[5] S. Katti, H. Rahul, W. Hu, D. Katabi, M. Medard, and J. Crowcroft,

“XORs in the air: Practical wireless network coding,” inProc. of ACMSIGCOMM, August 2006.

[6] S. Chachulski, M. Jennings, S. Katti, and D. Katabi, “Trading structurefor randomness in wireless opportunistic routing,” inACM SIGCOMM,2007.

[7] S. Katti, S. Gollakota, and D. Katabi, “Embracing wireless interference:Analog network coding,” inProc. of ACM SIGCOMM, 2007.

[8] R. Chandra, V. Ramasubramaniam, and K. Birman, “Anonymous gossip:Improving multicast reliability in mobile ad hoc networks,” in Proc. ofICDCS, 2001.

[9] J. Luo, P. Eugster, and J.-P. Hubaux, “Route driven gossip: Probabilisticreliable multicast in ad hoc networks,” inProc. of IEEE Infocom, 2003.

[10] B. Scheuermann, M. Transier, C. L. M. Mauve, and W. Effelsberg,“Backpressure multicast congestion control in mobile ad-hoc networks.”in Proc. of CoNEXT, 2007.

[11] J. Park, M. Gerla, D. S. Lun, Y. Yi, and M. Medard, “Codecast:a network-coding-based ad hoc multicast protocol,”IEEE WirelessCommunications, vol. 13, no. 5, 2006.

[12] K. Tang, K. Obraczka, S.-J. Lee, and M. Gerla, “A reliable, congestion-controlled multicast transport protocol in multimedia multi-hop net-works,” in Proc. of WPMC, 2004.

[13] E. Pagani and G. Rossi, “Reliable broadcast in mobile multihop packetnetworks,” inProc. of MobiCom, 1997.

[14] A. Sobeih, H. Baraka, and A. Fahmy, “ReMHoc: A reliable multicastprotocol for wireless mobile multihop ad hoc networks,” inIEEEConsumer Communications and Networking Conference (CCNC), 2004.

[15] V. Rajendran, Y. Yi, K. Obraczka, S.-J. Lee, K. Tang, andM. Gerla,“Combining source- and localized recovery to achieve reliable multicadtin multi-hop ad hoc networks,” inProc. of Networking, 2004.

[16] D. Koutsonikolas and Y. C. Hu, “The case for FEC-based reliablemulticast in wireless mesh networks,” inProc. of DSN, 2007.

[17] L. Rizzo and L. Visicano, “RMDP: an FEC-based reliable multicastprotocol for wireless environments,”Mobile Computing and Communi-cations Review, vol. 2, no. 2, 1998.

[18] D. Lun, M. Medard, and R. Koetter, “Efficient operation of wirelesspacket networks using network coding,” inProc. of IWCT, 2005.

[19] M. Ghaderi, D. Towsley, and J. Kurose, “Reliability Gain of NetworkCoding in Lossy Wireless Networks ,” inProc. of IEEE INFOCOM,2008.

[20] H. W. Holbrook, S. K. Singhal, and D. R. Cheriton, “Log-based receiver-reliable multicast for distributed interactive simulation,” in Proc. of ACMSIGCOMM, 1995.

[21] D. Aguayo, J. Bicket, S. Biswas, G. Judd, and R. Morris, “Link-level measurements from an 802.11b mesh network,” inProc. of ACMSIGCOMM, August 2004.

[22] S. Floyd, V. Jacobson, C.-G. Liu, S. McCanne, and L. Zhang, “A reliableframework for light-weight sessions and application levelframing,”IEEE/ACM Transactions on Networking, 1997.

[23] L. Rizzo, “Effective erasure codes for reliable computer communicationprotocols,” ACM Comp. Comm. Review, vol. 27, no. 2, 1997.

[24] J. Nonnenmacher, E. Biersack, and D. Towsley, “Parity-based lossrecovery for reliable multicast transmission,” inACM SIGCOMM, 1997.

[25] E. Schooler and J. Gemmel, “Using multicast FEC to solvethe midnightmadness problem,” Technical Report, MSR-TR-97-25, Tech. Rep., 1997.

[26] M. Luby, “LT codes,” inProc. of 43rd FoCS, 2002.[27] P. Maymounkov and D. Mazieres, “Rateless codes and big downloads,”

in Proc. of IPTPS, 2003.[28] A. Shokrollahi, “Raptor codes,” inProc. of IEEE International Sympo-

sium on Information Theory (ISIT), 2004.[29] A. W. Eckford and W. Yu, “Rateless slepian-wolf codes,”in Proc. of

39th Asilomar Conference on Signals, Systems and Computers, 2005.[30] E. Vollset and P. Ezhilchelvan, “A survey of reliable broadcast protocols

for mobile ad-hoc networks,” University of Newcastle upon Tyne, Tech.Rep. CS-TR-792, 2003.

[31] S. Gupta and P.Srimani, “An adaptive protocol for reliable multicast inmobile multi-hop radio networks,” inProc. of IEEE Workshop on MobileComputing Systems and Applications, 1999.

[32] T. Gopalsamy, M. Singhal, and P. Sadayappan, “A reliable multicastalgorithm for mobile ad hoc networks,” inProc. of ICDCS, 2002.

[33] K. Tang, K. Obraczka, S.-J. Lee, and M. Gerla, “Reliableadaptivelightweight multicast protocol,” inProc. of ICC, 2004.

[34] D. S. J. D. Couto, D. Aguayo, J. C. Bicket, and R. Morris, “A high-throughput path metric for multi-hop wireless routing,” inProc. of ACMMobiCom, 2003.

[35] E. Rozner, J. Seshadri, Y. Mehta, and L. Qiu, “Simple opportunisticrouting protocol for wireless mesh networks,” inProc. of WiMesh, 2006.

[36] S.-J. Lee, M. Gerla, and C.-C. Chiang, “On-Demand Multicast RoutingProtocol,” in Proc. of IEEE WCNC, September 1999.

[37] C. Gkantsidis, W. Hu, P. Key, B. Radunovic, S. Gheorghiu, and P. Ro-driguez, “Multipath code casting for wireless mesh networks,” in Proc.of ACM CoNEXT, 2007.

[38] C. Reis, R. Mahajan, M. Rodrig, D. Wetherall, and J. Zahorjan,“Measurement-based models of delivery and interference instatic wire-less networks,” inProc. of ACM SIGCOMM, 2006.

[39] X. Zhang and B. Li, “Optimized multipath network codingin lossywireless networks,” inProc. of IEEE ICDCS, 2008.

[40] D. Koutsonikolas, Y. C. Hu, and K. Papagiannaki, “How toevaluateexotic wireless routing protocols?” inProc. of ACM HotNets-VII, 2008.

[41] B. Scheuermann, C. Lochert, and M. Mauve, “Implicit hop-by-hopcongestion control in wireless multihop networks,”Elsevier Ad HocNetworks, vol. 6, no. 2, pp. 260–286, Apr. 2008.

[42] Y. Lin, B. Li, and B. Liang, “CodeOR: Opportunistic routing in wirelessmesh networks with segmented network coding,” inProc. of IEEE ICNP,2008.

[43] X. Zeng, R. Bagrodia, and M. Gerla, “Glomosim: A libraryfor parallelsimulation of large-scale wireless networks,” inProc. of PADS Work-shop, May 1998.

[44] “http://www.engineering.purdue.edu/mesh.”[45] madwifi, “http://madwifi.org.”[46] “More source code,” http://people.csail.mit.edu/szym/more/README.html.[47] Linux Advanced Routing and Traffic Control, “http://lartc.org//lartc.

html/.”Dimitrios Koutsonikolas (S’04) received the un-dergraduate degree in electrical and computer en-gineering from the National Technical University ofAthens (NTUA), Greece, in 2004.

He is currently a Ph.D. student of electrical andcomputer engineering at Purdue University, WestLafayette, IN. He has interned at Thomson ParisResearch Lab. His research interests include wirelessmesh, ad hoc, sensor, and cellular networks.

Mr Koutsonikolas received the Ross Fellowshipfrom Purdue University in 2004, and the Tellabs

Fellowship from the Center for Wireless Systems and Applications (CWSA),Purdue University, in 2006. He is a student member of ACM and USENIX.

Y. Charlie Hu is an Associate Professor of Elec-trical and Computer Engineering and Computer Sci-ence (by courtesy) at Purdue University. He receivedhis Ph.D. degree in Computer Science from HarvardUniversity in 1997. From 1997 to 2001, he was aresearch scientist at Rice University. His researchinterests include operating systems, distributed sys-tems, Internet measurement and routing analysis,and wireless networking. He has published over100 papers in these areas. Dr. Hu received theNSF CAREER Award in 2003. He is a member of

USENIX and a senior member of ACM and IEEE.

Chih-Chun Wang joined Purdue School of Elec-trical and Computer Engineering in 2006 as an As-sistant Professor. He received the B.E. degree fromNational Taiwan University in 1999, and the M.S.and Ph.D. degrees in E.E. from Princeton Universityin 2002 and 2005, respectively. His current researchinterests are in the graph-theoretic and algorithmicanalysis of iterative decoding and of network coding.His other research interests fall in the general areasof optimal control, information theory, detectiontheory, coding theory, iterative decoding algorithms,

and network coding. He received the NSF CAREER Award in 2009.