COOPERATIVE MULTI-AGENT SYSTEMSIN MOBILE AD HOC NETWORKS

Joseph P. MackerWilliam Chao

Myriam AbramsonIan Downard

Information Technology DivisionNaval Research Laboratory

ABSTRACTTwo important enabling but evolving technologiessupporting future DoD network-centric systems atthe tactical edge are, mobile ad hoc networking(MANET) and Multi-Agent Systems (MAS).Despite their value in enabling more autonomousnetwork system operation scenarios, openresearch and engineering questions remainregarding robust interoperation, standardization,and design ofthese two technologies. We describerecent research and development that is helping tobetter understand crosslayer design issues withinboth MAS and MANET. We describe the problemarea and the open software components developedto support our research. We summarize recentmodeling and simulation advances in a mixed MASand MANET scenario environment. MANETmulticast approaches for interagentcommunications are discussed and described.Some early analysis of MAS performance ispresented using a variety of interagent MANETcommunication models. The behavior of MASautonomous cooperative teamwork and roleallocation within disruptive and dynamic MANETenvironments is examined. We conclude byoutlining open issues and areas offurther work.BACKGROUND AND MOTIVATIONThere is planned deployment of Mobile Ad hocNetworking (MANET) type network routingtechnology at the battlespace forward edge or the"the first tactical mile". There is also earlydeployment of agent-based systems occurring nowwith more extensive future deploymentenvisioned. At present, there remains a limitedunderstanding of appropriate architectural designtradeoffs in adapting upper layer protocols andapplications in these environments. Also previousdesign work done with agent based networkingand software has often assumed benign network

behavior and highly stable infrastructures notMANET environments. Figure 1 provides a highlevel depiction of some of the commoncharacteristics differentiating mainstream Internet,

High Performance The Mainstream

Networks Internet

Some Common Characteristics Some Common Characteristics

*Stable infrastructure *Mixed range of assets

*Fiber optic/High-speed Mixed mediaRF/wireless optical *Tending to higher bandwidth*Highest bandwidth Or .s.*Low latency *Low to high latency-Connection-oriented links *Table-based routing

* Policy-basedQoS * Mixed policies in forwardingand QoS

Figure 1: Network Problem Area Focus

high performance network, and tactical edge typeenvironments. The characteristics depicted inFigure 1 emphasize why focused research shouldcontinue to be performed on tactical edge solutionsincluding a multilayered examination ofperformance issues and tradeoffs.

PROBLEM FOCUSOur technical focus is to improve design andperformance of both MANET and MAStechnology within dynamic network centricproblem scenarios.Some of the technical goals include the following:

- Investigate MAS design robustness instressed, mobile network environments

- Develop network protocol enhancementsand improve modeling to study behaviorwhere needed

- Improve design tradeoff understanding bystudying a combined solution (not wellexamined to date)

1

- Develop working models and software thatcan more easily transition

The core technical challenge of our work involvestackling cross-disciplinary issues of dynamicnetwork protocol and multi-agent system design.Separately, MAS and MANET encompass twochallenging research and engineering areas. Whenconsidered as a combined technical solution, thechallenges increase due to complex crosslayerdesign issues and behaviors [CDCP05]. Wesummarize these challenges as follows:

MANET Project-Related Challenges:* Rich taxonomy of design choices exist* Present lack of modeling and support for

interagent communication components* Crosslayer performance design issues* Possible agent and middleware self-

organization interactions given MANETbehaviors and dynamics

* Protocol scalability and robustnessMAS Project-Related Challenges:

* Rich taxonomy of design choices exist* Lack of modeling and analysis in stressed

network environments* Adaptation of well-known strategies in

disruptive MANET environments* Designs supporting efficient inter-agent

communicationsPREVIOUS WORKOngoing research and development is being donein both MANET [B04][CM99][POI] and MAS[W02] technology areas by numerous parties. Weare leveraging existing and previous workaccomplishments to execute our research goals.Our previous accomplishments in this area weredocumented in [MAMC05]. Here we update ourresearch approach and results and provide moredetailed evaluation of cooperative multi-agentscenarios using specific MANET protocoltechnology.MODELING APPROACH AND ISSUESFigure 2 illustrates the modeling systemrequirements and related issues that were requiredin developing a framework for planned researchactivities. A fundamental need was to develop

distributed MANET modeling capabilities in bothsimulation and emulation and this required thefollowing components:

* Means of producing and controllingnetwork dynamics and node mobility

* MANET protocol prototypes and thepotential for supporting middlewaresystem components

* Multi-agent software prototypes withflexible network interface components

* Environmental model interface to thedistributed agents

,

Multi-Agent System.- - Interagent communications_

-localized vs. global-unicast vs. multicast

- Effectiveness in MANET- Role allocation algorithms- External input (non network)

I"pt

I

Figure 2: System Modeling Requirements

RECENT MODELING PROGRESSFigure 3 depicts a system diagram of the modelingprogress accomplished in both simulation andemulation environments. The Agent Toolkit hasbeen recently developed to allow a more crossplatform and extensible approach to be pursued in

Agent Toolkit -

AgentJ/Protolib

Environment 1/0- State variable- Location- Mobility

Figure 3: Key Component Modeling Diagram

developing and enhancing java software agentsplaced within our test environments for

2

experimentation. AgentJ [ADMT06] and theProtolib software components shown in Figure 3allowed for standard java code to be ported andimplemented within the NS-2 simulationenvironment. This has enabled more flexibility andless abstraction in simulating application layercomponents. This will also likely ease theinclusion of additional application agents in futureefforts. As also shown in Figure 3, a distributedenvironmental simulation model and agentinterface was added to the NS-2 environment toprovide external dynamic agent stimulus andenvironmental modeling input. Thisenvironmental modeling channel and approach canbe used to represent external sensor systems andother dynamic environmental phenomena [D04].Recent modeling advancements go beyondprevious reported results allowing us to scale ourexperiments and include additional capabilitiessuch as MANET multicast routing and morecomplex stimulus.

RECENT MAS AND MANET STUDIESWe have constructed a number of dynamicproblem solving scenarios that exercise both multi-agent system interactions and the underlyingMANET communications on the move. Anexample of one of our primary agent team-basedproblem solving scenarios is a prey/predatorpursuit problem. The prey/predator problem is acanonical example of agent teamwork in theliterature. We adapted this problem to furtherstudy interagent behavior and teamworkperformance and have included MANET routingand communications support.A challenge in carrying out planned experiments isthe actual data collection and measurement ofsystem effectiveness. While we routinely collectand are interested in detailed protocol and networkstatistics, we are now also interested in measuresof effectiveness for MAS. Since we are studyingdynamic role allocation and agent teamwork wedeveloped initial approaches to measurecoordination quality. Given a scenario, one metricwe measure is the time or number of steps it takesto complete a task or set of tasks. This provides arough effectiveness measure of teamworkperformance given a particular system design. Wealso gather and examine statistics in a coordinationmetric that measures interactions between agentsperforming distributed role allocation (e.g.,

number of messages received and role collisionevents).In previous work [ACM05], we reported on theinitial examination of prey/predator agent systemswithin simulated networks in the Repast agent-based modeling tool [NCV06]. In recentexperiments, we have performed scaled studies (>60 agents) in NS-2 simulation to examine largeagent/team performance. This is done whilemaintaining highly detailed protocol andapplication modeling capturing complexbehavioral and performance interactions acrosslayers. Figure 4 shows a snapshot visualization ofa 30 predator agent and 30 prey agent simulationscenario recently executed. This simulationscenario involves 60 fully functional java agents,of which 30 predator agents operate with a highlydetailed MANET and IP network protocol stackmodel. This test also includes the operation ateach node of a MANET unicast and multicastforwarding protocol to enable a self-organizingnetwork even during periods of topologyfragmentation.Our predator/prey teamwork research approachalso involves examining differing interagentcommunication requirements and capabilities thatmatch various role allocation algorithms beingexamined. As reported in [ACM05] we used ananalytical model to examine a range of roleallocation strategies that require different typesand levels of interagent network messaging. Anearly set of results demonstrated that theDistributed Constraint Optimization (DCO)approach based upon the Hungarian method[K55] [PS98] outperformed other role allocationmethods in simulation. We are examining thissame method in a more detailed scaled networkscenario with network congestion and dynamictopology disruption as additional parameters.The basic prey/predator scenarios being used canbe described as follows. The scenario is initializedwith a number of predators and preys in randomgeographic locations. Predators sense preys usingan abstracted sensor model and can communicatewith other predators using MANET networkingsupported within the NS-2 framework. Thelimited sensing and wireless communication rangeof each predator provides both a limitedenvironment view and a limited directly connectedcommunication neighborhood. The scenario tasks

3

require at minimum N predators to surround andtherefore capture each prey. Although the multipleprey team and surrounding requirements establisha simple role allocation problem, the scenario ishighly dynamic and uses detailed application andnetworking models to carry out an experiment.Due to limited communication capabilities andranges, interagent networking is needed to providemultihop connectivity. The predator network mayalso undergo unpredictable fragmentation andmerging due to movement required to accomplishthe task. To support research, most protocol andenvironment conditions are parameterized andsome examples include the following:

* Sensor ranges and qualities for each preyand predator can be programmed. Specialnodes may be added to haveheterogeneous sensing capability.

* Differing speeds and capabilities can beestablished for each agent. Agents may beprogrammed to remain fixed if desired.

* Predator agents have limited wirelesscommunication range that isprogrammable. Other environment effectsand propagation models can be added tothe scenario.

* A variety of MANET networkingprotocols and transport methods can beused by the agents during experimentation.Fragmentation and coalescing ofconnected networks occurs duringscenarios and is supported.

* Predators must form teams of minsize = d,pursue and surround a prey to accomplisha successful capture. These parameterscan be adjusted, but the default team sizeis four.

* High level tasks (e.g., capture) can bedefined and the number of steps can bemeasured along with other metrics andnetwork protocol statistics.

* Different network protocols, roleallocation strategies, and external networkdisruption can be added to create differentscenarios.

Figure 4, shows a screenshot from an examplemulti-agent simulation. The links drawndemonstrate the dynamic topology that is possible

between the agent nodes. These dynamic links areshown between cooperative predator agentsdynamically forming ad hoc networks andperforming cooperative task allocation to captureprey (red denotes captured, green denotesuncaptured). Scout nodes are stationary, butcontribute to predator task allocation by providinglong-range prey detection.

time=l20 I

SCOIA15h kbultl 6\ yscOllt 7

g redi prey2 t

P3eyl pred4 Pr cIL

prSily2 preD'

pf-y6 PVey2 P'0,p:0 3

p _ predl

Figure 4: Multi-Agent Predator TeamworkSimulation

MANET MULTICAST FOR INTRA-AGENTCOMMUNICATIONSIn order to provide enhanced network routing andnetwork messaging mechanism for agentteamwork communications within MANETs, weare applying related research and prototypedevelopment of Simplified Multicast Forwarding(SMF). SMF research and emerging IETFspecifications are discussed in [M06] and[MDC04]. SMF provides a simple multicastforwarding capability for application data flowswithin a MANET routing area. SMF also providesa forwarding process compatible with differentneighborhood discovery protocols and optimizedrelay set selection algorithms. In MAS teamworkrole allocation strategies, group communicationand maintenance of either environmental or agentdecision state is often required to be shared. SMFprovides an efficient MANET networkingmechanism to quickly forward this state in adynamic environment. In one class of roleallocation strategy, the distributed stochasticalgorithm (DSA) [FMO1], role changes are madestochastically by the agents to reduce conflicts andcommunicated until a consistent solution isobtained. In a second class, as in the distributed

4

coordination optimization (DCO), the stateinformation, or beliefs, is communicated betweenagents to infer their respective role using a

constraint optimization method. We are examiningall these classes within our research. In all thesecases, while different information is being sharedamongst agents there is a common requirement forcommunications of either state or intent. Theapplication of SMF provides an efficientnetworking means of routing this group-basedinteragent communication within a highly dynamicnetwork environment. While SMF is efficient ifmany agents require the same message, futurework will study the system design tradeoffs as

communication overhead and congestionconditions increase within our network scenarios.

CASE STUDY: WIRELESS NRLEMULATOR EXPERIMENTSFactorial experiments were conducted in thewireless NRL emulator [CMW03] to study theimpact of various parameters on the coordinationperformance of multi-agent systems in MANET.Using the prey/predator scenario described aboveand a fixed network density (10 nodes), theperformance was measured in terms coordination[ACM05] and robustness (coefficient of variationw.r.t the median number of steps required toaccomplish the goal). This robustness metricencapsulates the trade-off between performanceand reliability. The communication range was

allowed to vary in the 100-300 meters range tobetter reflect heterogeneous environments. Otherparameters such as the speed of the agents werealso allowed to vary. Figures 5 and 6 showcomparative experimental results in differentcongestion environments for different roleallocation algorithms. These results show howperformance rapidly degrades as congestionincreases.

The algorithms analyzed in these figures invokedifferent random processes for solving theconstraint optimization problems in our

prey/predator scenarios. DCO 70% willcommunicate status messages used to resolveconflicts only 70% of the time. In the case ofDSA, a role change to resolve conflicts will occur

only 70% of the time, and if a role change occurs,it will be communicated to the other agents.

Coordination Performance in Wireless Emulator

C,,

0

0)

0

.00

0

C5

0.6 40Hungarian DCO

U

0.58 \\\\\ Hungarian DCO 70% SDSA70%0.56 1

0.54

0.52

0.48

0.460.8 33.36 66.72 10

Congestion (kbps)

Figure 5: Comparative coordinationperformance at different congestion levels

Robustness Performance in Wireless Emulator

Ca)

nI

L,

C)0

C/,C/,

C/,I

-Q

0

1.

O.

0.

0.

O.0.8 33.36 66.72

Congestion (kbps)100

Figure 6: Comparative robustness atdifferent congestion levels

CASE STUDY: INTERAGENT AD HOCNETWORKING VALUEUsing and applying the detailed modelingcomponents, agent role allocation designs, andMANET protocols previously mentioned, we

present Figure 7 to illustrate how communicationnetworking can improve agent task completion. Infigure 7 along the x axis we start out with a

slightly fragmented physical network topology.Under these conditions, we measure the time or

steps it takes for the agent group to complete theprey capturing task. In subsequent experiments,we increase the transmit power parameter or

communication range achievable by predatoragents, thus increasing the probability of forming a

physically connected network topology. From thecurve, we illustrate the variance of particular

5

Hungarian DCO -Hungarian DCO 70% *

12 DSA 70% A

8

6

4 t

2

0o

measurements and the fact that as networkingprobability increases the agent coordinationgenerally improves. As network density increaseswe have also shown from related SMF researchthat we can control the amount of network trafficoverhead growth for multicast data flows toreasonable levels.

IDecreased probability of ad hoc networkin:g]

,, Increasing networking density11,i I~~~~~~~~~~

tR5~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~60§0.0 apa)

Figure 7: Task Completion vs. NetworkLikelihood

While this is only one basic example, itdemonstrates that there are performance gainsfrom networking agents. It also indicates thatthese gains can be measured in detailed modelingenvironments. Environmental conditions, sensor

models, agent coordination strategies, andMANET protocol parameters can all be adjustedaccordingly to examine different design andscenario tradeoffs. Further detailed scenariosspecific studies are largely the scope of futureapplied work that is planned.

FUTURE WORKPart of the challenge of studying MAS systems isdeveloping approaches and actually measuring theeffectiveness of performance. This is even more

challenging within highly complex environmentswith competing goals. Using the outlinedsimulation and emulation test environmentsdiscussed here, our future work will furtherexplore a set of use case scenarios to exerciseMAS and MANET protocol performance.

CONCLUSIONWe have briefly described recent research anddevelopment that is examining crosslayer issueswhich affect performance of MAS techniquesoperating within MANET environments. Wesummarized our recent modeling and simulationadvances in achieving a mixed MAS and MANET

scenario environment that can be used for furthercomplex analysis. MANET multicast approachesfor interagent communications were also brieflydiscussed and results from using this techniqueshowed significant agent system improvements asnetworks became less fragmented and were able totake advantage of multicast. We presented furtheremulation and simulation analysis of MASautonomous cooperative teamwork and roleallocation within disruptive and dynamic MANETenvironments. While interagent communicationsrequire additional network overheard and areprone to disruption within a MANET scenario,early results from our work indicate thatappropriate crosslayer design approaches canimprove MAS performance. Further work hasbeen outlined to continue more complex scenariosand the range of design tradeoffs.

ACKNOWLEDGEMENTSWe would like to acknowledge the contributions ofother NRL scientists for their valuable input onwork related to this project. Ranjeev Mittu, JustinDean, Brian Adamson, and Jeff Weston all deserveacknowledgements for their contributions tovarious ideas and software components used incarrying out this research.

REFERENCES[CDCP05] Marco Conti, et al. Cross-LayerSupport for Group-Communication Applicationsin MANETs. IEEE ICPS REALMAN Workshop,July 14, 2005.[B04] Basagni, et al, Mobile Ad Hoc Networking,Chapter 9, IEEE Press, Wiley Interscience 2004.[CM99] S. Corson and J. Macker, "Mobile Ad HocNetworking (MANET): Routing ProtocolPerformance issues and EvaluationConsiderations," RFC2501, JAN 1999.[PO1] Charles Perkins, editor, Ad Hoc Networking,Addison-Wesley. 2001.[W02] Woolridge, M., An Introduction toMultiAgent Systems, John Wiley and Sons, Feb2002.[MAMC05] Macker J.P., Abramson, M. Mittu, R,Chao W., "Multi-Agent Systems in Mobile Ad hocNetwork Architectures", in IEEE MILCOM 2005Proceedings, November 2005.[MDC04] Macker J. P., J. Dean , W. Chao,"Simplified Multicast Forwarding in Mobile Ad

6

0-.-_a)

E0C-)_lecn

H-0

hoc Networks", IEEE MILCOM 2004Proceedings, November 2004.[ADMT06] Brian Adamson, Ian Downard, JoeMacker, Ian Taylor, "AgentJ: Enabling Java NS-2Simulations for Large Scale DistributedMultimedia Applications ", DistributedFrameworks for Multimedia Applicationsconference, May 2006.[D04] Downard, I., "Simulating Sensor Networksin NS-2", NRL Formal Report 5522-04-10, May2004.[ACM05] Abramson, M. Chao, W. Mittu, R."Design & Evaluation of Distributed RoleAllocation Algorithms in Open Environments",International Conference on Artificial Intelligence,2005.[M06] Macker, J.P., editor, ""Simplified MulticastForwarding for MANET", draft-ietf-manet-smf-02.txt, March 2006, work in progress,http://www.ietf.org.[FMO1] S. Fitzpatric, L. Meertens, "Anexperimental assessment of a stochastic, anytime,decentralized, soft colourer for sparse graphs,"First Symposium on Stochastic Algorithms:Foundations and Applications, 2001, pp. 49-64.[CMWO3] W. Chao, J. Macker, J Weston, "NRLMobile Network Emulator," IEEE MILCOM2003, Oct 2003.[K55] Kuhn, H. W., "The Hungarian method forthe assignment problem", Naval ResearchLogistics Quarterly, vol. 2, pp. 83-87, 1955.[PS98] C. H. Papadimitriou and K. Steiglitz,Combinatorial Optimization: Algorithms andComplexity, Dover Publications, 1998.[NCV06] M. J. North, N. T. Collier, and J. R. Vos,"Experiences creating three implementations of theRepast agent modeling toolkit," ACMTransactions on Modeling and ComputerSimulation, Vol. 16, No. 1, pp. 1-25, 2006

7