Agent-BasedSoftwareEngineering

Michael Wooldridge

MitsubishiElectricDigital Library Group18thFloor, CentrePoint,103New OxfordStreet

LondonWC1A 1EB,UnitedKingdommjw@dlib.com

September19,1997

Abstract

The technologyof intelligent agentsandmulti-agentsystemsseemsset to radicallyalter theway in which complex, distributed,opensystemsareconceptualizedandimple-mented.The purposeof this paperis to considerthe problemof building a multi-agentsystemasa softwareengineeringenterprise.Thearticle focuseson threeissues:(i) howagentsmightbespecified;(ii) how thesespecificationsmightberefinedor otherwisetrans-formedinto efficient implementations;and(iii) how implementedagentsandmulti-agentsystemsmightsubsequentlybeverified,in orderto show thatthey arecorrectwith respectto their specifications.Theseissuesare discussedwith referenceto a numberof case-studies. The article concludesby settingout someissuesandopenproblemsfor futureresearch.

1 Intr oduction

Intelligentagentsareninety-ninepercentcomputerscienceandonepercentAI.OrenEtzioni [12]

Over its 40-yearhistory, Artificial Intelligence(AI) hasbeensubjectto many andvariedcrit-icisms. Perhapsthe mostpersistentandtroublingof theseis thatAI hassimply failed to de-liver on its promises.Clearly, themoreextremepredictionsof someAI researchers(suchashuman-qualityintelligent robotswithin five decades)have not beenrealized.This would notbesoworrying if it wasobvious thatAI hadpaidoff in someotherway: if, for example,AItechniqueswerestandardcomponentsin workadaysoftware. But this is not the case. Evencomparatively mundaneAI techniques(suchasrule-basedsystems)arestill regardedashome-opathicmedicineby a significantproportionof themainstreamcomputersciencecommunity.Why is this? Therearemany reasons,including, for example,the reluctanceof softwarede-velopersto learnaboutandapplynew technologies,andtheinappropriatenessof mainstreamsoftwareengineeringtechniquesandtoolsfor AI systemdevelopment.But at leastpartof theansweris alsothatmany AI researcherseitherignoreor elseglossover thepragmaticconcernsof softwaredevelopment,for thesimplereasonthatthey do not regardthemselvesassoftware

1

engineers.AI pridesitself on beingmulti-disciplinary, takingcontributionsfrom many otherfields;but softwareengineeringis generallyregardedasneitheracontributornoraconcern.

Themostrecentinfantsto emergefromtheAI nurseryarethenotionsof anintelligentagentandagent-basedsystem[66]. An intelligentagentis generallyregardedasanautonomousde-cisionmakingsystem,whichsensesandactsin someenvironment(wediscussthequestionofwhatanagentis in moredetailbelow). Agentsappearto beapromisingapproachto developingmany complex applications,rangingfrom INTERNET-basedelectroniccommerceandinforma-tion gatheringto industrial processcontrol (see[28] for a survey). But unlessresearchersrecognisethatagent-basedsystemsareaboutcomputerscienceandsoftwareengineeringmorethanthey areaboutAI, thenwithin a decade,we may well be askingwhy agenttechnologysufferedthesamefateassomany otherAI ideasthatseemedgoodin principle.

In summary, theaim of this paperis to considertheproblemof building agent-basedsys-temsasasoftwareengineeringenterprise.In sodoing,thepaperconstructsaframework withinwhich futurework on agent-basedsoftwareengineeringmaybeplaced.Thepaperbeginsbymotivatingandintroducingtheideaof agent-basedsystems,andthengoesonto discussthekeysoftwareengineeringissuesof specification,refinement/implementation,andverificationwithrespectto agent-basedsystems.We begin by briefly discussingthequestionof whata speci-ficationis, andgo on to considerwhatanagent-basedspecificationmight look like. We thendiscusssomeof the dimensionsalongwhich an agent-basedspecificationframework mightvary, with particularreferenceto thenotionof agentsasrational,mentalisticsystems[57, 49].We subsequentlydiscussthekey issueof implementingor refiningagent-basedspecificationsinto systems,andfinally, we considertheverificationof agent-basedsystems.Throughoutthearticle,wetakecarebothto illustratethevariousissueswith casestudies,andto draw parallelswith moremainstreamsoftwareengineeringresearchwhereverpossible.Thearticleconcludeswith adiscussionof futurework directions.

It shouldbe notedthat the emphasisof this paperis on formal methodsfor agent-basedsoftwareengineering.This bias reflectsthe currentstateof the field. As the areamatures,andmoreagent-basedsystemsaredeployed, we will naturallyseean increasingnumberofstructuredbut informaldevelopmenttechniquesemerge.

2 Agent-BasedSystems

By anagent-basedsystem, we meanonein which thekey abstractionusedis thatof anagent.Agent-basedsystemsmay containa singleagent,(as in the caseof userinterfaceagentsorsoftwaresecretaries[39]), but arguablythe greatestpotentiallies in the applicationof multi-agentsystems[6]. By an agent, we meana systemthat enjoys the following properties[66,pp116–118]:

� autonomy: agentsencapsulatesomestate(that is not accessibleto other agents),andmake decisionsaboutwhat to do basedon this state,without the direct interventionofhumansor others;

� reactivity: agentsare situatedin an environment,(which may be the physicalworld,a uservia a graphicaluser interface,a collection of other agents,the INTERNET, orperhapsmany of thesecombined),areableto perceivethisenvironment(throughtheuseof potentiallyimperfectsensors),andareableto respondin a timely fashionto changesthatoccurin it;

2

� pro-activeness: agentsdo not simply act in responseto their environment,they areableto exhibit goal-directedbehaviour by takingtheinitiative;

� socialability: agentsinteractwith otheragents(andpossiblyhumans)via somekind ofagent-communicationlanguage [17], andtypically have the ability to engagein socialactivities (suchascooperative problemsolvingor negotiation)in orderto achieve theirgoals.

To moreclearlyunderstandwhatis meantby theseproperties,considerthefollowing scenario.Imagineanautonomousautomaticpilot controllinganaircraft, thatwe presentwith thegoalof safely landing at someairport. We expect the systemto plan how to achieve this goal(perhapsby makinguseof pre-compiledplans,ratherthanreasoningfrom first-principles),andif necessary, weexpectit to generatesubsidiarygoals(e.g.,ascendto analtitudeof 30,000feet,thenproceedduenorth at a speedof. . . ). This is what we meanby pro-activeness.We alsoexpectthesystemto try to executeits plans,but not blindly. Thus,in theeventof unforeseencircumstances(e.g.,a changein weatherconditions,a fault in theaircraft,a requestfrom air-traffic control),we expectthesystemto respondto thenew situationaccordingly, in time fortheresponseto beuseful.A systemthatspenthoursdeliberatingaboutwhatto donext wouldbenouseasanauto-pilot.This is whatwemeanby reactiveness.Wealsoexpectourauto-pilotto beableto cooperatewith air-traffic controllersandperhapsotheraircraftin orderto achieveits goals. This is what we meanby socialability. Note that this exampleis intendedto beextreme:it is notaclaimaboutwhatagentscancurrentlydo,or whatthey will beableto do inthenearfuture.It is merelyintendedto highlight thetypeof featuresweultimatelyhopeto seein agents.

Theconceptof anintelligentautonomousagentdid not appearin a vacuum.It is a naturaldevelopmentof variousothertrendsin AI andcomputerscience.In thesubsectionsthatfollow,we discusssomeancestorsof agents,andidentify theattributesthatmake themdistinct fromtheir forbears.

Agentsand AI

Thedisciplineof intelligentagentshasemergedlargely from researchin AI. In fact,onewayof definingAI is asthe problemof building an intelligentagent(RusellandNorvig’s recenttextbookon AI more-or-lesstakesthis view [53]). But it is importantto distinguishbetweenthe broadintelligencethat is the ultimategoal of the AI community, andthe intelligenceweseekin agents.Theonly intelligencerequirementwegenerallymakeof ouragentsis thattheycanmake an acceptabledecisionaboutwhat actionto performnext in their environment,intime for this decisionto beuseful. Otherrequirementsfor intelligencewill bedeterminedbythe domainin which the agentis applied: not all agentswill needto be capableof learning,for example. Capabilitiessuchascommonsensereasoning(a la CYC [19]) arenot requiredfor many importantapplicationdomains.Thus,aswepointedout in theintroduction,we viewtheapplicationandexploitationof agenttechnologyprimarily asa computerscienceproblem.Agentsaresimply softwarecomponentsthatmustbedesignedandimplementedin muchthesameway that othersoftwarecomponentsare. However, AI techniquesare often the mostappropriatewayof building agents.

3

Agentsand Expert Systems

Expert systemswere the AI technologyof the 1980s[23]. An expert systemis one that iscapableof solvingproblemsor giving advicein someknowledge-richdomain[26]. A classicexampleof anexpertsystemis MYCIN, whichwasintendedto assistphysiciansin thetreatmentof blood infectionsin humans. Perhapsthe most importantdistinctionbetweenagentsandexpert systemsis that expert systemslike MYCIN are inherentlydisembodied. By this, wemeanthatthey donot interactdirectlywith any environment:they gettheir informationnotviasensors,but througha useractingasmiddleman. In addition,expertsystemsarenot usuallyrequiredto operatein anything like real-time. Finally, we do not generallyrequireexpertsystemsto becapableof co-operatingwith otheragents.

Despitethesedifferences,someexpertsystems,(particularlythosethatperformreal-timecontroltasks),look verymuchlikeagents.A goodexampleis ARCHON [27], whichstartedlifeasacollectionof expertsystems,andendedupbeingviewedasamulti-agentsystem.ARCHON

operatesin thedomainof industrialprocesscontrol.

Agentsand Objects

Object-orienteddevelopmenttechniqueshavebeenpromotedas‘bestpractice’by theacademiccomputersciencecommunityfor at leasta decade.Fueledby recentpopularinterestin lan-guagessuchasJAVA, object-orientedapproachesarefinally leaving the relative backwaterofacademiaandenteringthe mainstream.While thereis muchongoingdebateaboutmany as-pectsof object-orienteddevelopment,thereis broadagreementthatanobjectis anentity thatencapsulatessomestateanda collectionof methods,correspondingto operationsthatmaybeperformedonthatstate.Methodsaretypically invokedasaresultof messagessentto theobject(onemaythink of thesemessagesasrequestsfor services).

Themostobviousdifferencebetweenthe ‘standard’objectmodelandour view of agent-basedsystemsis thatin traditionalobject-orientedprograms,thereis asinglethreadof control.In contrast,agentsareprocess-like,concurrentlyexecutingentities.However, therehave beenvariantson the basicobject model in which objectsare more like processes:object-basedconcurrentprogrammingmodelssuchasACTORS [1] have long beenrecognizedasanelegantmodel for concurrentcomputation,and ‘active object’ systemsare also quite similar; evencomparatively earlyon in thedevelopmentof object-orientedprogramming,it wasrecognizedthatsomethinglikeagentswouldbeanaturalnext step.

Giventheseremarks,it mayseemthatagentsareidenticalto (active)objectsin mostimpor-tant respects:they encapsulatebothstateandbehaviour, andcommunicatevia messagepass-ing1. But agentsarenot simplyobjectsby anothername.This is becauseanagentis a rationaldecisionmakingsystem: werequireanagentto becapableof reactiveandpro-activebehaviour,andof interleaving thesetypesof behaviour asthesituationdemands.Theobject-orientedre-searchcommunityhasnothingwhatsoever to sayaboutbuilding systemsthat arecapableofthis kind of behaviour. In contrast,thedesignof suchsystemsis a fundamentalresearchtopic

1Thefollowing is from a textbookonobject-orientedprogramming:

‘Thereis a tendency [. . . ] to think of objectsas“actors” andendow themwith human-like inten-tions andabilities. It’s temptingto think aboutobjects“deciding” what to do abouta situation,[and] “asking” otherobjectsfor information.[. . . ] Objectsarenot passivecontainersfor stateandbehaviour, but aresaidto betheagentsof aprogram’sactivity.” [44, p7]

4

in theintelligentagentscommunity[66]2. In addition,theobject-orientedcommunityhasnotaddressedissueslike cooperation,competition,negotiation,computationaleconomies,andsoon,thatform thefoundationfor multi-agentsystemsdevelopment[6].

2.1 AgentsasRational Systems

An obvious problemis how to conceptualizesystemsthat arecapableof rationalbehaviourof the type discussedabove. Oneof the mostsuccessfulsolutionsto this probleminvolvesviewing agentsasintentionalsystems[10], whosebehaviour canbepredictedandexplainedintermsof attitudessuchasbelief, desire,andintention[47]. Therationalefor this approachisthatin everydaylife, weusea folk psychology to explainandpredictthebehaviour of complexintelligentsystems:people.For example,we usestatementssuchasMichael intendsto writea paper in orderto explain Michael’s behaviour. Oncetold this statement,we expectto findMichaelshelvingothercommitmentsanddevelopingaplantowrite thepaper;wewouldexpecthim to spenda lot of time at his computer;we would not besurprisedto find him in a grumpymood;but wewouldbesurprisedto find him ata latenightparty.

This intentionalstance, wherebythebehaviour of a complex systemis understoodvia theattribution of attitudessuchasbelieving anddesiring,is simply an abstraction tool. It is aconvenientshorthandfor talkingaboutcomplex systems,whichallowsusto succinctlypredictandexplain theirbehaviour withouthaving to understandhow they actuallywork. Now, muchof computerscienceis concernedwith looking for goodabstractionmechanisms,sincetheseallow systemdevelopersto managecomplexity with greaterease:witnessproceduralabstrac-tion, abstractdatatypes,andmostrecently, objects.So,why not usetheintentionalstanceasan abstractiontool in computing— to explain, understand,and,crucially, programcomplexcomputersystems?

For many researchersin AI, this ideaof programmingcomputersystemsin termsof ‘men-talistic’ notionssuchasbelief,desire,andintentionis thekey componentof agent-basedcom-puting. Theconceptwasarticulatedmostclearlyby Yoav Shoham,in his agent-orientedpro-gramming(AOP)proposal[57]. Thereseemto bea numberof argumentsin favour of AOP.First, it offers us a familiar, non-technicalway to talk aboutcomplex systems.We neednoformal trainingto understandmentalistictalk: it is partof oureverydaylinguisticability.

Secondly, AOPmayberegardedasa kind of ‘post-declarative’ programming.In procedu-ral programming,sayingwhata systemshoulddo involvesstatingpreciselyhow to do it, bywriting a detailedalgorithm.Proceduralprogrammingis difficult becauseit is hardfor peopleto think in termsof thedetailrequired.In declarativeprogramming(a la PROLOG), theaim isto reducetheemphasisoncontrolaspects:we statea goalthatwe wantthesystemto achieve,andlet abuilt-in controlmechanism(e.g.,goal-directedrefutationtheoremproving) figureoutwhat to do in order to achieve it. However, in order to successfullywrite efficient or largeprogramsin a languagelike PROLOG, it is necessaryfor theprogrammerto haveadetailedun-derstandingof how thebuilt-in controlmechanismworks. This conflictswith oneof themaingoalsof declarative programming:to relieve the userof the needto dealwith control issues.In AOP, theideais that,asin declarative programming,we stateour goals,andlet thebuilt-incontrolmechanismfigureout what to do in orderto achieve them. In this case,however, thecontrolmechanismimplementssomemodelof rationalagency (suchastheCohen-Levesquetheoryof intention[9], or theRao-Georgeff BDI model[47]). Hopefully, this computational

2Of course,boththeideaof anagentandtheideaof anobjectbothoweagreatdebtto thepioneeringwork ofHewitt onopensystemsandtheACTOR model[1].

5

modelcorrespondsto our own intuitive understandingof (say)beliefsanddesires,andsoweneednospecialtrainingto useit. Returningto theexampleweusedabove,supposewetold ourautopilotagentto landat LAX airport. Thenwe would expecttheagentto continueto attemptto landat LAX until it hadsucceeded,or elseit discoveredthat this goalwasimpossible.Wewould not expect the agentto acceptany othergoal that wasnot consistentwith landingatLAX, andwe would expecttheagentto continueattemptingdifferentstrategiesfor achievingthegoalin theeventof difficulties. Ideally, asAOPprogrammers,we would not beconcernedwith howtheagentachievesits goals.Thereality, asever, doesnotquiteliveup to theideal.

Interestingly, we againfind thatresearchersfrom a moremainstreamcomputingdisciplinehave adopteda similar setof ideas. In theoreticalcomputerscience,logicsof knowledge areusedto reasonaboutdistributedsystems[13]. Theideais thatwhenspecifyingsuchsystems,oneoftenmakesuseof statementssuchasthefollowing:

IF processi knowsprocessj hasreceivedmessagem�THEN processi shouldsendprocessj messagem� . (1)

This kind of statementmay be formalisedusinga logic of knowledge. Althoughmostworkon knowledgetheory in distributedsystemshasbeenratherabstract,andunconnectedwithsoftwarepractice,researchershave recentlybegunto payseriousattentionto thepossibilityofdirectly programmingsystemsusingstatementssuchas(1), above [13, pp233–271].Theideais verycloseto AOP.

Now thatweunderstandwhatanagentis, wecanbegin to look atsoftwareengineeringforagent-basedsystems.

3 Specification

The software developmentprocessbegins by establishingthe client’s requirements.Whenthis processis complete,a specificationis developed,which setsout the functionality of thenew system.Thepurposeof this sectionis to considerwhata specificationfor anagent-basedsystemmight look like. Whataretherequirementsfor anagentspecificationframework?Whatsortof propertiesmustit becapableof representing?To answerthisquestion,we returnto thepropertiesof agents,asdiscussedin theprecedingsection.

Weobservedabovethatagentsaresituatedin anenvironment,andareableto perceive thisenvironmentthroughsensorsof somekind. Agentsthushave informationabouttheir envi-ronment.This leadsto our first requirement:that the agentspecificationframework mustbecapableof representingboth thestateof theenvironmentitself, andthe informationanagenthasabouttheenvironment.It is worthmakingsomecommentsaboutwhatpropertiesthisinfor-mationmight have. First, the informationanagenthasmaybe incorrect.Theagent’s sensorsmaybe faulty, the informationmight beout of date,or theagentmayhave beendeliberatelyor accidentallygivenfalseinformation.Secondly, the informationanagenthasis not directlyavailableto otheragents:agentsdo not sharedatastructures,anddo not have accessto theprivatedatastructuresof otheragents(this is partof whatwe meantby autonomy).Third, theenvironmentwill containotheragents,eachwith theirown informationabouttheenvironment.Thusanagentmayhave informationaboutthestateof otheragents:we mayneedto representsuch‘nested’information.Notethatit is commonpracticeto referto theinformationavailableto anagentasthatagent’sbeliefs.

6

Now considerthenotionof reactivity. Softwaresystemsmaybebroadlydividedinto twotypes: functionaland reactive[45]. A functionalsystemis one that simply takessomein-put, performssomecomputationover this input, andeventuallyproducessomeoutput. Suchsystemsmay be viewed asfunctionsf � I � O from a set I of inputsto O of outputs. Theclassicexampleof sucha systemis a compiler, which canbe viewed asa mappingfrom asetI of legal sourceprogramsto a setO of correspondingobjector machinecodeprograms.Although the internalcomplexity of the mappingmay be great(e.g., in the caseof a reallycomplex programminglanguage),it is neverthelessthe casethat functionalprogramsare,ingeneral,inherentlysimplerto specify, design,andimplementthanreactive systems.Becausefunctionalsystemsterminate,it is possibleto usepre-andpost-conditionformalismsin orderto reasonaboutthem[24]. In contrast,reactive systemsdo not terminate,but rathermaintainanongoinginteractionwith theirenvironment.It is thereforenotpossibleto usepre-andpost-conditionformalismssuchasHoarelogic to reasonaboutthem.Instead,reactivesystemsmustbespecifiedin termsof their ongoingbehaviour. Thenext requirementfor our agentspecifi-cationframework is that it mustbe capableof representingthis inherentlyreactive natureofagentsandmulti-agentsystems.Note that oneof the mostsuccessfulformalismsdevelopedfor specifyingreactive systemsis temporallogic. The ideais thatwhenspecifyinga reactivesystem,oneoftenwantsto staterequirementssuchas‘if a requestis received,thena responseis eventuallysent’.Suchrequirementsareeasilyandelegantlyexpressedin temporallogic.

The third aspectof agentsasdiscussedabove is pro-activeness, by which we meanthatagentsareable to exhibit goal-directedbehaviour. (Note that we usethe term ‘goal’ fairlyloosely. We includesuchnotionsascommitmentsor obligationsin our usage.) It doesnotfollow that in orderto exhibit goal-directedbehaviour, anagentmustexplicitly generateandrepresentgoals[38], althoughthisisby farthemostcommonapproach.Ouragentspecificationframework mustbecapableof representingtheseconative(goal-directed)aspectsof agency.

Finally, our agentsare able to act. Agentsdo not typically have completecontrol overtheir environment(our auto-pilotcannotcontrol the weather),but they aregenerallyabletoinfluencetheirenvironmentby performingactions,andmayhavereliablecontroloverportionsof it. We requiresomeway of representingtheseactionswithin our specificationframework.To summarize,an agentspecificationframework must be capableof capturingat least thefollowing aspectsof anagent-basedsystem:

� thebeliefsagentshave;

� theongoinginteractionagentshavewith theirenvironment;

� thegoalsthatagentwill try to achieve;

� theactionsthatagentsperformandtheeffectsof theseactions.

Whatsortof specificationframework is capableof representingsuchaspectsof asystem?Themostsuccessfulapproachappearsto be the useof a temporal modallogic [7] (spacerestric-tions prevent a detailedtechnicaldiscussionon suchlogics — see,e.g., [66] for a detailedoverview andextensive references).A typical temporalmodalagentspecificationframeworkwill contain:

� normalmodallogic connectivesfor representingagent’sbeliefs;

� temporallogic connectivesfor representingthe dynamicsof the system— its ongoingbehaviour;

7

InformationAspectsknowledge[13]belief [33]collective informationattitudes[20]

Temporal Aspectslinearvsbranching[11]densevsdiscrete[4]directreferencevstenseoperators[56]pointbasedvs interval based[2]

ConativeAspectsdesireintention[9]obligation[42]choice[5]collectivepro-attitudes[34]

Actionsdirectrepresentation[22]implicit representation[55]

Table1: Dimensionsof AgentSpecification

� normalmodallogic connectivesfor representingconatives(e.g.,desires,intentions,obli-gations);

� someapparatusfor representingtheactionsthatagentsperform.

Giventheserequirements,thereareagreatmany dimensionsalongwhichanagentspecificationframework may vary: someof thesedimensionsaresummarizedin Table1. Note that thereis by no meansany consensuson the desirablepropertiesof what we might call an ‘agenttheory’. For example,two of thebestknown agenttheoriesaretheCohen-Levesquetheoryofintention[9], andtheRao-Georgeff belief-desire-intentionmodel[47]. TheCohen-Levesquemodel takesasprimitive just two attitudes:beliefsandgoals. Otherattitudes(in particular,the notionof intention) arebuilt up from these.In contrast,Rao-Georgeff take intentionsasprimitives, in addition to beliefsandgoals. Also, Cohen-Levesqueadopta linear temporalmodel,(andstatea numberof objectionsto branchingtemporalmodels),whereasbranchingtime is moreor lessessentialto the Rao-Georgeff model. As a result, the two formalisms,thoughcloselyrelated,areirreconcilable.

Thekey technicalproblemfacedby agenttheoristsis developingamodelthatgivesagoodaccountof theinterrelationshipsbetweenthevariousattitudesthattogethercompriseanagentsinternalstate.Somecontemporarymodelsarereviewedin [66].

CaseStudy: The Belief-Desire-Intention Model

Oneof the mostsuccessfulagenttheoriesis the belief-desire-intention(BDI) modelof RaoandGeorgeff (see[47] for extensive references).The technicaldetailsof BDI aresomewhatinvolved,andsohere,weshallsimplysummarizethemainconceptsthatunderpinBDI models.

8

As thenamesuggests,theinternalstateof aBDI agentis comprisedof threekey datastruc-tures,which areintendedto looselycorrespondto beliefs,desires,andintentions.An agent’sbeliefsareintendedto representtheinformationit hasabouttheworld, aswesuggestedabove.Beliefswill typically berepresentedsymbolically: in theProceduralReasoningSystem(PRS)— thebest-known BDI implementation— beliefslook verymuchlike PROLOG facts[18]. Anagent’s desiresmay be thoughtof as the tasksallocatedto it. An agentmay not in fact beableto achieve all of its desires,andin humans,desiresmayevenbeinconsistent.An agent’sintentionsrepresentdesiresthat it hascommittedto achieving. The intuition is thatasagentswill not,in general,beableto achieveall theirdesires,evenif thesedesiresareconsistent,theymustthereforefix uponsomesubsetof availabledesiresandcommit resourcesto achievingthem. Chosendesiresareintentions. Theseintentionswill thenfeedbackinto futuredecisionmaking: for example,anagentshouldnot in futureadoptintentionsthatconflict with thoseitcurrentlyholds.

The BDI modelof agency hasbeenformalisedby RaoandGeorgeff in a family of BDIlogics [48, 51]. Theselogicsareextensionsto theexpressive branchingtime logic CTL

�[11],

which alsocontainnormalmodalconnectivesfor representingbeliefs,desires,andintentions.Most work on BDI logics hasfocussedon possiblerelationshipsbetweenthe three‘mentalstates’[48], andmorerecently, on developingproof methodsfor restrictedforms of the log-ics [51].

3.1 Discussion

Specificationlanguagesfor agent-basedsystemsareanorderof magnitudemorecomplex thanthe comparatively simpletemporalandmodal languagesthat have becomecommonplaceinmainstreamcomputerscience.Typically, they aretemporallogicsenrichedby a family of ad-ditionalmodalconnectives,for representingthe‘mentalstate’of anagent.Thereareanumberof problemswith suchlanguages,in additiontheir conceptualcomplexity. The mostworry-ing of theseis with respectto their semantics.While the temporalcomponentof theselogicstendsto be ratherstandard,the semanticsfor the additionalmodalconnectivesaregiven inthe normalmodal logic tradition of possibleworlds [7]. So, for example,an agent’s beliefsin somestatearecharacterisedby a setof differentstates,eachof which representsonepos-sibility for how the world could actuallybe, given the informationavailableto the agent. Inmuchthe sameway, an agentsdesiresin somestatearecharacterisedby a setof statesthatareconsistentwith theagentsdesires.Intentionsarerepresentedsimilarly. Thereareseveraladvantagesto thepossibleworldsmodel: it is well-studiedandwell-understood,andtheasso-ciatedmathematics(known ascorrespondencetheory) is extremelyelegant. Theseattractivefeaturesmake possibleworlds the semanticsof choicefor almostevery researcherin formalagenttheory. However, therearealsoanumberof seriousdrawbacksto possibleworldsseman-tics. First,possibleworldssemanticsimply thatagentsarelogically perfectreasoners,(in thattheirdeductivecapabilitiesaresoundandcomplete),andthey have infinite resourcesavailablefor reasoning.No realagent,artificial or otherwise,hastheseproperties.

Secondly, possibleworlds semanticsaregenerallyungrounded. That is, thereis usuallynopreciserelationshipbetweentheabstractaccessibilityrelationsthatareusedto characterizeanagentsstate,andany concretecomputationalmodel. As we shallseein latersections,thismakesit is difficult to go from a formal specificationof a systemin termsof beliefs,desires,and so on, to a concretecomputationalsystem. Similarly, given a concretecomputationalsystem,thereis generallyno way to determinewhatthebeliefs,desires,andintentionsof that

9

systemare. If temporalmodallogicsof thetypediscussedabove areto betakenseriouslyasspecificationlanguages,thenthisproblemis significant.

4 Implementation

Specificationis not theendof thestory in softwaredevelopment.Oncegivena specification,we mustimplementa systemthat is correctwith respectto this specification.Thenext issuewe consideris this move from abstractspecificationto concretecomputationalmodel. Thereareat leastthreepossibilitiesfor achieving this transformation:

1. manuallyrefinethespecificationinto anexecutableform via someprincipledbut infor-mal refinementprocess(asis thenormin mostcurrentsoftwaredevelopment);

2. somehow directlyexecuteor animatetheabstractspecification;or

3. somehow translateor compilethespecificationinto aconcretecomputationalform usinganautomatictranslationtechnique.

In thesub-sectionsthatfollow, weshallinvestigateeachof thesepossibilitiesin turn.

4.1 Refinement

At the time of writing, most software developersusestructuredbut informal techniquestotransformspecificationsinto concreteimplementations.Probablythemostcommontechniquesin widespreadusearebasedon theideaof top-down refinement.In this approach,anabstractsystemspecificationis refinedinto anumberof smaller, lessabstractsub-systemspecifications,which togethersatisfythe original specification.If thesesub-systemsarestill too abstracttobe implementeddirectly, then they arealso refined. The processrecursesuntil the derivedsub-systemsaresimpleenoughto be directly implemented.Throughout,we areobligedtodemonstratethateachsteprepresentsa truerefinementof themoreabstractspecificationthatprecededit. This demonstrationmay take the form of a formal proof, if our specificationispresentedin, say, Z [60] or VDM [29]. Moreusually, justificationis by informalargument.

For functionalsystems,therefinementprocessiswell understood,andcomparativelystraight-forward. Refinementcalculi exist, which enablethesystemdeveloperto take a pre-andpost-conditionspecification,andfrom it systematicallyderive an implementationthroughthe useof proof rules[43]. Part of this reasonfor this comparative simplicity is that thereis oftenaneasilyunderstandablerelationshipbetweenthe pre- andpost-conditionsthat characterizeanoperationandtheprogramstructuresrequiredto implementit.

For reactivesystems,refinementis notsostraightforward.This is becausereactivesystemsmustbespecifiedin termsof their ongoingbehaviour. In contrastto pre-andpost-conditionformalisms,it is not soeasyto determinewhatprogramstructuresarerequiredto realisesuchspecifications.Therefinementproblemfor agent-basedsystems,wherespecificationsmayberegardedaseven moreabstractthan thosefor reactive systems,is harderstill. As a conse-quence,researchershaveonly justbegunto investigatetherefinementof agent-basedsystems.In thesubsectionsthatfollow, weshallreview two examplesof thiswork.

10

CaseStudy: Agentsin Z

Luck andd’Invernohave developedan agentspecificationframework in the Z language,al-though,aswe shall see,the typesof agentsconsideredin this framework aresomewhatdif-ferentfrom thosediscussedabove [36]. They definea four-tieredhierarchyof theentitiesthatcanexist in an agent-basedsystem.They startwith entities, which areinanimateobjects—they have attributes(colour, weight, position),but nothingelse. They thendefineobjectstobeentitiesthathavecapabilities(e.g.,tablesareentitiesthatarecapableof supportingthings).Agentsarethendefinedto beobjectsthathavegoals,andarethusin somesenseactive;finally,autonomousagentsaredefinedto beagentswith motivations.Theideais thatachaircouldbeviewedastakingonmy goalof supportingmewhenI amusingit, andcanhencebeviewedasanagentfor me. But we would not view a chairasanautonomousagent,sinceit hasno mo-tivations(andcannoteasilybeattributedthem).Startingfrom this basicframework, Luck andd’Invernogoon to examinethevariousrelationshipsthatmightexist betweenagentsof differ-enttypes.In [37], they examinehow anagent-basedsystemspecifiedin their framework mightbe implemented.They found that therewasa naturalrelationshipbetweentheir hierarchicalagentspecificationframework andobject-orientedsystems:

‘The formal definitionsof agentsand autonomousagentsrely on inheriting thepropertiesof lower-level components.In the Z notation,this is achieved throughschemainclusion[. . . ]. This is easilymodelledin ¡C++ by deriving oneclassfromanother. [. . . ] Thuswe movefrom a principledbut abstracttheoreticalframeworkthrougha moredetailed,yet still formal,modelof thesystem,down to anobject-orientedimplementation,preservingthehierarchicalstructureateachstage.’ [37]

TheLuck-d’Invernoformalismis attractive,particularlyin thewaythatit capturestherelation-shipsthatcanexist betweenagents.Theemphasisis placedon thenotionof agentsactingforanother, ratherthanon agentsasrationalsystems,aswe discussedabove. Thetypesof agentsthat theapproachallows usto developarethusinherentlydifferentfrom the ‘rational’ agentsdiscussedabove. So, for example,theapproachdoesnot helpus to constructagentsthatcaninterleavepro-activeandreactivebehaviour. This is largelya resultof thechosenspecificationlanguage:Z. This languageis inherentlygearedtowardsthespecificationof operation-based,functionalsystems.Thebasiclanguagehasno mechanismsthatallow usto easilyspecifytheongoingbehaviour of anagent-basedsystem3.

CaseStudy: A Methodologyfor BDI Agents

In section3,wenotedthatthebelief-desire-intention(BDI) modelis oneof themostsuccessfulgeneralframeworksfor agency. In [32], Kinny et al proposea four-stagedesignmethodologyfor systemsof BDI agents.Themethodologyis closelylinkedto a specificrealizationof theBDI model:thePRS architecture[18]. Themethodologymaybesummarizedasfollows:

1. Identify therelevant rolesin theapplicationdomain,andon thebasisof these,developanagentclasshierarchy. An examplerolemightbeweathermonitor, wherebyagenti isrequiredto makeagentj awareof theprevailing weatherconditionseveryhour.

3Thereareof courseextensionsto Z designedfor thispurpose.

11

2. Identify theresponsibilitiesassociatedwith eachrole, theservicesrequiredby andpro-videdby therole,andthendeterminethegoalsassociatedwith eachservice.With respectto the above example,the goalswould be to find out the currentweather, andto makeagentj awareof this information.

3. For eachgoal,determinetheplansthatmaybeusedto achieve it, andthecontext con-ditionsunderwhich eachplanis appropriate.With respectto theaboveexample,a planfor thegoalof makingagentj awareof theweatherconditionsmight involve sendingamessageto j.

4. Determinethe belief structureof the system— the informationrequirementsfor eachplanandgoal. With respectto theabove example,we might proposea unarypredicatewindspeed

�x� to representthefact that thecurrentwind speedis x. A planto determine

thecurrentweatherconditionswouldneedto beableto representthis information.

Note that the analysisprocesswill be iterative, as in more traditionalmethodologies.Theoutcomewill beamodelthatcloselycorrespondsto thePRS agentarchitecture.As aresult,themovefrom end-designto implementationusingPRS is relatively simple.

Kinny et al illustratetheir methodologyby applyingit to an implementedair traffic man-agementsystemcalledOASIS. Thissystem,currentlybeingdeployedatSidney airportin Aus-tralia, is, by any measure,a largeanddifficult application.It is arguablythemostsignificantagentapplicationyet developed.Thattheagentapproachhasbeensuccessfullyappliedin thisdomainis encouraging;theuseof themethodologyevenmoreso.

4.2 Dir ectly ExecutingAgent Specifications

Onemajordisadvantagewith manualrefinementmethodsis thatthey introducethepossibilityof error. If no proofsareprovided, to demonstratethat eachrefinementstepis indeeda truerefinement,thenthecorrectnessof the implementationprocessdependsuponlittle morethanthe intuitions of the developer. This is clearly anundesirablestateof affairs for applicationsin which correctnessis a major issue.Onepossibleway of circumventingthis problem,thathasbeenwidely investigatedin mainstreamcomputerscience,is to get rid of the refinementprocessaltogether, anddirectlyexecutethespecification.

It mightseemthatsuggestingthedirectexecutionof complex agentspecificationlanguagesis naive. (It is exactly thekind of suggestionthatdetractorsof symbolicAI hate.)Oneshouldbe thereforebe very carefulaboutwhatclaimsor proposalsonemakes. However, in certaincircumstances,thedirectexecutionof agentspecificationlanguagesis possible.

Whatdoesit mean,to executea formula � of logic L? It meansgeneratinga logicalmodel,M, for � , suchthatM �� [15]. If this coulddonewithout interferencefrom theenvironment— if theagenthadcompletecontrolover its environment— thenexecutionwould reducetoconstructive theoremproving, wherewe show that � is satisfiableby building a modelfor � .In realityof course,agentsarenot interference-free:they mustiteratively constructamodelinthepresenceof input from theenvironment.Executioncanthenbeseenasa two-way iterativeprocess:

� environmentmakessomethingtrue;

� agentrespondsby doingsomething,i.e.,makingsomethingelsetruein themodel;

12

� environmentresponds,makingsomethingelsetrue;

� . . .

Executionof logical languagesandtheoremproving arethuscloselyrelated.This tells usthatthe executionof sufficiently rich (quantified)languagesis not possible(sinceany languageequalin expressivepower to first-orderlogic is undecidable).

A useful way to think aboutexecutionis as if the agentis playing a gameagainsttheenvironment.Thespecificationrepresentsthegoalof thegame:theagentmustkeepthegoalsatisfied,while the environmenttries to prevent the agentdoing so. The gameis playedbyagentandenvironmenttakingit in turnsto build a little moreof themodel.If thespecificationever becomesfalsein the (partial) model,thenthe agentloses. In real reactive systems,thegameis never over: the agentmustcontinueto play forever. Of course,somespecifications(logically inconsistentones)cannotever besatisfied.A winningstrategy for building modelsfrom (satisfiable)agentspecificationsin thepresenceof arbitraryinput from theenvironmentis anexecutionalgorithmfor thelogic.

CaseStudy: Concurrent METATEM

ConcurrentMETATEM is a programminglanguagefor multi-agentsystems,that is basedontheideaof directly executinglineartime temporallogic agentspecifications[16, 14]. A Con-currentMETATEM systemcontainsa numberof concurrentlyexecutingagents,eachof whichis programmedby giving it a temporallogic specificationof the behaviour it is intendedtheagentshouldexhibit. An agentspecificationhasthe form � i Pi Fi, wherePi is a temporallogic formulareferringonly to thepresentor past,andFi is a temporallogic formulareferringto thepresentor future.ThePi Fi formulaeareknown asrules. Thebasicideafor executingsuchaspecificationmaybesummedup in thefollowing slogan:

on thebasisof thepastdo thefuture.

Thuseachrule is continuallymatchedagainstan internal,recordedhistory, andif a matchisfound, thenthe rule fires. If a rule fires, thenany variablesin the future time partareinstan-tiated,andthe future time part thenbecomesa commitmentthat the agentwill subsequentlyattemptto satisfy. Satisfyingacommitmenttypically meansmakingsomepredicatetruewithintheagent.Hereis asimpleexampleof aConcurrentMETATEM agentdefinition:

ask�x� give

�x����

ask�x��� �

give�x��� � ask

�x��� �

give�x�

give�x ��� give

�y� �

x y�Theagentin this exampleis a controllerfor a resourcethat is infinitely renewable,but whichmayonly bepossessedby oneagentat any giventime. Thecontrollermustthereforeenforcemutualexclusion. The predicateask

�x � meansthat agentx hasasked for the resource.The

predicategive�x� meansthat the resourcecontrollerhasgiven the resourceto agentx. The

resourcecontroller is assumedto be the only agentable to ‘give’ the resource. However,many agentsmayaskfor theresourcesimultaneously. Thethreerulesthatdefinethis agent’sbehaviour maybesummarizedasfollows:

Rule1: if someoneasks,theneventuallygive;

13

Rule2: don’t giveunlesssomeonehasaskedsinceyou lastgave;and

Rule3: if you give to two people,thenthey mustbe the sameperson(i.e., don’t give to morethanonepersonata time).

NotethatConcurrentMETATEM agentscancommunicateby asynchronousbroadcastmessagepassing,thoughthedetailsarenot importanthere.

4.3 Compiling Agent Specifications

An alternative to direct executionis compilation. In this scheme,we take our abstractspec-ification, andtransformit into a concretecomputationalmodelvia someautomaticsynthesisprocess.Themainperceivedadvantagesof compilationover directexecutionarein run-timeefficiency. Directexecutionof anagentspecification,asin ConcurrentMETATEM, above,typi-cally involvesmanipulatingasymbolicrepresentationof thespecificationatruntime. Thisma-nipulationgenerallycorrespondsto reasoningof someform, which is computationallycostly(andin many cases,simply impracticablefor systemsthat mustoperatein anything like realtime). In contrast,compilationapproachesaim to reduceabstractsymbolicspecificationsto amuchsimplercomputationalmodel,which requiresno symbolicrepresentation.The‘reason-ing’ work is thusdoneoff-line, at compile-time;executionof thecompiledsystemcanthenbedonewith little or no run-timesymbolicreasoning.As a result,executionis muchfaster. Theadvantagesof compilationover directexecutionarethusthoseof compilationover interpreta-tion in mainstreamprogramming.

Compilationapproachesusuallydependupon the closerelationshipbetweenmodelsfortemporal/modallogic (which are typically labeledgraphsof somekind), andautomata-likefinite statemachines. Crudely, the idea is to take a specification� , and do a constructiveproof of the implementabilityof � , whereinwe show that the specificationis satisfiablebysystematicallyattemptingto build a modelfor it. If the constructionprocesssucceeds,thenthespecificationis satisfiable,andwe have a modelto prove it. Otherwise,thespecificationisunsatisfiable.If we have a model,thenwe ‘readoff ’ the automatonthat implements� fromits correspondingmodel. The mostcommonapproachto constructive proof is the semantictableauxmethodof Smullyan[59].

In mainstreamcomputerscience,thecompilationapproachto automaticprogramsynthesishasbeeninvestigatedby a numberof researchers.Perhapstheclosestto our view is theworkof PnueliandRosner[46] on theautomaticsynthesisof reactivesystemsfrom branchingtimetemporallogic specifications.The goal of their work is to generatereactive systems,whichsharemany of the propertiesof our agents(the main differencebeing that reactive systemsarenot generallyrequiredto becapableof rationaldecisionmakingin theway we describedabove). To do this, they specifya reactivesystemin termsof a first-orderbranchingtime tem-poral logic formula � x � y A � � x � y � : the predicate� characterisesthe relationshipbetweeninputsto thesystem(x) andoutputs(y). Inputsmaybethoughtof assequencesof environmentstates,andoutputsascorrespondingsequencesof actions.TheA is a branchingtime temporallogic connective meaning‘on all paths’,or ‘in all possiblefutures’. The specificationis in-tendedto expressthefactthat in all possiblefutures,thedesiredrelationship� holdsbetweentheinputsto thesystem,x, andits outputs,y. Thesynthesisprocessitself is rathercomplex: itinvolvesgeneratinga Rabintreeautomaton,andthencheckingthis automatonfor emptiness.PnueliandRosnershow thatthetimecomplexity of thesynthesisprocessis doubleexponentialin thesizeof thespecification,i.e.,O

��� � c � n � , wherec is aconstantandn ���� is thesizeof the

14

specification� . Thesizeof thesynthesizedprogram(thenumberof statesit contains)is of thesamecomplexity.

ThePnueli-Rosnertechniqueis rathersimilar to (andin factdependsupon)techniquesde-velopedbyWolper, Vardi,andcolleaguesfor synthesizingBuchiautomatafromlineartemporallogic specifications[61]. Buchi automataarethosethatcanrecognise� -regular expressions:regular expressionsthat may containinfinite repetition. A standardresult in temporallogictheoryis thata formula � of lineartime temporallogic is satisfiableif andonly if thereexistsa Buchi automatonthat acceptsjust the sequencesthat satisfy � . Intuitively, this is becausethesequencesover which lineartime temporallogic is interpretedcanbeviewedas � -regularexpressions.This resultyields a decisionprocedurefor linear time temporallogic: to deter-minewhethera formula � is satisfiable,constructanautomatonthatacceptsjust the(infinite)sequencesthat correspondto modelsof � ; if the setof suchsequencesis empty, then � isunsatisfiable.Thetechniquefor constructinganautomatonfrom thecorrespondingformulaiscloselybasedonWolper’s tableauproofmethodfor temporallogic [62].

Similarautomaticsynthesistechniqueshavealsobeendeployedto developconcurrentsys-tem skeletonsfrom temporallogic specifications.Mannaand Wolper presentan algorithmthat takesasinput a linear time temporallogic specificationof the synchronizationpart of aconcurrentsystem,andgeneratesasoutputa CSPprogramskeleton([25]) that realizesthespecification[41]. The ideais that the functionalityof a concurrentsystemcangenerallybedividedinto two parts:a functionalpart,which actuallyperformstherequiredcomputationintheprogram,anda synchronizationpart,whichensuresthatthesystemcomponentscooperatein the correctway. For example,thesynchronizationpartwill be responsiblefor any mutualexclusionthat is required. The synthesisalgorithm, (like the synthesisalgorithmfor Buchiautomata,above), is basedon Wolper’s tableauproof methodfor temporallogic [62]. Verysimilarwork is reportedby ClarkeandEmerson[8]: they synthesizesynchronizationskeletonsfrom branchingtime temporallogic (CTL) specifications.

CaseStudy: SituatedAutomata

Perhapsthe best-known exampleof this approachto agentdevelopmentis the situatedau-tomataparadigmof RosenscheinandKaelbling [52, 31]. In this approach,an agenthastwomaincomponents:

� a perceptionpart,which is responsiblefor observingtheenvironment,andupdatingtheinternalstateof theagent;and

� an action part, which is responsiblefor decidingwhat actionto perform,basedon theinternalstateof theagent.

RosenscheinandKaelblingdevelopedtwo programsto supportthedevelopmentof thepercep-tion andactioncomponentsof anagentrespectively. The RULER programtakesa declarativeperceptionspecificationandcompilesit down to a finite statemachine.The specificationisgivenin termsof a theoryof knowledge.Thesemanticsof knowledgein thedeclarativespeci-ficationlanguagearegivenin termsof possibleworlds,in thewaydescribedabove. Crucially,however, thepossibleworldsunderlyingthis logic aregivena precisecomputationalinterpre-tation,in termsof thestatesof afinite statemachine.It is thispreciserelationshipthatpermitsthesynthesisprocessto takeplace.

15

Theactionpartof anagentin RosenscheinandKaelbling’s framework is specifiedin termsof goal reductionrules, which encodeinformationabouthow to achieve goals. The GAPPS

programtakesasinput a goalspecification,anda setof goalreductionrules,andgeneratesasoutputasetof situationactionrules, whichmaybethoughtof asa lookuptable,definingwhattheagentshoulddo undervariouscircumstances,in orderto achieve thegoal. Theprocessofdecidingwhatto do is thenvery simplein computationalterms,involving no reasoningat all.(A similar technique,calleduniversalplans, wasdevelopedby Schoppers[54].)

4.4 Discussion

Structuredbut informal refinementtechniquesarethe mainstayof real-world softwareengi-neering. If agent-orientedtechniquesareever to becomewidely usedoutsidethe academiccommunity, theninformal, structuredmethodsfor agent-baseddevelopmentwill beessential.Onepossibilityfor suchtechniques,followedby Luck andd’Inverno,is to useastandardspec-ification technique(in their case,Z), and usetraditional refinementmethods(in their case,object-orienteddevelopment)to transformthespecificationinto an implementation.This ap-proachhastheadvantageof beingfamiliar to a muchlargeruser-basethanentirelynew tech-niques,but suffers from thedisadvantageof presentingtheuserwith no featuresthatmake itparticularlywell-suitedto agentspecification.It seemscertainthat therewill be muchmorework on manualrefinementtechniquesfor agent-basedsystemsin the immediatefuture, butexactlywhatform thesetechniqueswill take is notclear.

Now considerthepossibilityof directlyexecutingagentspecifications.A numberof prob-lems immediatelysuggestthemselves. The first is that of finding a concretecomputationalinterpretationfor theagentspecificationlanguagein question.To seewhatwe meanby this,considermodelsfor theagentspecificationlanguagein ConcurrentMETATEM. Theseareverysimple: essentiallyjust lineardiscretesequencesof states.Temporallogic is (amongstotherthings)simplya languagefor expressingconstraintsthatmusthold betweensuccessivestates.Executionin ConcurrentMETATEM is thusa processof generatingconstraintsaspast-timeantecedentsaresatisfied,andthentrying to build a next statethat satisfiestheseconstraints.Constraintsareexpressedin temporallogic, which implies that they may only be in certain,regularforms. Becauseof this, it is possibleto deviseanalgorithmthatis guaranteedto buildanext stateif it is possibleto doso.Suchanalgorithmis describedin [3].

The agentspecificationlanguageuponwhich ConcurrentMETATEM is basedthushasaconcretecomputationalmodel,anda comparatively simpleexecutionalgorithm.Contrastthisstateof affairs with the kinds of temporalmodalagentspecificationlanguagesdiscussedinsection3, wherewe have not only a temporaldimensionto the logic, but alsomodalitiesforreferringto beliefs,desires,andsoon. In general,thesemodelshave ungroundedsemantics.That is, thesemanticstructuresthatunderpintheselogics(typically accessibilityrelationsforeachof themodaloperators)have no concretecomputationalinterpretation.As a result,it isnotclearhow suchagentspecificationlanguagesmightbeexecuted.

Anotherobviousproblemis thatexecutiontechniquesbasedontheoremproving areinher-ently limited whenappliedto sufficiently expressive(first-order)languages,asfirst-orderlogicis undecidable.However, complexity is a problemevenin thepropositionalcase.For ‘vanilla’propositionallogic, thedecisionproblemfor satisfiabilityis NP-complete[13, p72]; for lineartemporallogic, theproblemis PSPACE-complete[58]; for simplemodallogicsof knowledge,theproblemis NP-complete,andfor morecomplex modallogicsof knowledge,theproblemisEXPTIME-complete[13,p73]; for logicsthatcombinetemporaland(S5)modalaspects,thede-

16

cisionproblemvariesfrom PSPACE-completein thesimplestcaseto � �� -complete,(andhenceundecidable) in thepropositionalcase,dependingonwhatsemanticassumptionsaremade[13,p289].

Turning to automaticsynthesis,we find that the techniquesdescribedabove have beendevelopedprimarily for propositionalspecificationlanguages.If we attemptto extendthesetechniquesto moreexpressive,first-orderspecificationlanguages,thenweagainfind ourselvescomingup againstthe undecidabilityof quantifiedlogic. Even in the propositionalcase,thetheoreticalcomplexity of theoremproving for modalandtemporallogics is likely to limit theeffectivenessof compilationtechniques:givenanagentspecificationof size1000,a synthesisalgorithmthatrunsin exponentialtimewhenusedoff-line is nomoreusefulthananexecutionalgorithmwhich runsin exponentialtimeon-line.

Anotherproblemwith respectto synthesistechniquesis thatthey typically resultin finite-state,automatalike machines,thatarestrictly lesspowerful thanTuring machines.In partic-ular, thesystemsgeneratedby theprocessesoutlinedabove cannotmodify their behaviour atrun-time. In short,they cannotlearn.While for many applications,this is acceptable— evendesirable— for equallymany others,it is not. In expertassistantagents,of thetypedescribedin [39], learningis prettymuchtheraisond’etre. Attemptsto addressthis issuearedescribedin [30].

5 Verification

Oncewe have developeda concretesystem,we needto show that this systemis correctwithrespectto ouroriginalspecification.Thisprocessis known asverification, andit is particularlyimportantif we have introducedany informality into thedevelopmentprocess.For example,any manualrefinement,donewithout a formal proof of refinementcorrectness,createsthepossibilityof a faulty transformationfrom specificationto implementation.Verificationis theprocessof convincingourselvesthatthetransformationwassound.We candivideapproachesto theverificationof systemsinto two broadclasses:(1) axiomatic; and(2) semantic(modelchecking). In the subsectionsthat follow, we shall look at the way in which thesetwo ap-proacheshaveevidencedthemselvesin agent-basedsystems.

5.1 Axiomatic Approaches

Axiomatic approachesto programverificationwerethefirst to enterthemainstreamof com-puterscience,with thework of Hoarein the late1960s[24]. Axiomatic verificationrequiresthat we cantake our concreteprogram,andfrom this programsystematicallyderive a logi-cal theorythat representsthe behaviour of the program.Call this the programtheory. If theprogramtheoryis expressedin thesamelogical languageastheoriginalspecification,thenver-ification reducesto a proof problem:show thatthespecificationis a theoremof (equivalently,is a logical consequenceof) theprogramtheory.

Thedevelopmentof a programtheoryis madefeasibleby axiomatizingtheprogramminglanguagein which the systemis implemented.For example,Hoarelogic givesus moreorlessan axiom for every statementtype in a simple PASCAL-like language.Oncegiven theaxiomatization,theprogramtheorycanbederivedfrom theprogramtext in asystematicway.

Perhapsthemostrelevantwork from mainstreamcomputerscienceis thespecificationandverificationof reactive systemsusingtemporallogic, in theway pioneeredby Pnueli,Manna,

17

and colleagues[40]. The idea is that the computationsof reactive systemsare infinite se-quences,which correspondto modelsfor linear temporallogic4. Temporallogic canbeusedboth to developa systemspecification,andto axiomatizea programminglanguage.This ax-iomatizationcanthenbeusedtosystematicallyderivethetheoryof aprogramfromtheprogramtext. Boththespecificationandtheprogramtheorywill thenbeencodedin temporallogic, andverificationhencebecomesaproofproblemin temporallogic.

Comparatively little work hasbeencarriedout within theagent-basedsystemscommunityonaxiomatizingmulti-agentenvironments.Weshallreview justoneapproach.

CaseStudy: Axiomatizing two Multi-Agent Languages

In [63], an axiomaticapproachto the verificationof multi-agentsystemswasproposed.Es-sentially, theideawasto usea temporalbelief logic to axiomatizethepropertiesof two multi-agentprogramminglanguages.Givensuchanaxiomatization,a programtheoryrepresentingthepropertiesof thesystemcouldbesystematicallyderivedin theway indicatedabove.

A temporalbelieflogic wasusedfor two reasons.First,atemporalcomponentwasrequiredbecause,aswe observed above, we needto capturethe ongoingbehaviour of a multi-agentsystem.A belief componentwasusedbecausetheagentswe wish to verify areeachsymbolicAI systemsin their own right. That is, eachagentis a symbolic reasoningsystem,whichincludesa representationof its environmentanddesiredbehaviour. A belief componentin thelogic allowsusto capturethesymbolicrepresentationspresentwithin eachagent.

Thetwo multi-agentprogramminglanguagesthatwereaxiomatizedin thetemporalbelieflogic wereShoham’s AGENT0 [57], andFisher’sConcurrentMETATEM(seeabove). Thebasicapproachwasasfollows:

1. First,asimpleabstractmodelwasdevelopedof symbolicAI agents.Thismodelcapturesthe fact that agentsaresymbolic reasoningsystems,capableof communication.Themodelgivesan accountof how agentsmight changestate,andwhat a computationofsucha systemmight look like.

2. Thehistoriestracedout in theexecutionof suchasystemwereusedasthesemanticbasisfor a temporalbelief logic. This logic allowsusto expresspropertiesof agentsmodelledat stage(1).

3. The temporalbelief logic wasusedto axiomatizethe propertiesof a multi-agentpro-gramminglanguage.This axiomatizationwasthenusedto developtheprogramtheoryof amulti-agentsystem.

4. The proof theoryof the temporalbelief logic wasusedto verify propertiesof the sys-tem[65].

Note that this approachrelieson the operationof agentsbeingsufficiently simple that theirpropertiescanbeaxiomatizedin thelogic. It worksfor Shoham’s AGENT0 andFisher’s Con-currentMETATEM largelybecausetheselanguageshaveasimplesemantics,closelyrelatedtorule-basedsystems,which in turn have a simplelogical semantics.For morecomplex agents,

4Thesetof all computationsof areactivesystemis atree-likestructure,correspondingtoamodelfor branchingtime temporallogic [11].

18

anaxiomatizationis not sostraightforward. Also, capturingthesemanticsof concurrentexe-cutionof agentsis not easy(it is, of course,anareaof ongoingresearchin computersciencegenerally).

5.2 SemanticApproaches:Model Checking

Ultimately, axiomaticverificationreducesto a proof problem. Axiomatic approachesto ver-ification are thus inherentlylimited by the difficulty of this proof problem. Proofsarehardenough,even in classicallogic; the addition of temporaland modal connectives to a logicmakesthe problemconsiderablyharder. For this reason,moreefficient approachesto verifi-cationhave beensought.Oneparticularlysuccessfulapproachis thatof modelchecking. Asthe namesuggests,whereasaxiomaticapproachesgenerallyrely on syntacticproof, modelcheckingapproachesarebasedon thesemanticsof thespecificationlanguage.

Themodelcheckingproblem,in abstract,is quitesimple:givenaformula � of languageL,anda modelM for L, determinewhetheror not � is valid in M, i.e., whetheror not M L � .Model checking-basedverificationhasbeenstudiedin connectionwith temporallogic [35].Thetechniqueonceagainreliesuponthecloserelationshipbetweenmodelsfor temporallogicand finite-statemachines. Supposethat � is the specificationfor somesystem,and is aprogramthatclaimsto implement� . Then,to determinewhetheror not truly implements� ,weproceedasfollows:

� take , and from it generatea modelM ! that correspondsto , in the sensethat M !encodesall thepossiblecomputationsof ;

� determinewhetheror not M !"#� , i.e., whetherthespecificationformula � is valid inM ! ; theprogram satisfiesthespecification� just in casetheansweris ‘yes’.

The main advantageof modelcheckingover axiomaticverificationis in complexity: modelcheckingusingthe branchingtime temporallogic CTL ([8]) canbe donein polynomialtime(O� $��&%'M (� , where $�� is the sizeof the formula to be checked,and M is the sizeof the

modelagainstwhich � is to bechecked— thenumberof statesit contains)5.

CaseStudy: Model CheckingBDI Systems

In [50], RaoandGeorgeff presentanalgorithmfor modelcheckingAOPsystems.More pre-cisely, they give an algorithmfor taking a logical model for their (propositional)BDI agentspecificationlanguage,anda formulaof thelanguage,anddeterminingwhethertheformulaisvalid in themodel. The techniqueis closelybasedon modelcheckingalgorithmsfor normalmodallogics[21]. They show thatdespitetheinclusionof threeextra modalities,(for beliefs,desires,andintentions),into theCTL branchingtimeframework, thealgorithmis still quiteeffi-cient,runningin polynomialtime. Sothesecondstepof thetwo-stagemodelcheckingprocessdescribedabovecanstill bedoneefficiently. However, it is notclearhow thefirst stepmightberealisedfor BDI logics. Wheredoesthelogical modelcharacterizinganagentactuallycomesfrom — canit bederivedfrom anarbitraryprogram , asin mainstreamcomputerscience?Todo this,we wouldneedto takeaprogramimplementedin, say, PASCAL, andfrom it derive the

5Perhapssurprisingly, modelcheckingfor linear temporallogic is actuallymorecomplex thanfor branchingtime(PSPACE-complete).

19

belief,desire,andintentionaccessibilityrelationsthatareusedto givea semanticsto theBDIcomponentof thelogic. Because,aswenotedearlier, thereis noclearrelationshipbetweentheBDI logic andtheconcretecomputationalmodelsusedto implementagents,it is notclearhowsuchamodelcouldbederived.

5.3 Discussion

Axiomatic approachesto the verificationof multi-agentsystemssuffer from two main prob-lems. First, the temporalverificationof reactive systemsreliesupona simplemodelof con-currency, wheretheactionsthatprogramsperformareassumedto beatomic.Wecannotmakethis assumptionwhenwe move from programsto agents.The actionswe think of agentsasperformingwill generallybemuchmorecoarsegrained.As a result,we needa morerealisticmodelof concurrency. Onepossibility, investigatedin [64], is to modelagentexecutioncyclesasintervalsover the realnumbers,in thestyleof the temporallogic of reals[4]. Thesecondproblemis thedifficulty of theproof problemfor agentspecificationlanguages.As we notedin section3, thetheoreticalcomplexity of proof for many of theselogicsis quitedaunting.

With respectto model-checkingapproaches,the mainproblem,aswe indicatedabove, isagainthe issueof ungroundedsemanticsfor agentspecificationlanguages.If we cannottakeanarbitraryprogramandsay, for thisprogram,whatits beliefs,desires,andintentionsare,thenit is notclearhow wemightverify thatthisprogramsatisfiedaspecificationexpressedin termsof suchconstructs.

6 Conclusions

Agent-basedsystemsarea promisingdevelopment,not just for AI, but for computersciencegenerally. If intelligent agenttechnologysucceeds,then it will provide a solution to manyimportantbut difficult softwareproblems.Thechallengenow beforetheintelligentagentcom-munity is to ensurethatthetechniquesdevelopedparticularlyover thepastdecadefor buildingrationalagentsmakeasmoothtransitionfrom theresearchlabto thedeskof theeverydaycom-puterworker. This is by no meanseasy, astheexpertsystemsexperiencedemonstrates.If thecommunityis to succeedin thisendeavour, thenit will needto takeveryseriouslythecommentby OrenEtzioni, thatopenedthis paper:agentsaremorea problemof computerscienceandsoftwareengineeringthanAI.

In thispaper, wehavesetouta roadmapfor work in agent-basedsoftwareengineering.Wehave examinedthe fundamentalproblemsof specification,implementation,and verificationfrom thepoint of view of agent-basedsystems.Throughout,we have beencarefulto draw asmany parallelsaspossiblewith moremainstreamsoftwareengineering.

Acknowledgments

I would like to thankMichaelFisherfor the(many) discussionswehavehadonsoftwareengi-neeringfor agent-basedsystems,andalsoAdamKellettandNick Jenningsfor theircommentson thispaper. Thiswork wassupportedby theEPSRC undergrantGR/K57282.

20

References

[1] G. Agha. ACTORS:A Model of Concurrent Computationin DistributedSystems. TheMIT Press:Cambridge,MA, 1986.

[2] J.F. Allen. Towardsageneraltheoryof actionandtime.Artificial Intelligence, 23(2):123–154,1984.

[3] H. Barringer, M. Fisher, D. Gabbay, G. Gough,andR. Owens.METATEM: A frameworkfor programmingin temporallogic. In REXWorkshopon StepwiseRefinementof Dis-tributedSystems:Models,Formalisms,Correctness(LNCSVolume430), pages94–129.Springer-Verlag:Heidelberg, Germany, June1989.

[4] H. Barringer, R. Kuiper, andA. Pnueli. A really abstractconcurrentmodelandits tem-poral logic. In Proceedingsof theThirteenthACM Symposiumon thePrinciplesof Pro-grammingLanguages, pages173–183,1986.

[5] N. BelnapandM. Perloff. Seeingto it that: a canonicalform for agentives. Theoria,54:175–199,1988.

[6] A. H. BondandL. Gasser, editors.Readingsin DistributedArtificial Intelligence. MorganKaufmannPublishers:SanMateo,CA, 1988.

[7] B. Chellas. Modal Logic: An Introduction. CambridgeUniversity Press:Cambridge,England,1980.

[8] E.M. ClarkeandE.A. Emerson.Designandsynthesisof synchronizationskeletonsusingbranchingtime temporallogic. In D. Kozen,editor, Logicsof Programs— Proceedings1981(LNCSVolume131), pages52–71.Springer-Verlag:Heidelberg, Germany, 1981.

[9] P. R. CohenandH. J. Levesque.Intentionis choicewith commitment.Artificial Intelli-gence, 42:213–261,1990.

[10] D. C. Dennett.TheIntentionalStance. TheMIT Press:Cambridge,MA, 1987.

[11] E. A. EmersonandJ. Y. Halpern. ‘Sometimes’and‘not never’ revisited: on branchingtimeversuslineartime temporallogic. Journalof theACM, 33(1):151–178,1986.

[12] O.Etzioni. Moving uptheinformationfoodchain:Deploying softbotsontheworld-wideweb. In Proceedingsof the ThirteenthNational Conferenceon Artificial Intelligence(AAAI-96), Portland,OR,1996.

[13] R. Fagin,J. Y. Halpern,Y. Moses,andM. Y. Vardi. ReasoningAboutKnowledge. TheMIT Press:Cambridge,MA, 1995.

[14] M. Fisher. A survey of ConcurrentMETATEM — the languageandits applications.InD. M. GabbayandH. J.Ohlbach,editors,Temporal Logic —Proceedingsof theFirst In-ternationalConference(LNAI Volume827), pages480–505.Springer-Verlag:Heidelberg,Germany, July1994.

[15] M. Fisher. Executabletemporallogic. TheKnowledgeEngineeringReview, 1996.

21

[16] M. FisherandM. Wooldridge.Executabletemporallogic for distributedA.I. In Proceed-ingsof theTwelfthInternationalWorkshoponDistributedArtificial Intelligence(IWDAI-93), pages131–142,HiddenValley, PA, May 1993.

[17] M. R. GeneserethandS. P. Ketchpel. Softwareagents. Communicationsof the ACM,37(7):48–53,July1994.

[18] M. P. Georgeff andA. L. Lansky. Reactivereasoningandplanning.In Proceedingsof theSixthNationalConferenceon Artificial Intelligence(AAAI-87), pages677–682,Seattle,WA, 1987.

[19] R. V. GuhaandD. B. Lenat. Enablingagentsto work together. Communicationsof theACM, 37(7):127–142,July1994.

[20] J.Y. Halpern.Knowledgeandcommonknowledgein adistributedenvironment.Journalof theACM, 37(3),1990.

[21] J.Y. HalpernandM. Y. Vardi. Model checkingversustheoremproving: A manifesto.InV. Lifschitz, editor, AI andMathematicalTheoryof Computation— Papers in Honor ofJohnMcCarthy. AcademicPress,1991.

[22] D. Harel. Dynamiclogic. In D. GabbayandF. Guenther, editors,Handbookof Philo-sophicalLogic VolumeII — Extensionsof ClassicalLogic, pages497–604.D. ReidelPublishingCompany: Dordrecht,TheNetherlands,1984.(Syntheselibrary Volume164).

[23] F. Hayes-Roth,D. A. Waterman,and D. B. Lenat, editors. Building Expert Systems.Addison-Wesley: Reading,MA, 1983.

[24] C. A. R. Hoare.An axiomaticbasisfor computerprogramming.Communicationsof theACM, 12(10):576–583,1969.

[25] C. A. R. Hoare. Communicatingsequentialprocesses.Communicationsof the ACM,21:666–677,1978.

[26] P. Jackson.Introductionto ExpertSystems. Addison-Wesley: Reading,MA, 1986.

[27] N. R. Jennings,J.Corera,I. Laresgoiti,E. H. Mamdani,F. Perriolat,P. Skarek,andL. Z.Varga.UsingARCHONto developreal-world DAI applicationsfor electricitytransporta-tion managementandparticleacceleratorcontrol. IEEEExpert, dec1996.

[28] N. R. JenningsandM. Wooldridge.Applying agenttechnology. AppliedArtificial Intel-ligence, 9(6):357–370,1995.

[29] C. B. Jones. SystematicSoftware DevelopmentusingVDM (secondedition). PrenticeHall, 1990.

[30] L. P. Kaelbling.Learningin EmbeddedSystems. TheMIT Press:Cambridge,MA, 1993.

[31] L. P. Kaelbling and S. J. Rosenschein.Action and planningin embeddedagents. InP. Maes,editor, DesigningAutonomousAgents, pages35–48.TheMIT Press:Cambridge,MA, 1990.

22

[32] D. Kinny, M. Georgeff, andA. Rao. A methodologyandmodellingtechniquefor sys-temsof BDI agents. In W. Van de Veldeand J. W. Perram,editors,AgentsBreakingAway: Proceedingsof theSeventhEuropeanWorkshoponModellingAutonomousAgentsin a Multi-AgentWorld, (LNAI Volume1038), pages56–71.Springer-Verlag:Heidelberg,Germany, 1996.

[33] K. Konolige. A DeductionModel of Belief. PitmanPublishing: LondonandMorganKaufmann:SanMateo,CA, 1986.

[34] H. J.Levesque,P. R.Cohen,andJ.H. T. Nunes.Onactingtogether. In Proceedingsof theEighth National Conferenceon Artificial Intelligence(AAAI-90), pages94–99,Boston,MA, 1990.

[35] O.LichtensteinandA. Pnueli.Checkingthatfinite stateconcurrentprogramssatisfytheirlinearspecification.In Proceedingsof theEleventhACM SymposiumonthePrinciplesofProgrammingLanguages, pages97–107,1984.

[36] M. Luck andM. d’Inverno. A formal framework for agency andautonomy. In Proceed-ings of the First InternationalConferenceon Multi-Agent Systems(ICMAS-95), pages254–260,SanFrancisco,CA, June1995.

[37] M. Luck, N. Griffiths,andM. d’Inverno.Fromagenttheoryto agentconstrution:A casestudy. In Intelligent AgentsIII — Proceedingsof the Third InternationalWorkshoponAgentTheories,Architectures,andLanguages(ATAL-96). Springer-Verlag: Heidelberg,Germany, 1997.

[38] P. Maes. Situatedagentscan have goals. In P. Maes,editor, DesigningAutonomousAgents, pages49–70.TheMIT Press:Cambridge,MA, 1990.

[39] P. Maes. Agentsthat reducework and informationoverload. Communicationsof theACM, 37(7):31–40,July1994.

[40] Z. MannaandA. Pnueli. Temporal Verificationof ReactiveSystems— Safety. Springer-Verlag:Heidelberg, Germany, 1995.

[41] Z. MannaandP. Wolper. Synthesisof communicatingprocessesfrom temporallogicspecifications.ACM TransactionsonProgrammingLanguagesandSystems, 6(1):68–93,January1984.

[42] J. J.Ch.Meyer andR. J.Wieringa,editors.DeonticLogic in ComputerScience— Nor-mativeSystemSpecification. JohnWiley & Sons,1993.

[43] C. Morgan. Programmingfrom Specifications(secondedition). PrenticeHall Interna-tional: HemelHempstead,England,1994.

[44] Inc. NeXT Computer. Object-OrientedProgrammingand the ObjectiveC Language.Addison-Wesley: Reading,MA, 1993.

[45] A. Pnueli.Specificationanddevelopmentof reactivesystems.In InformationProcessing86. Elsevier SciencePublishersB.V.: Amsterdam,TheNetherlands,1986.

23

[46] A. PnueliandR. Rosner. On thesynthesisof a reactive module. In Proceedingsof theSixteenthACM SymposiumonthePrinciplesof ProgrammingLanguages(POPL), pages179–190,January1989.

[47] A. S.RaoandM. Georgeff. BDI Agents:from theoryto practice.In Proceedingsof theFirst InternationalConferenceonMulti-AgentSystems(ICMAS-95), pages312–319,SanFrancisco,CA, June1995.

[48] A. S. RaoandM. P. Georgeff. Modeling rationalagentswithin a BDI-architecture.InR. FikesandE. Sandewall, editors,Proceedingsof Knowledge RepresentationandRea-soning(KR&R-91), pages473–484.MorganKaufmannPublishers:SanMateo,CA, April1991.

[49] A. S. RaoandM. P. Georgeff. An abstractarchitecturefor rationalagents.In C. Rich,W. Swartout,andB. Nebel,editors,Proceedingsof Knowledge RepresentationandRea-soning(KR&R-92), pages439–449,1992.

[50] A. S.RaoandM. P. Georgeff. A model-theoreticapproachto theverificationof situatedreasoningsystems.In Proceedingsof the ThirteenthInternationalJoint ConferenceonArtificial Intelligence(IJCAI-93), pages318–324,Chambery,France,1993.

[51] A. S. RaoandM. P. Georgeff. Formalmodelsanddecisionproceduresfor multi-agentsystems.TechnicalNote61, AustralianAI Institute,Level 6, 171La TrobeStreet,Mel-bourne,Australia,June1995.

[52] S. Rosenscheinand L. P. Kaelbling. The synthesisof digital machineswith provableepistemicproperties. In J. Y. Halpern,editor, Proceedingsof the 1986ConferenceonTheoretical Aspectsof ReasoningAboutKnowledge, pages83–98.MorganKaufmannPublishers:SanMateo,CA, 1986.

[53] S. RussellandP. Norvig. Artificial Intelligence: A ModernApproach. Prentice-Hall,1995.

[54] M. J. Schoppers.Universalplansfor reactive robotsin unpredictableenvironments. InProceedingsof theTenthInternationalJoint ConferenceonArtificial Intelligence(IJCAI-87), pages1039–1046,Milan, Italy, 1987.

[55] K. Segerberg. Bringing it about.Journalof PhilosophicalLogic, 18:327–347,1989.

[56] Y. Shoham.ReasoningAboutChange: TimeandCausationfromtheStandpointof Arti-ficial Intelligence. TheMIT Press:Cambridge,MA, 1988.

[57] Y. Shoham.Agent-orientedprogramming.Artificial Intelligence, 60(1):51–92,1993.

[58] A. P. SistlaandE. M. Clarke. The complexity of propositionallinear temporallogics.Journalof theACM, 32(3):733–749,1985.

[59] R. M. Smullyan.First-OrderLogic. Springer-Verlag:Heidelberg, Germany, 1968.

[60] M. Spivey. TheZ Notation(secondedition). PrenticeHall International:HemelHemp-stead,England,1992.

24

[61] M. Y. Vardi and P. Wolper. Reasoningaboutinfinite computations. InformationandComputation, 115(1):1–37,1994.

[62] P. Wolper. The tableaumethodfor temporallogic: An overview. Logiqueet Analyse,110–111,1985.

[63] M. Wooldridge.TheLogical Modellingof ComputationalMulti-AgentSystems. PhDthe-sis,Departmentof Computation,UMIST, Manchester, UK, October1992. (Also avail-able as TechnicalReportMMU–DOC–94–01,Departmentof Computing,ManchesterMetropolitanUniversity, ChesterSt.,Manchester, UK).

[64] M. Wooldridge. This is MYWORLD: The logic of an agent-orientedtestbedfor DAI.In M. WooldridgeandN. R. Jennings,editors,Intelligent Agents: Theories,Architec-tures,andLanguages(LNAI Volume890), pages160–178.Springer-Verlag:Heidelberg,Germany, January1995.

[65] M. WooldridgeandM. Fisher. A decisionprocedurefor a temporalbelief logic. In D. M.GabbayandH. J. Ohlbach,editors,Temporal Logic — Proceedingsof the First Inter-national Conference(LNAI Volume827), pages317–331.Springer-Verlag: Heidelberg,Germany, July1994.

[66] M. WooldridgeandN. R. Jennings.Intelligentagents:Theoryandpractice.TheKnowl-edgeEngineeringReview, 10(2):115–152,1995.

25