Using Agent-based Modelling Approaches to Supportthe Development of Safety Policy for Systems of Systems

Martin Hall-May and Tim Kelly

Department of Computer ScienceUniversity of York, York, YO10 5DD, UK

{martin.hall-may, tim.kelly}@cs.york.ac.uk

Abstract. A safety policy defines the set of rules that governs the safe interac-tion of agents operating together as part of a system of systems (SoS). Agentautonomy can give rise to unpredictable, and potentially undesirable, emergentbehaviour. Deriving rules of safety policy requires an understanding of the ca-pabilities of an agent as well as how its actions affect the environment and con-sequently the actions of others. Methods for multi-agent system design can aidin this understanding. Such approaches mention organisational rules. However,there is little discussion about how they are derived. This paper proposes mod-elling systems according to three viewpoints: an agent viewpoint, a causal view-point and a domain viewpoint. The agent viewpoint captures system capabilitiesand inter-relationships. The causal viewpoint describes the effect an agent’s ac-tions has on its environment as well as inter-agent influences. The domain view-point models assumed properties of the operating environment.

1 Introduction

1.1 Making Systems of Systems Safe

A system of systems (SoS) is a large-scale network of autonomous, heterogeneous, andoften mobile entities that are individually purposeful, and yet are expected to inter-operate towards a common purpose. SoS are, perhaps more than single platform sys-tems, characterised by the interaction of feedback loops. Astonishingly complex be-haviour can arise from the iteration of relatively simple cycles of behaviour. The term‘OODA’ — observe, orient, decide, act — coined by Boyd [1], describes the process ofrelating perceptions of the environment to actions in that same environment, which al-lows systems to interact and operate together in a shared space. It is just this interactionof many autonomously operating cycles of behaviour that can give rise to hazards, andhence to accidents.

The authors explained in [2] that a safety policy can be used to restrict the behaviourof the component systems of a SoS such that hazards are avoided or mitigated throughcorrective action. As described in this paper, a safety policy decomposition proceedsfrom top-level safety objectives down to low-level constraints on system behaviour.However, at each step of the decomposition assumptions are inevitably made that relateto mental models of the system of systems whose behaviour the policy is intended togovern. Making these models explicit is a first step towards allowing the information

that they contain to be used in a systematic policy decomposition process. Developingsuch models is not the principal concern of deriving a safety policy, but is a necessaryprecursor to a successful decomposition.

The informal role of models in policy making is well established [3]. Theoreticalas well as empirical models are used by Government and other decision makers toset policies on a number of issues, ranging from health and the economy to the en-vironment. Indeed, the defence industry uses models to guide combat decisions basedon their knowledge, assumptions and best guesses of enemy capability, the anticipatedoperational environment as well as the configuration and inter-operation of their ownforces.

In order to inform a policy decomposition we must have an understanding of theenvironment in which the systems are expected to operate, the type of knowledge theyemploy in decision-making processes, the capabilities they have and the ways in whichthese are used to interact with one another.

1.2 Learning from Multi-Agent Systems

It has been said that “much confusion still remains about words and phrases for systems-of-systems type problems, let alone the best modeling approaches for dealing withthem” [4]. There is surprisingly little consensus on appropriate modelling techniques.Is it perhaps possible to draw inspiration from related domains whose problem areasshare the characteristics of SoS? Indeed, the community of research concerning the de-sign of intelligent agents would seem to be a rich area with much to contribute to ourarea of research. Multi-agent systems (MAS) deal with the problem of many interactingautonomous agents, each of which may have its own goals and objectives.

Although multi-agent systems would seem germane to the problem area of SoS, it isimportant to recognise key differences. Often the focus of a MAS is on software agents(as opposed to embodied agents) and agents are described as ‘mobile’ only in the sensethat they can move their code between hosts. These software agents act primarily in the‘information’ world, whereas our focus is on agents that can also act in the physicalworld — e.g. an unmanned air vehicle (UAV). It is worth highlighting that agents arestill considered embodied even if they are operating in a simulation of the real world.Contrast, for example, a simulation of a UAV agent with a meeting scheduler agent.

Pynadath and Tambe [5] sum up the agent community’s very different approach tosafety, as the cancellation by an agent of a meeting that a human intended to attendis considered a ‘catastrophic’ event. SoS and MAS share many characteristics, amongthem autonomous entities, local knowledge and decentralised decision-making. How-ever, it is also necessary to recognise the primary distinction between them, namelythat a SoS comprises entities that are capable of physical, not just computational, in-teraction. This capability is arguably the reason that the term SoS is much used in themilitary domain and is key in investigating issues of safety, since physical interaction isa prerequisite for death or injury to occur.

1.3 Structure of the Paper

Section 2 will expand on the notion of a safety policy. Sections 3–5 will introduce threeviewpoints by which a SoS can be modelled. Section 6 will examine how these modelscan be used to inform a safety policy decomposition. Section 7 will summarise thepaper.

Throughout the paper we will illustrate concepts and techniques where possiblewith reference to an example SoS for the military domain. Figure 1 gives a represen-tation of the example SoS, which depicts a minimal set of systems operating togethernecessary to mount an attack and to clear a given region of a guerrilla enemy force. Thesystems communicate via a shared ‘data fusion’ picture, indicated by jagged lines, andinclude a UAV with sensors capable of target identification, long range artillery capableof launching suppressing fire on a target and infantry (carried by transport helicopters)capable of neutralising an already suppressed target.

Fig. 1. A Concept View of a System of Systems for Anti-guerrilla Operations (AGO)

2 Safety Policy

Pynadath and Tambe [5] state that “it is unreasonable (if not impossible) to have humansspecify sufficient safety conditions to completely determine correct agent behaviour”.Quite so, the important distinction is that safety policy specifies constraints that areorthogonal to normal functional behaviour. A safety policy aims to circumscribe thepotentially hazardous but functionally possible behaviour, in such a fashion that it leavesonly that which is considered acceptably safe. It is not the job of safety policy to definefunctionally correct behaviour — for this is merely rigorous specification — policy isseparate from the behaviour to achieve goals.

Sørby [6] describes a safety policy as being analogous to security policy in that itinfluences the stakeholders in a system by enforcing a number of safety requirements,which are in turn influenced by safety standards (see Figure 2). Taken together, the setof safety requirements should ensure the safety of the system.

Safety Policy Stakeholders

Safety

SafetyRequirements

SafetyStandards

ensuresenforces

influences

influences

1..*1

1..*1..*

1 1..*

Fig. 2. Excerpt of Sørby’s Safety Ontology Relating Safety Policy to Safety Requirements

MAS development often mentions organisational rules [7], however there is gener-ally little explanation of how the rules are derived. This paper represents a step towardsa more structured, systematic process to their derivation, which increases traceabilityof rules from high-level objectives and increases confidence in the completeness of therule set. Policy is a hierarchical decomposition of high-level policy objectives into con-straints over agent actions and interactions. The Goal Structuring Notation (GSN) [8]— typically used to construct safety cases — can be used to represent policy decompo-sition structures.

To support policy decomposition, we propose modelling the SoS from three view-points: an agent viewpoint, a domain viewpoint and a causal viewpoint. The agent view-point captures the technical aspects of the SoS and its constituent component systems,including their attributes and relationships in terms of planned interactions. A domainviewpoint provides a consistent terminology and model of the SoS environmental as-sumptions. A causal viewpoint focuses on the way factors such as actions, states andother variables influence each other in the SoS, potentially leading to unplanned inter-actions. These viewpoints are the focus of this paper.

3 Agent Viewpoint

Despite the recent increase in interest in engineering systems of systems [9], there hasbeen little consensus on a successful modelling approach. An object-oriented (OO) ap-proach has been recommended for the specification and analysis of requirements forsystems of systems [10, 11]. Caffall and Michael propose treating the SoS as a singleentity comprising abstract classes, rather than decomposing the SoS into its constituentsystems in a functional fashion.

Agent UML [12] extends the standard Unified Modelling Language typically usedto model OO systems in various ways to enable agent-oriented design. As previouslymentioned, MAS techniques often focus on software agents. Commensurately, AgentUML focuses on capabilities in terms of logical or mathematical operations and servicessuch as ‘computation’, rather than capabilities for physical interaction.

The PASSI (Process for Agent Societies Specification and Implementation) method-ology makes use of conventional and agent UML notation to express five different mod-els of increasing specificity for the design of agent-based societies [13]. The process canbe viewed as comprising analysis and design activities. For the purposes of this paperwe concentrate on the analysis phase, including the modelling of system requirements

and the agent society. Each of the five models has several phases, each of which utilisesa particular (Agent) UML diagram to capture specific information.

implementation level design support, some others not use a standard notation like UML or use a design philosophy that is far from the common experience of the greatest part of software engineers who are usually skilled with object-oriented approaches. Another important aspect that we difficulty found in other methodologies was a specific attention towards code reuse or patterns.

A definition of what is an agent in our approach can be helpful for the explanation of PASSI. We think at an agent as the instance of an agent class that is the software implementation of an autonomous entity capable of pursuing an objective through its autonomous decisions, actions and social relationships. An agent may occupy several functional roles to achieve its goals. A role is the function temporarily assumed by the agent in the society while pursuing a sub-goal. During this activity the agent uses one or more of its tasks. A task is a series of elementary pieces of behavior (actions) necessary for a specific purpose. Each task carries out one of the agent’s decisions/actions/social relationships.

In this definition we can find the concepts of agent, role, task and action. These are some of the elements that will compose our system. In the following we will discuss the way we use to define these component to obtain the desired behavior.

In order to better understand our approach it can be useful to specify that the implementation of each agent will be done using a class derived from the base-agent type of the chosen platform; the tasks will be implemented as subclasses of the agent-class and the actions are methods of these classes. In this perspective, a role will be the result of a series of behaviors realized by the actions of several different tasks of the same agent.

We will illustrate the methodology with an example coming from robotics: we will design a multi-agent system in order to obtain some specific behaviors from a robot provided with video, IR and odometry sensors.

We will illustrate the methodology with an example coming from robotics: we will design a multi-agent system in order to obtain some specific behaviors from a robot provided with video, IR and odometry sensors. The models and phases of PASSI are:

1. System Requirements Model. A model of the system requirements in terms of agency and purpose.

It is composed of four phases: (a) Domain Description (D.D.): A functional description of the system using conventional use-case diagrams. (b) Agent Identification (A.Id.): The phase of attribution of responsibility to agents, represented as stereotyped UML packages. (c) Role Identification (R.Id.): A series of sequence diagrams exploring the responsibilities of each agent through role-specific scenarios. (d) Task Specification (T.Sp.): Specification of the capabilities of each agent with activity diagrams.

2. Agent Society Model. A model of the social interactions and dependencies among the agents involved in the solution. Developing this model involves three steps in addition to part of the previous model: (a) Role Identification (R.Id.): See the System Requirements Model. (b) Ontology Description (O.D.): Use of class diagrams and OCL constraints to describe the knowledge ascribed to individual agents and the pragmatics of their interactions. (c) Role Description (R.D.). Class diagrams are used to show the roles played by agents, the tasks involved, communication capabilities and inter-agent dependencies. (d) Protocol Description (P.D.). Use of sequence diagrams to specify the grammar of each pragmatic communication protocol in terms of speech-act performatives.

3. Agent Implementation Model. A classical model of the solution architecture in terms of classes and methods, the most important difference with common Object-oriented approach is that we have two different levels of abstraction, the social (multi-agent) level and the single-agent level. This model is composed of the following steps: (a) Agent Structure Definition (A.S.D.): Conventional class diagrams describe the structure of solution agent classes. (b) Agent Behavior Description (A.B.D.): Activity diagrams or statecharts describe the behavior of individual agents.

4. Code Model. A model of the solution at the code level requiring the following steps to produce: (a) Generation of code from the model using one of the functionalities of the PASSI add-in. It is possible to generate not only the skeletons but also largely reusable parts of the methods implementation based

Deployment Model

System Requirements Model

TasksSpecification

RolesIdentification

Agent Implementation Model

StructureDefinition

BehaviorDescription

Code Model

CodeProduction

OntologyDescription

RolesDescription

ProtocolsDescription

Agent Society Model

InitialRequirements

AgentsIdentification

DomainDescription

DeploymentConfiguration

Multi-AgentStructureDefinition

BehaviorDescription

Single-Agent

AgentTest

SocietyTest

Next Iteration

Fig. 1. The models and phases of the PASSI methodology

Fig. 3. The Models and Phases of the PASSI Methodology

Figure 3 shows the relationships between the models and phases of PASSI. PASSIencourages identification of agents early on in the development process. This is sup-ported by the claim that the types of MAS targeted by this process comprise agents thatcan be ‘bidden’ or influenced but not deterministically controlled [14]. Therefore, it isdesirable to allocate required behaviours to “loci of responsibility” as soon as possible.

Agent Identification Diagram

TheatreCommand<<Agent>>

Artillery<<Agent>>

Infantry<<Agent>>

UAV<<Agent>>

Helicopter<<Agent>>

Scan Ground(from UAV)

Scan Air(from UAV)

Environment(from 01-Domain D...)

Patrol Area(from UAV)

<<include>>

<<include>>

Suppress Enemy(from Artillery)

User(from 01-Domain D...)

Detect Enemy(from UAV)

<<include>>

<<include>>

Deliver Artillery Fire(from Artillery)

Secure Area(from TheatreCommand)

<<communicate>>

<<communicate>>

Target(from 01-Domain D...)

Transport Special Forces(from Helicopter)

Neutralise Enemy(from Infantry)

<<communicate>>

Deploy Special Forces(from Helicopter)

<<include>>

<<communicate>>

<<include>>

Share Intelligence(from TheatreCommand)

<<include>>

File: H:\rose\AGO2.mdl 08:53:47 03 March 2006 Use Case Diagram: 02-Agent Identification phase / Agent Identification Page 1 (1, 1)

Fig. 4. An Agent Identification Diagram for the AGO System of Systems

Using the PASSI methodology the example SoS configured for anti-guerilla opera-tions and presented in Figure 1 has been modelled. Figure 4 shows a description of theSoS in terms of use cases. The use cases represent the domain of functionality of the

SoS, and external actors have been identified with which the SoS interacts. These arethe user of the SoS, which is most likely to be a set of commands coming from outsideof the theatre of interest, the environment and the enemy targets. Delineating what isinside the system boundary is a significant challenge even for a single platform, andrespectively more difficult for a SoS [15].

In order to secure the area, the enemy must be detected (distinguished from the en-vironment), suppressed and finally neutralised altogether. Due to the large geographicalnature of the area of interest, more than one system is needed to secure it. Due to the na-ture of SoS, i.e. the systems are geographically dispersed, but can only act and observelocally, there is also a need to share intelligence between them. The agent identificationdiagram is derived from a higher level domain description diagram, which is not shownhere due to space constraints. An agent is described as a package of use cases. The usecases that make up the functionality of the SoS have been assigned as the responsibilityof one of five agents: artillery, infantry, UAV, helicopter or theatre command.

: User Controller : TheatreCom... Nominator : UAV Suppressor : Artill... Transporter : Helic... Confirmer : UAV Neutraliser : Infantry : Env... : Tar...Fuser : TheatreCommand

: User : UserController :

TheatreCommandController :

TheatreCommandNominator :

UAVNominator :

UAVSuppressor :

ArtillerySuppressor :

ArtilleryTransporter :

HelicopterTransporter :

HelicopterConfirmer :

UAVConfirmer :

UAVNeutraliser :

InfantryNeutraliser :

Infantry : Environment : Environment : Target : TargetFuser :

TheatreCommandFuser :

TheatreCommand

1: StartMission

2: OrderPatrolArea

3: PatrolArea

4: Update(targetPos)

5: QueryTargets

6: Target(pos)

7: FireShell

8: Update(targetSuppressed)

9: QueryTargets

10: TargetSuppressed(pos)

11: QueryTargets

12: Target(pos)

13: OrderTakeOff

14: Destroy

15: Update(targetNeutralised)

16: QueryTargets

17: TargetNeutralised(pos)

18: QueryTargets

19: NoTargesRemaining

20: MissionComplete

File: H:\rose\AGO2.mdl 08:55:58 03 March 2006 Sequence Diagram: 03-Role Identification phase / Anti-guerrilla Page 1

Fig. 5. A Role Identification Sequence Diagram for the AGO System of Systems

Figure 5 shows the role identification diagram, which describes an envisaged sce-nario. This diagram adds detail to the «communicate» relationships between agents inFigure 4 by identifying the role that agents play and the sequence in which messagesoccur. The role identification diagram is abstract, in that it does not show the reality ofthe interactions. An agent viewpoint is important when developing policy in order toconsider the types of communications that will occur as well as which agent relies onthe services or knowledge of another.

4 Domain Viewpoint

The PASSI methodology employs a knowledge-centric agent model. Agents act toachieve their objectives on the basis of local goals and knowledge, which increasesthrough communication with other agents and exploration of the real world. It is inter-esting to note that PASSI eschews the use of Agent UML’s extension to the class dia-gram in favour of the conventional notation for representing agents. Instead, knowledgeis represented in the domain ontology description phase as a conventional class dia-gram, where each class represents a concept, a predicate or action on that concept. Thecommunication ontology description diagram shows the agent and its ascribed knowl-edge attributes and the communicative relationships between them.

Communication among agents is considered different to that between agents andexternal ‘actors’. Principally, agent communication proceeds according to certain pro-tocols and uses a specific ontology. Communication with agents not part of the SoS(e.g. the enemy, non-squad troops) might occur indirectly through sensing devices.

The representation of knowledge is very important to mental models, and there-fore highly relevant to our particular application (since inconsistencies between agents’mental states are a significant factor in the cause of SoS accidents). Hence, even thoughPASSI allows us to model knowledge using class diagrams, it is worthwhile treating theconstruction of the domain viewpoint as a significant exercise in its own right.

An ontology represents knowledge of what exists — from the Greek ontos (thatwhich exists) + logos (knowledge of) — and is important in order to disambiguateconcepts in communications between agents. When one agent talks to another aboutan ostensibly common concept, they might have very different mental images of whatis being discussed. For instance, consider the potential for confusion when discussingthe many definitions of the word ‘target’. However, the confusion may be even moresubtle, as was the case with the mix-up between imperial and metric units that allowedthe Mars Climate Orbiter to fly too close to the planet’s surface and thus be destroyed[16].

Ontologies are used in domain modelling, conceptual modelling and knowledgeengineering. Aside from providing a common understanding and vocabulary, they canbe used to give meaning and potentially a taxonomical organisation to domain termsprovided by a subject-matter expert (SME). According to Guarino and Welty [17], “on-tologies are becoming increasingly popular in practice, but a principled methodologyfor building them is still lacking.” Indeed, there are many ways in which one mightbuild an ontology from a data dictionary as provided by a SME. What is needed is anorganising principle with which to structure the ontology.

Much work has gone into creating so-called upper ontologies. These include Cyc[18], SUMO [19] and SENSUS [20], which describe high-level terms, under whichdomain-specific ontologies can be organised. The upper ontology typically includesconcepts such as physical, tangible and abstract entities. A mid-level ontology mightinclude specialised versions of these concepts to do with time and space. Finally, a do-main ontology is geared towards the particular application of interest. Valente et al [21]suggest for a military application the use of several ontologies including terms encom-passing Communications, Organisations, Physical resources and Service descriptions.

Domain Ontology Description Diagram

SARSensorDataSensorValue[] : i...

<<concept>>

SelfDescription<<concept>>

SelfPositionXY

<<concept>>

WorldDescription<<concept>>

11

EnvironmentDescription<<concept>>

11

isTargetvalue : Boolea...

<<predicate>>

GeodeticCoordinateSystemlatitudelongitude

<<concept>>

TargetDescription<<concept>>

0..n0..n

isSuppressedvalue : Boolea...

<<predicate>>isNeutralisedvalue : Boolea...

<<predicate>>

TargetPositionXY

<<concept>>

GetTargetInfo

Get()

<<action>>

TheatrePictureTargets[] : TargetPositi...

<<concept>>

UpdateTheatrePicture

Update()

<<action>>

File: H:\rose\AGO.mdl 10:22:34 03 March 2006 Class Diagram: Domain / Domain Ontology Description Page 1

Fig. 6. A Domain Ontology Diagram for the AGO System of Systems

Figure 6 shows the domain ontology diagram for the AGO system. This is necessar-ily simplified and could potentially be extended to include the standard terms providedby any of the above-mentioned upper ontologies. Accompanied by the communica-tions ontology diagram, which ascribes ontologies to agents as concepts to be used incommunications, this gives an indication of how the misinterpretation of common real-world artifacts in local mental models can occur.

5 Causal Viewpoint

The causal viewpoint recognises that accidents in a SoS arise out of, as Perrow de-scribed, dysfunctional interactions [22]. The accident model, STAMP [23], recognisesthe importance of this type of interaction in safety-critical applications. Similarly, wemust take a more systems-theoretic approach to describing the relationships betweencausal factors in the lead up to an accident, rather than traditional chain-of-event failuremodels such as Fault Trees. As described, the behaviour of a SoS is typified by multipleinteracting feedback loops. This means that it is not possible to take a mechanical ap-proach to working through the causal chain, because many factors influence each otheras well as, indirectly, themselves.

The inter- and intra-agent behaviour can be described as a feedback-based loop “ob-serve, orient, decide, act” (OODA). Orient means updating one’s mental model basedon new observations, where these are interpreted through experience, training, tradi-tions, previous observations etc. The interpreted observations allow the agent to decideamong a number of alternatives and to enact the chosen one, causing some effect on theenvironment.

A consideration of the OODA cycle of agent behaviour can help in structuring thetask specification diagram for individual agents. In Figure 7 the UAV goes through acycle of scanning the ground for threats (observation), processing the data (orientation),

UAVUAV T.Sp.:Interacting Agents

Theatre Command.UpdateTheatrePicture

ScanGround

IdentifyTarget ObserveTargetSuppressed

ObserveTargetNeutralised

SendTargetUpdate

File: H:\rose\AGO.mdl 09:14:32 24 February 2006 Activity Diagram: UAV / Tasks Specification Page 1

Fig. 7. A Task Specification Diagram for the UAV Agent

deciding whether the data represents a new threat or a change to the state of an existingknown target (decision) and finally passing this information to Theatre Command (ac-tion). It is only during the observation and action activities that interaction with otheragents can occur. When observing, the agent not only uses its own senses to investigatethe environment but can also integrate information provided by other agents. Similarly,it can provide the results of its decision-making process to other agents through itsactions.

Figure 8 shows several loops for the Theatre Command agent, whose behaviour ismore complicated than the UAV and which interacts with (influences) more agents inso doing. The sequence diagram (Figure 5) describing the interaction between roles inthe AGO scenario must be consistent with the task specification diagrams. That is, thesequence of events described must be able to be generated from the interaction of allthe agents’ OODA cycles.

It is clear, however, that the task specification models do not capture all the un-planned interactions that are required when considering how to develop a safety pol-icy. Planned actions are those that it is anticipated the agent will undertake in orderto complete its task. We want to model agents with only local abilities and imperfectknowledge, i.e. they cannot assess the current state of the world, nor observe the effectsof their actions on all other agents. The causal viewpoint needs to capture problems offailure, uncertainty and the effects of an agent’s actions. For this we can take inspirationfrom another branch of agent theory, namely Multi-agent Influence Diagrams (MAIDs)[24].

MAIDs are an extension to Bayesian Belief Networks (BBNs) and decision net-works. Using them it is possible to represent how agents’ decisions are influenced byvarious factors. These factors include probabilistic variables (represented by circular‘chance’ nodes) that are in effect ‘decided’ by the environment, as well as the results ofother agents’ decisions (rectangular nodes). Decision networks also require specifyingthe utility of the decisions to the agents, i.e. their preference for the result of a particulardecision (represented by diamond-shaped nodes). This is not an aspect of MAIDs thatis of immediate application, since we do not intend to ‘solve’ the MAID, i.e. calculatewhich combination of decisions results in the greatest utility for all the agents. Rather,we are simply using the notation to represent which variables influence which other

TheatreCommandTheatreCommand T.Sp.:Interacting Agents

User.StartMission<<External Agent>>

UAV.ScanGround

Gun.QueryTargets

Infantry.QueryTargets

UAV.SendTargetUpdate

Gun.ReceiveTargetInfo

Infantry.ReceiveTargetInfo

User.EndMission<<External Agent>>

ReceiveMissionOrders

OrderPatrolArea

ReceiveTargetQuery

SendTargetInformation

UpdateTheatrePicture

TargetsRemainNoTargets

SendMissionComplete

File: H:\rose\AGO.mdl 14:16:50 27 February 2006 Activity Diagram: Mission Control / Tasks Specification Page 1

Fig. 8. A Task Specification Diagram for the Theatre Command Agent

variables and whether these variables are determined by chance or are under the controlof an intelligent agent. In terms of the PASSI model, chance nodes belong to actors, i.e.the enemy, the environment, whereas decision nodes belong to the agents in the system.

The construction of a MAID also relies on a consideration of the OODA loop asit relates decisions (the choice between a number of actions) to the observations thatthe agent has made, i.e. the information it has available to it at the time of making thedecision. Observations can be the value of probabilistic variables (the enemy has beendestroyed) or actions taken by other agents, either directly or indirectly observed. Anexample of a direct observation might be that the artillery immediately observes thedecision of the helicopters to take off. In reality, agents have limited knowledge and canonly make localised observations, hence an indirect observation such as the UAV report-ing that the helicopters have taken off is more likely. This observation is then subject toother factors, such as the probability of the failure of the UAV to send the message, themessage getting lost or corrupted by the network, or even being misinterpreted by therecipient. All these factors combine to form the mental picture of the agent (in this casethe artillery) and influence how close this picture approximates reality.

Figure 9 shows an example MAID for modelling of the artillery’s decision to launchsuppressing fire upon a given target. From this we can see that this decision is ultimatelydependent on the result of the UAV’s reconnaissance. However, there are many addi-tional factors on which the artillery’s action (or inaction) is based. These result fromthe fact that the artillery cannot directly observe the UAV’s decision, but rather rely on

NominateTarget

(from UAV)

EnemyStatus

(t)

UAVSensor

(t)

SensorStatus

(t)

CommsStatus

(t)

UAVSensor(t+1)

SensorStatus(t+1)

CommsStatus(t+1)

LaunchAttack

(from Artillery)

EnemyStatus(t+1)

GunAccuracy

(t+1)

EnemyStatus(t-1)

UAVSensor

(t-1)

SensorStatus(t-1)

DataFusion

(t-1)

CommsStatus(t-1)

Win

Cost(from

Artillery)

Cost(from UAV)

Key

UAV Gun Heli

DataFusion

(t)

DataFusion(t+1)

Fig. 9. A Multi-agent Influence Diagram for the AGO System of Systems

the data fusion picture, which is affected by the previous state of the theatre picture,the state of the communications network, the UAV’s sensor, and so on. Even in thissimplified model, the number of influences are numerous.

The application of MAIDs must be judicious because they entail some rather lim-iting assumptions. The common prior assumption considers that two agents that havea common set of prior observations will make the same decision. The assumption ofperfect recall does not allow for an agent to forget any observation that it has made andassumes it has all this evidence available to it when making a decision. It is importantin MAIDs to explicitly represent the informational links to decision nodes, because theassumption of perfect recall only applies to each individual agent. Any particular agentmay not necessarily have made the same observations as other agents, nor be able toobserve the results of other agents’ decisions. Therefore, informational links must beexplicitly added to the graph as dashed lines.

6 Using Models in Policy Decomposition

Models serve two purposes in policy decomposition. Firstly, they aid in decomposingsafety goals by, together with patterns of decomposition, providing the policy-makerwith factors that should be considered in the achievement of the top-level goal. Sec-ondly, the models provide a vocabulary for the expression of these goals. In this way,templates can be created that are more structured than the “verb-phrase noun-phrase”of traditional safety case goal statements.

6.1 Example 1

From Figure 4, it can be seen that securing the area involves both suppressing and neu-tralising the enemy. The former use case includes functionality that has been assignedto the artillery agent, which is capable of delivering suppressing fire onto the target.The latter use case includes functionality that has been assigned to both the helicopterand infantry agents, which brings them into proximity of the target. It is also obviousthat, although the helicopter and infantry communicate their efforts, there is no com-munication with the artillery. The hazard therefore exists that the infantry may be in thevicinity of the target at the same time as the artillery fires upon it. Given that no directcommunication is possible between the two agents (nor indeed necessary to achievethe aims of the mission), a safety policy must be derived to make the artillery awareof the helicopter’s movements. This leads to the policy decomposition shown in Figure10 (represented in GSN). The left-hand side of the policy structure concentrates on in-forming the artillery of the infantry locations so as to avoid accidental attack, whereasthe right-hand side focuses on informing the infantry to avoid an area designated as atarget for the artillery.

Avoid Friendly Attack

Artillery must avoidattacking friendlyagents

Friendly Agents

Decompose overfriendly agents

Avoid Infantry Attack

Artillery must avoidattacking infantry

Shared target

Infantry transportedclose to artillery targetby helicopter

Location of Agent

Decompose overknowledge of infantrylocation

Avoid Known FriendlyLocationsArtillery must avoidattacking infantry at knownlocation

Avoid Unknown FriendlyLocationsArtillery must avoidattacking infantry atunknown location

Provide Location

Infantry must update theatrecommand with currentlocation at least every tenminutes

Artillery Avoid Attack OnLocationArtillery must avoid attackon last known infantrylocation in shared picture

Shared Picture

Communication betweeninfantry and artillery is viashared picture held bytheatre command

Artillery Alert Attack

Artillery must alert theatrecommand at least ten minutesbefore launching an attack

Infantry Avoid ArtilleryTargetInfantry must avoid areas thatare designated as currenttarget for the artillery in theshared picture

Fig. 10. An Excerpt from the Safety Policy Decomposition for the AGO SoS

6.2 Example 2

In understanding how an agent might misinterpret its environment we need to know ad-ditional properties about the domain in which it operates. The domain ontology (Figure6) allows us to consider the possibility of ontology mismatches between agents whencommunicating. In this instance, the common ontology revolves round the location andstatus of a target. Due to the critical nature of these concepts, the policy governing theagents interactions should make sure that there is no chance of a mismatch betweenmental models, e.g. the artillery fires on a target that has already been suppressed, orthe helicopters transport troops to an incorrect location due to a misunderstanding inthe way target locations are represented.

6.3 Example 3

The causal model (Figure 9) prompts us to consider the other factors that influence theobservations that decide an agent’s actions. In this case, the artillery’s decision to launchis not directly dependent on the UAV’s observations, since many other factors outsideof the agent’s control are also involved. This must lead us to consider some level ofcorroboration or cross-checking of the targets nominated by the UAV before the long-range artillery fires blind. Unfortunately, space restrictions do not allow the inclusionof the policy decomposition structure that governs this behaviour.

7 Summary

Safety policy is the rules that govern safe interaction of systems operating as part ofa SoS. Decomposing safety policy objectives into rules that individual systems canimplement is a difficult task. In this paper we have taken inspiration from agent-basedtechniques for modelling the SoS in order to support the systematic decompositionof safety policy. We suggest modelling the SoS from three viewpoints, namely agent,domain and causal viewpoints. It can be seen that none of the viewpoints in isolationcan be used to complete the safety policy decomposition. Each is important in revealingdifferent aspects of the SoS being modelled. The aim of this work is now to identifyand explicitly define links between the safety policy decomposition and elements of thesystem models.

8 Acknowledgement

This work is carried out under the High Integrity Real Time Systems Defence andAerospace Research Partnership (HIRTS DARP), funded by the MoD, DTI and EPSRC.The current members of the HIRTS DARP are BAE SYSTEMS, QinetiQ, Rolls-RoycePlc and the University of York.

References

1. Boyd, J.R.: A discourse on winning and losing. Unpublished briefing, Air University Library,Maxwell AFB, Alabama, Report No. MU43947 (1987)

2. Hall-May, M., Kelly, T.P.: Defining and decomposing safety policy for systems of systems.In: Proceedings of the 24th International Conference on Computer Safety, Reliability andSecurity (SAFECOMP ’05). Volume 3688 of LNCS., Fredrikstad, Norway, Springer-Verlag(2005) 37–51

3. Weinstein, M.C., Toy, E.L., Sandberg, E.A., Neumann, P.J., Evans, J.S., Kuntz, K.M., Gra-ham, J.D., Hammitt, J.K.: Modeling for health care and other policy decisions: Uses, roles,and validity. Value Health 4(5) (2001) 348–61

4. DeLaurentis, D.A., Callaway, R.K.: A system-of-systems perspective for future public pol-icy. Review of Policy Research 21(6) (2004)

5. Pynadath, D.V., Tambe, M.: Revisiting asimov’s first law: A response to the call to arms.In: Proceedings of the 8th International Workshop on Agent Theories, Architectures, andLanguages (Intelligent Agents VII). Volume 2333 of LNCS., Seattle, WA, Springer-VerlagGmbH (2001) 307–320

6. Sørby, K.: Relationship between security and safety in a security-safety critical system:Safety consequences of security threats. Masters thesis, Norges Teknisk-NaturvitenskapeligeUniversitet, Trondheim, Norway (2003)

7. Zambonelli, F., Jennings, N., Wooldridge, M.: Organizational rules as an abstraction for theanalysis and design of multi-agent systems. Journal of Knowledge and Software Engineering11(3) (2001) 303–328

8. Kelly, T.P.: Arguing Safety—A Systematic Approach to Managing Safety Cases. DPhilthesis, University of York, Heslington, York, YO10 5DD, UK (1998)

9. Keating, C., Rogers, R., Unal, R., Dryer, D., Sousa-Poza, A., Safford, R., Peterson, W.,Rabadi, G.: System of systems engineering. Engineering Management Journal 15(3) (2003)36–45

10. Caffall, D.S., Michael, J.B.: System-of-systems design from an object-oriented paradigm.In: Proceedings of the Monterey Workshop: Radical Innovations of Software and SystemsEngineering in the Future, Venice, Italy, U.S. Army Research Office (2002) 146–157

11. Pfaender, H., DeLaurentis, D., Mavris, D.: An object-oriented approach for conceptualdesign exploration of UAV-based system-of-systems. In: Proceedings of 2nd AIAA “Un-manned Unlimited” Conference. Volume 2003-6521 of AIAA., San Diego, CA (2003)

12. Bauer, B., Müller, J.P., Odell, J.: Agent UML: A formalism for specifying multiagent soft-ware systems. In: Proceedings of the 1st International Workshop on Agent-Oriented Soft-ware Engineering. Volume 1957 of LNCS., Limerick, Ireland, Springer-Verlag (2000) 91–104

13. Cossentino, M., Potts, C.: PASSI: A process for specifying and implementing multi-agentsystems using UML. (2002)

14. Jackson, M.: Problem Frames. Addison Wesley, Wokingham, England (2001)15. Alexander, R., Hall-May, M., Despotou, G., Kelly, T.: Towards using simulation to evaluate

safety policy for systems of systems. In: Proceedings of the 2nd International Workshopon Safety and Security in Multi-Agent Systems (SASEMAS ’05), Utrecht, The Netherlands(2005) 5–21

16. Stephenson, A.: Mars climate orbiter mishap investigation board: Phase i report. Technicalreport, NASA (1999)

17. Guarino, N., Welty, C.A.: A formal ontology of properties. In: Proceedings of the 12th Eu-ropean Workshop on Knowledge Acquisition, Modeling and Management, Springer-Verlag(2000) 97–112

18. Guha, R.V., Lenat, D.B.: Cyc: A midterm report. AI Magazine 11 (1990) 32–5919. Niles, I., Pease, A.: Towards a standard upper ontology. In: Proceedings of the International

Conference on Formal Ontology in Information Systems, ACM Press (2001) 2–920. Swartout, B., Patil, R., Knight, K., Russ, T.: Toward distributed use of large-scale ontolo-

gies. In: Proceedings of the 10th Knowledge Acquisition for Knowledge-Based SystemsWorkshop, Banff, Alberta, Canada (1996)

21. Valente, A., Holmes, D., Alvidrez, F.C.: Using ontologies to build web service-based archi-tecture for airspace systems. In: Proceedings of the 8th International Protégé Conference.(2005)

22. Perrow, C.: Normal Accidents: Living with High-Risk Technologies. Princeton UniversityPress (1999)

23. Leveson, N.G.: A new accident model for engineering safer systems. Safety Science 42(4)(2004)

24. Koller, D., Milch, B.: Structured models for multi-agent interactions. In: Proceedings of the8th conference on Theoretical Aspects of Rationality and Knowledge, Siena, Italy, MorganKaufmann Publishers Inc. (2001) 233–248