edmonds & bryson, the insufficiency of formal design methods - the necessity of an experimental...

Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-1

The Insufficiency of Formal Design Methods

- the necessity of an experimental approach for the understanding and control of complex MAS

Bruce EdmondsCentre for Policy ModellingManchester Metropolitan

University

Joanna BrysonDepartment of Computer

Science,University of Bath


Main Themes• The need for a shift of emphasis:

– From verification towards validation– From the “10%” towards the “90%” (in the adage)– From engineering (of new systems) to adaptation (of

existing systems)

• This goes against implicit assumptions (in SE):– Design/engineering is “better” than testing/adaptation– Computational systems are deterministic (in reality)– Since (aspects of) software systems can be

represented as formal objects that formalist approaches will be able to deliver (increased) reliability

• Whilst this shift may be resistible with single, closed systems it will be unavoidable with MAS


Outline of the talk

1. Some formal limitations to formal approaches (why a “Hilbert Programme” for MAS won’t work)

2. Software production strategies(the primary SE goal and strategies to get there)

3. A more “Scientific” approach for SE(utilising the classic experimental method)


Part 1: Some Formal Limitations of Formal Methods

- how simple MAS can beand it still be impossible to bridge the

specification program gap


Formal Approach

Idea is to write specifications in a formal language

Often of a logical or set-theoretic nature

Two undisputed advantages:1. Specification is unambiguous and concise

2. Specifications can be syntactically manipulated (as in proofs, checkers etc.)

Is thus a sort of lingua franca for software engineers

As with any language, there are difficulties that arise when attempting to translate to and from itTo it (from the informal) - the “requirements problem”

From it (to programs) - the “problems” that follow


The Programming Problem Posed

SxA given formal specification

PyAny program that

satisfies Sx

A translating program?

T

Is there a general, effective or systematic method of finding a program that satisfies a given specification?


The Programming Problem Answered

No, if the language of specification is expressive enough


PyAny program that

satisfies Sx

A translating program?

T


The Checking Problem Posed


PyA given program

A checking program?

T

Is there a general, effective or systematic method of checking whether a given program satisfies a given

specification?


The Checking Problem Answered


PyA given program

A checking program?

T

No, if the language of specification is expressive enough


What “expressive enough” means

E.g. Those that are able to express basic arithmetic statements about the behaviour of programs

That is, able to express statements such as:– Output of program Px is always < 100

– Program Px will not finish before time 1000

– Output of Px is always greater than Py

– Output of Px is a approximates that of Py (±10%)

(There are other ways such expressiveness can be established: 2-place predicates etc.)

Thus in SE terms “expressive enough” is, in fact, the minimum necessary for almost all real systems


Proof Sketch (programming problem answer)

Define the “nth limited halting problem”, LHn(x,y), as:Does Px ever halt with input y where both x,yn?

Each LHn(x,y) is computable as a finite lookup table

LHn(x,y) is effectively expressible in a language with arithmetic via the construction in (Gödel 1933) as SHm - that is, m here is computable from n

Now if there were a translating program, T, then: given x and y; let z=max(x,y); compute SHz; use T to find a program to compute LHz(x,y) from SHm; and use this to find whether Px(y) halts; but this is impossible (Turing 1936).


An example: how “simple” a MAS can be and still be beyond formal methods

• Giving Agent System with Plans (GASP)• Fixed number of agents: A1, A2, …An

• Each agent, Ax, has– a single store, Sx

– a fixed number of plans: Px1, Px2, …

• Each Plan, Pxy, is composed of instructions:– A fixed number of “give one to …”– And one final test instruction: If Si is zero then do plan j

next, otherwise plan k next

• Each time click, all do: get 1 unit; use current plan to: [do giving (while they have);

do test instruction; note next plan].


An illustration of a GASP system

Plan 1:G3

G2

JZ2,1,3

Plan 2:

JZ1,2,3

Plan 3:G2

G2

G2

JZ2,3,3

Agent 1

1 2 3

Agent 2

1 2 3

Agent 3

Etc.

Check if zero

4

27Store:


Facts about GASP systems

• They are Turing Complete (see paper for proof outline), hence:

…many questions about their behaviour are (in general) undecidable even when given their program

• In particular, a specification adequate to specifying the behaviour of GASP systems will need basic arithmetic and so…

…the previous results about the (difficulties of the) relationship between an adequate specification language and systems hold


Part 2: Software Production Strategies

- Specification and Design Strategies and their problems


Goals and Strategies for engineering computational (IT) systems

To produce IT systems that work well in practice when working in their operational context

One particular strategy to achieve this is the “formal design strategy” (FDS), summarised as:1. Agree the goals for the system; 2. Write a specification that would meet these goals;3. Implement a system that meets this specification.

• Works well for relatively simple, closed, static and analysable cases

• FDS is obviously a “straw man” but consistent with much rhetoric in MAS and true to the extent that there is an over-emphasis on these stages


Some (well known) causes of problems with the FDS

• Context of operation is (at least partially) unknown to designers

• Good in practice operation requires meaningful, complex and abstract goals,

• Thus, either one has a:High-level specification, in which case you can’t

guarantee that the system works according to its specification

A Low-level specification in which case you can’t guarantee that the specification achieves the goals

Or many staged levels of specification with many chances for errors due to repeated translation


Some kinds of complexity

• Syntactic Complexity– When the computational ‘distance’ between

initial conditions and outcomes is too great to be analytically bridgeable

There are different views of a system

• Semantic Complexity– When any formal representation is necessarily

incomplete Models are context-dependent You probably need many of them


Inseparable system embedding

When the particular system is embedded into a wider system such that…the wider system can not be separated from the particular system to aid analysis…without changing the behaviour of both particular system and the wider system,…so that off-line analysis and design is difficult and ineffective.

When “particular system” = agent (i.e. MAS), such embedding can emerge in a wide variety of kinds of systems and environment making off-line and

formal design strategies inappropriate


Engineered Agent Approach

Beliefs, intentions, etc.

Automatic verification

Ontologies, protocols

Agents, groups, teams

Logics

Roles

Deferred decision making

Testing, model checking

Adaptive Approach

Actor

Simulation

Agents, groups, societies

Social&biological analogies

Duplication, competition

Social&individual learning

Post hoc exploration/experimentation

Some (well known) software production strategies

• Abstraction• Automation• Standardisation• Modularity• Formalisation• Transparency• Redundancy• Adaptivity• Testing


Why such over-emphasis on (the rhetoric of) formal design strategies?

Some guesses:• Illusion that computational systems are

deterministic at macro level in practice because they are in theory at micro level

• Blame is contained to specification system production stages, and hence substantially defrayed from designer to user

• Fixed goals suit management, external consultants and academics

• Used to dealing with simple, closed systems• Engineering is seen as “better” than adaptation


Part 3: An Experimental Approach

- an alternative for messy systems and situations


What if the properties of most MAS are messy - more like Biology than Logic?

• Lots of kinds of agents, teams, trust, communication forms, etc.

• Lots of observation and exploration before any abstraction into theory possible

• A priori foundationalist studies based on plausibility probably worse than useless

• Success coming more from: what works in context, reliability, and adaptation to the unexpected

• And less from: abstraction, modularity and formal analysis


The Classic Experimental Method

• Theory developed empirically (not a priori)• Applied using well-validated processes and tools• How the theory can be applied using what

approximations is empirically established• The conditions under which a theory can be safely

applied (and how) developed over time• Useful properties can only be deduced after

theory has been validated• Has worked in messy systems (in science) where

there is little that can be completely generalised


What this might give us…

• Explicit, relevant and testable models/hypotheses concerning the properties of certain MAS

• With sets of conditions under which it has survived trials/testing (and the extent of success)

• Also a set of situations where the hypotheses failed, to indicate the limits of its applicability

• Giving confidence and guidance to those who wish to use these systems

• From which inferences can be made• A scientific basis on which to build sound

engineering practices (e.g. systematisation)


E.g. Engineering a bridge

• Use of well-validated general designs and strategies (e.g. arches, columns, suspension)

• Multiple approximate calculations (maximum stress, weight, compression)

• Use of well-validated components or components made using well-validated techniques (e.g. standard girders or cable)

• Simulations of the set-up (e.g. oscillations)

…still the unexpected may occur - no illusion that design proof can be used on whole

systems to achieve reliability


A Comparison of Approaches (at present time)

Formal/Inferential• Precise criteria for

success• Well-developed

methodology/tools• Taken from formal

sciences• Limited applicability• For Small components• As a check with

simplified models

Inductive/Experimental• Weak criteria of

success• Methodology/tools

need improving• From natural sciences• Wide applicability• On real systems• No certainty• An eternal task


Strengthening the whole chain of inference about MAS

Target MAS

Formal or Computational Model

Adjustable aspect (AA)

Observable outcome (OO)

Setting or set-up

Inference or results

It is the strength of the whole chain from AA to OO

that matters

Strengthening the inference stage is counterproductive if

this weakens the whole chain


Some practical steps…

• Change the rhetoric and expectations of AOSE• Import well-tried testing techniques from other

sciences, e.g. [Bryson, Lowe & Stein 2000]

• Independently replicating simulation experiments [Axtel et. al. 1996, Edmonds & Hales 2003]

• Extend the open source model to include hypotheses, test results etc. [Edmonds 2004]

• Develop experimental experience and methodology [MABS, MAMABS, ESOA, JASSS]

• Tools for analysis of MAS [Barber & Lam 2004]

• Show how this approach can help achieve robust engineering [Hales & Edmonds 2004]


Conclusion –for messy MAS and MAS environments we am suggesting…

• a move away from a foundationalist approach towards a more empiricist approach…

• with less emphasis on prior verification and more on post hoc validation…

• where reliability is sought from experimentally tested theories of system behaviour…

• (formalisation entering once there is such theory)• which may entail a loss of theoretical certainty…• but offers greater potential for adaptation and

hence in practice performance, including some robustness in uncertain and complex situations


The End

Bruce Edmondsbruce.edmonds.name

Joanna Brysonwww.cs.bath.ac.uk/~jjb

edmonds & bryson, the insufficiency of formal design methods - the necessity of an experimental...

Documents

orgpapersifdm slide

formal language

formal approaches

experimental approach

formal approach idea

formal objects

mas slide

formal specification