symposium on "the effectiveness of modelling and simulation” msg-031 briefing on progress so...
TRANSCRIPT
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Progress to date on NMSG 031 The Cost Effectiveness of Modelling
and Simulation
A Presentation by Gary Summons (QinetiQ) and Lt Cdr Alastair Macdonald (MOD)
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Introduction/Background:
• The use of modelling and simulation (M&S) has grown rapidly
• Continued growth will depend on M&S being seen as cost effective and adding value.
• Programmes generally make a specific business case for any M&S outlay and there is a need for generic business case guidance.
• By identifying non cost effective application areas, the barriers preventing the proliferation of M&S will be identified providing an indication of future research priorities.
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Aims
• Provide a framework to collect evidence associated with the cost effective application of M&S based upon the NATO procurement process:
– Look at how the use of M&S can be categorised and mapped to the NATO process and what measures of effectiveness should be documented.
– Produce a report discussing the cost effectiveness associated with the exploitation of M&S and providing evidence to support these arguments where it exists.
• This presentation will provide an insight in the work to date that the member countries have undertaken.
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
• Members of the NMSG– Canada– France – Germany– Netherlands– UK– US
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
NATO Procurement Process• In order to place information gathered into a useful context it was decided to use the
procurement process as a framework. – Allows end users to see both the applicability of M&S as well as its Cost Effectiveness and relate it to
their particular issues
• Each Nation has it’s own “flavour” of the procurement process it was therefore agreed to adopt the NATO instantiation and map each individual country’s to it– Each member country produced it’s own mapping following consultation with appropriate home nation
procurement specialist
– While it did not provide a “one to one” mapping for all countries there was a broad alignment
– Key to this activity was an understanding of decision points and outputs required at each stage
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
NATO Procurement Process• Mission need evaluation
• Pre-Feasibility
• Feasibility
• Project definition
• Design
• Production
• In-Service
Mis
sio
n
ne
ed
e
valu
atio
n
Pre
-fe
asi
bili
ty
Fe
asi
bili
ty
Pro
ject
d
efin
itio
n
De
sig
n
Pro
du
ctio
n
In s
erv
ice
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
NATO Procurement Process• Example showing the NATO process with UK’s mapped
Mis
sio
n
ne
ed
e
valu
atio
n
Pre
-fe
asi
bili
ty
Fe
asi
bili
ty
Pro
ject
d
efin
itio
n
De
sig
n
Pro
du
ctio
n
In s
erv
ice
Pre
-co
nce
pt
Ma
nu
fact
ure
De
mo
nst
ratio
n
Ass
ess
me
nt
Co
nce
pt
UK
In s
erv
ice
Dis
po
sal
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Modelling and Simulation (M&S) Classification• A way to provide common classification of M&S use is required• 3 separate views of this were investigated
– Application– Category– Generic use
• These classifications were then combined in a matrix with the NATO procurement process to provide an indication of M&S applicability
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Defence Applications for M&S
• Concept Development– Requirements gathering– Doctrine development (concepts)
• Procurement (Capability) Development– Supporting procurement process– Test and evaluation
• Warfighting Development– Mission rehearsal– Training and education
• Research
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
M&S Categories• Component level modelling • Individual/team virtual environment• Team/collective enviroments• Constructive platform level interactive• Constructive aggregate level interactive• Constructive closed analytical model• LVC (Virtual Simulation with constructive, platform) • Pure visualisation• Infrastructure and servicesThe group adopted this series of M&S categories, in doing so it recognises that there are many other ways this could have been “sliced”
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
NLNL
Mission need evaluation
Pre feasibility Feasibility Project definition Design and development
Production In-service
Component level modelling
Individual/team virtual environment
Team/collective enviroments
Constructive platform level interactive
Costructive aggregate level interactive
Constructive closed analytical model
Virtual Simulation with constructive, platform
Pure visualisation
Infrastructure and services
Example of Procurement Process vs M&S Categories
Key Used often Used occasionally Used infrequently
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Generic Uses
• The group also elected to look at generic uses of M&S• This approach allowed non M&S experts to evaluate the benefits of M&S to the procurement process• It is intended that both this and the M&S categories will point at the same underpinning evidence but will present it in different ways
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Generic Uses• Analysis (of alternative) cost performance trades• (Systems) Engineering• Experimentation and Concept Development• Training • Test• Exercise• Operation
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Mission need evaluation
Pre feasibility Feasibility Project definition Design and development
Production In-service
Analysis (of alternative) cost performance trades
(systems) engineering
Experimentation and Concept Development
Training
Test
Exercise
Operations
Key Good cost savingSome costing saving
Little or no cost saving
Procurement Process vs Generic Uses
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Establishing the case for M&S• We can determine cost effectiveness by weighing the costs associated with M&S against the benefit accrued• However determining the contributing factors to cost is not quite as simple• Cost needs to be considered with the following in mind:
– Sunk costs, those which have already been invested– Costs incurred whether M&S is employed or whether a more traditional approach is used– Costs not incurred because M&S is used– Costs only incurred because M&S is employed
• In some cases the use of M&S will need consideration under one or more elements i.e. the re-use of a model will have an element of sunk but most likely incur a cost associated with it’s new use
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Establishing the case for M&S• Benefits are more easily identified and can be classified as:
– Cost avoidance– Risk management– Quality of end product– Reduction in manpower costs– Reduction in Time– Reduction of Environmental impact– Increased capability– Reduction in High value asset dependency
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Establishing the case for M&S• Once the evidence for cost and benefit has been collected it will be associated with the particular category of M&S, Generic use and Procurement phase• We will inform the factors associated with the cost effective application of M&S• We will make available the evidence gathered in a coherent fashion so that decision makers can make evidence based judgements on the applicability of M&S to their particular situation• Based on this evidence it will be possible to identify areas where the application of M&S is cost effective and those areas it is not
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Evidence tool• To be useful all the evidence gathered needs to be readily accessible to end users and decision makers• A simple tool is required which will both present the existing evidence and aid the collection of new evidence• The tool must be able to present the information from the different views previously described:
– Stages of procurement process
– Different categories of M&S
– Different Generic Uses
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Example of Query in
Evidence tool
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Anticipated Results• Framework for cost effectiveness evaluation and comparison across projects/nations• More qualitative than quantitative evidence• Populated evidence schema• No general rule (one size does not fit all) difficult to generalise from specific examples
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Early Conclusions• Difficulty in collecting and analyzing data
• Quantifiable evidence hard to source
• No magic bullet solution for Business Case
• This work provides a framework for cost effect analysis
• Template for the continued gathering of evidence
• Allows comparison of cost effectiveness across different projects/national boundaries
• Problems identifying ‘sunk’ costs
• Cost of non-M&S approach vs M&S approach i.e. costs are still incurred if M&S not used
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
Anticipated Recommendations• Develop the Evidence Tool schema as a web based solution on the NATO web• Nations to continue populating schema as evidence becomes available• Further work needed on “sunk” vs specific costs• Further work required on Metadata
Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”
MSG-031 Briefing on Progress so far Poland, 13 October 2005
NMSG 031 The Cost Effectiveness of Modelling and Simulation
Gary Summons [email protected] Cdr Alastair Macdonald [email protected]