devs agents to support conformance testing of emerging defense information standards
DESCRIPTION
DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards. Bernard P. Zeigler, Arizona Center for Integrative Modeling and Simulation Tucson Arizona www.acims.arizona.edu and Joint Interoperability Test Command (JITC) Fort Huachuca, Arizona. Outline. - PowerPoint PPT PresentationTRANSCRIPT
DEVS Agents to Support Conformance Testing of Emerging
Defense Information Standards
Bernard P. Zeigler,Arizona Center for Integrative Modeling and Simulation
Tucson Arizonawww.acims.arizona.edu
andJoint Interoperability Test Command
(JITC)Fort Huachuca, Arizona
Outline• “Agent-supported simulation deals with the use of agents as a support
facility to enable computer assistance in problem solving or enhancing cognitive capabilities.” (ADS’06 definition)
• The agent-supported simulation metaphor applies to testing the conformance of multi-agent systems to complex defense information standards
• Structure control agents induce structural change in themselves or others to effectuate different behaviors under different circumstances
• Structure control implemented in Dynamic Structure DEVS enables automation of standards conformance testing
• DEVS formalism is capable of capturing the information-processing complexities, including the dynamics, underlying the MIL-STD-6016C standard
• DEVS modeling and simulation methodology is attaining core-technology for automated testing of military tactical data link standards
Background: Simulation-based development implies the need for simulation-based testing
-- distributed simulation allows testing a system that is first formulated as an abstract model -- both the System under Test (SUT) and the test device are coupled by an common interface
Network
Test Device
send/receive messages
System Under Test (SUT)
send/receivemessages
Raises the question: How to develop the Test Device in an authoritative manner?
Specified as abstract
model, e.g. UML
Connecting middle ware, e.g. HLA
Problem: Conflicting Requirements
To deal with the increasing complexity and advanced decision capabilities of C4ISR systems
=>testing methodology has to become more rigorous, in-depth and thorough
To keep up with the rapid change and short development life cycles expected from the system builders
=> tests have to be ready to conduct in time scales compatible with the agile development strategies of new systems.
Solution: employ DEVS-based M&S
• to increase capabilities for simulation-based testing and •as a basis to increase the automation of testing processes.
Testing of interface standards is a focus area for automated simulation-based testing.
Link-16 is required in all Joint and multi-national operations.
The Joint Interoperability Test Command (JITC) is developing an automated test generation methodology as its core technology for testing conformance of systems to Link-16 This methodology is fundamentally enabled by the DEVS formalized modeling and simulation approach
AWACS
TheaterWarning
ABL
DSP/SBIRS
F-15
JLENS
THAAD
PATRIOT
MEADS
ATACMSAVENGER
TEL
AEGIS (CEP)
SIS(MSCS)SIS(MSCS) Link-16specification
DEVS Representation of Link-16
Constraints(Exception)Rules
Stop
Modify C2Record for TN
1 2 3
RuleProcessing
Stop, Do Nothing,Alerts, Or jump to other
Transaction
TrackDisplay
Operatordecisions
Validity checking
TransmitMsg
Other ConsequentProcessing
Jumps (stimuli) to other
Transactions of specification
Transaction Level - example P.1.2 = Drop Track Transmit
Preparation Processing
Timeouts
PeriodicMsg
Input to
systemDEVS
Output from
system
t1
t2 t
3t4
Level
3 CoupledSystem
2 I/O System
1 I/O Function
0 I/O Frame
System Theory Provides Levels of Structure/Behavior
The JITC employed such testing for the initial major milestone evaluation of the Integrated Architecture Behavior Model (IABM) developed by the Joint Single Integrated Air Picture (SIAP) System Engineering Organization (JSSEO) in 2005.
The test exercise produced significant results that uncovered flaws in the model design and added acknowledged value to the model development.
Recent Successful Application
Types of Distributed Simulation Testing
Testing Description Example
standards conformance test whether system conforms to standardsupports interoperability
Tactical Data LinkStandards, Link-16, VMF, USMTF,
interoperability test whether systems can interoperate (at the syntactic, higher, levels)
Joint Translator Forwarder (JXF) – air to/from land exchange of tactical data
mission/capabilities test whether system of systems have capabilities required for mission
Joint Close Air Support (JCAS)
Test Driver controls the scenario
Multiplatform Distributed Simulation – controlled testing
Platform(System,
Component)
Platform(System,
Component)
Platform(System,
Component)
Test Driver
Distributed Observers look for opportunities to test
Multiplatform Distributed Simulation - uncontrolled testing
Platform(System,
Component)
Platform(System,
Component)
Platform(System,
Component)
Observer Observer Observer
Test Coordinator
Test Driver for Controlled Testing
Middleware
Coupled Test ModelCoupled Test Model
Jx1,data1Jx2,data2Jx3,data3
Jx4,data4 Jx1,data1Jx2,data2Jx3,data3
Jx4,data4 Jx1,data1Jx2,data2Jx3,data3
Jx4,data4
Component Test Model
1
Component Test Model
2
Component Test Model
3
SUT
Test Model Generation for Controlled Testing
Mirroring (flipping) the transactions of a SUT model (system model behavior selected as a test case) allows automated creation of a test model
holdSend(Jx1,data1,t1) holdSend (Jx2,data2,t2)
holdSend (Jx3,data3,t3) waitReceive(Jx4,data4)
receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)
receiveAndProcess(Jx3,data3) transmit(Jx4,data4)
Jx1,data1Jx2,data2Jx3,data3
Jx4,data4
t1 t2 t3 t4time
Test ModelTest Model
SUT Model
Test Manager for Opportunistic Testing• Replace Test Models by Test Detectors• Deploy Test Detectors in parallel, fed by the Observer• Test Detector activates a test when its conditions are met• Test results are sent to a Collector for further processing
Jx1,data1Jx2,data2Jx3,data3Jx4,data4
Test Detector 1
Test Detector 2
Test Detector 3
SUO ObserverOtherFederates
ResultsCollector
Test Manager
The Test Detector watches for the arrival of the given subsequence of messages to the SUO and then watches for the corresponding system output
• Define a new primitive, processDetect, that replaces holdSend• Test Detector
– Tries to match the initial subsequence of messages received by the SUO– When the initial subsequence is successfully matched, it enables waitReceive (or
waitNotReceive) to complete the test
processDetect(Jx1,data1,t1) processDetect(Jx2,data2,t2)
processDetect(Jx3,data3,t3) waitReceive(Jx4,data4)
receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)
receiveAndProcess(Jx3,data3) transmit(Jx4,data4)
Jx1,data1Jx2,data2Jx3,data3
Jx4,data4
t1 t2 t3 t4time
Test Test DetectorDetector
SUO
Test Detector Generation for Opportunistic Testing
Observer & System Under Observation (SUO)
System(e.g. DEVS)
inports outports
ObserverForSystem
inports outports
Observeroutports
Tap into inputs and outputs of SUO
Gather input/output dataand forward for testing
Example: Joint Close Air Support (JCAS) Scenario
Natural Language Specification
JTAC works with ODA!JTAC is supported by a Predator!JTAC requests ImmediateCAS to AWACS !AWACS passes requestImmediateCAS to CAOC! CAOC assigns USMCAircraft to JTAC!CAOC sends readyOrder to USMCAircraft !USMCAircraft sends sitBriefRequest to AWACS !AWACS sends sitBrief to USMCAircraft !USMCAircraft sends requestForTAC to JTAC !JTAC sends TACCommand to USMCAircraft !USMCAircraft sends deconflictRequest to UAV!USMCAircraft gets targetLocation from UAV!!
Observer of AWACS with JCAS
Data gathered by
Observer
addObserver(USMCAircraft, JCASNUM1);
Observer is connected to SUO andmonitors itsI/O traffic
Test Detector Prototype:Sequence Matcher
processDetect(J2.2,data1,t1)
processDetect(J3.2,data2,t2)
waitReceive(J7.0,data3,t3)
Sequential triggering, same as test models
Example of Effect of State: AWACS
Rules
R1: if phase = “passive” & receive= "ImmediateCASIn“then output = "CASResourcesSpec" & state = "doSurveillance“
R2: if state = "doSurveillance“ & receive= "sitBriefRequestIn“then output = "sitBriefOut“ & phase = “passive”
matchsequence 1:initial state = passiveprocessDetect(ImmediateCASIn,””,1)waitReceive(CASResourcesSpec,””)
matchsequence 2:initial state = doSurveillanceprocessDetect(sitBriefRequestIn,””,1)waitReceive(sitBriefOut,””)
state = doSurveillance
need to know the state to enable this sequence
i1 i2
o2o1
state = passive
Solution: make activation of matchsequence2 conditional on matchsequence1
matchsequence2 can only start when matchsequence1 has successfully been performed
Observation Test Of AWACS
Observer ofAWACS
AWACSTestManager
Problem with Fixed Set of Test Detectors
• after a test detector has been started up, a message may arrive that requires it to be re-initialized
• Parallel search and processing required by fixed presence of multiple test detectors under the test manager may limit the processing and/or number of monitor points
• does not allow for changing from one test focus to another in real-time, e.g. going from format testing to correlation testing once format the first has been satisfied
Solution
• on-demand inclusion of test detector instances• remove detector when known to be “finished”• employ DEVS variable structure capabilities• requires intelligence to decide inclusion and removal
Dynamic Test Suite: Features
• Test Detectors are inserted into Test Suite by Test Control
• Test Control selects Detectors based on incoming message
• Test Control passes on just received message and starts up Test Detector
• Each Detector stage starts up next stage and removes itself from Test Suite as soon as the result of its test is known– If the outcome is a pass (test is successful) then next stage is
started up
Dynamic Inclusion/Removal of Test Detectors
message arrives
add induced test detectors into test set
test detector subcomponent removes its enclosing test detector when test case result is known (either pass or fail)
Test Manager Active Test Suite
Test Control
removeAncestorBrotherOf(“TestControl");
addModel(‘test detector”);addCoupling2(" Test Manager ",“Jmessage",“test detector", “Jmessage");
Adding Ports and Coupling by Relation Methods can be issued by any atomic model:
• add my ports and coupling to ancestor relation of another model – E.g., find my ancestor that is a brother of the given model, and add my ports and associated internal and external coupling to this ancestor
A
B
C D
E
A
addMyPortsToAncestorBrotherOf(B)
Issued by E, adds E’s ports to C, which is ancestor brother of B, and does associated coupling
B
A
BC
D E
C D
E
Adding Ports and Coupling (cont’d) Methods can be issued by any atomic model: • add output ports and coupling from source to destination – adds outports of source as input ports of destination and couples them• add input ports and coupling from source to destination – adds inports of source as outputs of destination and couples them
A
addOutportsNcoupling(C,B))
Issued by E, allows E to communicate with B
B
C D
E
A
B
C D
E
addInportsNcoupling(C,B))
in1 out1
In1 out1
Removing Models by Name and by Relation Methods can be issued by any atomic model:
•remove model by name – accepts the unique name of any atomic or coupled model within hierarchy and removes it and its immediate coupling
• remove model by relation to another model – e.g., remove ancestor that is a brother of the given model, can be used to avoid providing atomic models with names of other models
A
B
C D
E
A
removeModel2(C)
removeAncestorBrotherOf(B)
Issued by D, removes its parent which in this case is its ancestor that is a brother of B
B
A
BC
D E
AWACS Opportunistic Testing in JCAS
Test Control
CAS Model with AWACSobservation
Initially empty Test Suite
AWACS Opportunistic Testing in JCAS (cont’d)
Test Control adds appropriate Test Detector and connects it in to interface,
Test Control observes CAS request message to AWACS
AWACS Opportunistic Testing in JCAS (cont’d)
Test Control passes on start signal and request message
First stage detector verifies request message receipt and prepares to start up second stage
AWACS Opportunistic Testing in JCAS (cont’d)
second stage waits for expected response from AWACS to request
First stage detector removes self from test suite
AWACS Opportunistic Testing in JCAS (cont’d)
Second stage observes correct AWACS response and removes itself and starts up second part
AWACS Opportunistic Testing in JCAS (cont’d)
At some later time, second part of Test Detector observes situation brief request message to AWACS First stage removes itself and starts up second stage
AWACS Opportunistic Testing in JCAS (cont’d)
Second stage observes situation brief output from AWACS thus passing test, It removes itself and enclosing Test Detector
Summary• DEVS formalism is capable of capturing the information-processing
complexities, including the dynamics, underlying the MIL-STD-6016C standard
• DEVS modeling and simulation methodology is attaining core-technology for automated testing of military tactical data link standards
• Structure control implemented in Dynamic Structure DEVS enables automation of standards conformance testing
• Structure control agents induce structural change in themselves or others to effectuate different behaviors under different circumstances
• The agent-supported simulation metaphor applies to testing the conformance of multi-agent systems to complex defense information standards