verification and validation of mission data libraries for electronic...

Verification and Validation ofMission Data Libraries for

Electronic Warfare Operational Support

Flight Lieutenant Charles WinsorRoyal Australian Air Force

Dr Thomas MillhouseNova Systems, Australia

Introduction

Electronic Support systems depend on libraries of accurate, current and relevant mission data to identify emitters and provide tactical threat warning to aircraft and ships

Electronic Warfare Operational Support is the process of updating these mission data libraries to adapt to changes in the operational environment

Introduction

Library reprogramming has historically been

treated like software development

Software engineering standards demand

time-consuming Verification and Validation

for mission critical systems

Verification and Validation provide assurance

that an Electronic Support system will perform

in its intended operational environment

Introduction

Excessive Verification and Validation

may compromise the essential

responsiveness of Electronic Warfare

Operational Support

We propose a new approach to balance

the rigour of software engineering with

the responsiveness of Electronic

Warfare Operational Support

Acknowledgement

Outline

Electronic Support

Mission Data Libraries


Verification and Validation

Finding the Balance

A New Approach

Electronic Support

Electronic Support is the branch of

Electronic Warfare that deals with the

passive intercept, analysis and

exploitation of electromagnetic radiation

Electronic Support is used in modern

warfare for surveillance, intelligence

collection and tactical threat warning of

radar-guided weapons

Electronic Support

Electronic Support systems aim to:

intercept and detect signals

locate, identify and report emitters

ReceiverSignal

ProcessorOperatorDisplay

Mission DataLibrary

INTERCEPT

DETECT

LOCATE

IDENTIFY

REPORT


Most Electronic Support systems are loaded pre-mission with reprogrammable mission data libraries

These libraries enable automatic identification of detected emitters

High quality libraries are essential to achieve timely threat warning: manual emitter identification by an

operator is slow

fast reactions needed against threats


Emitter parameters form the basis of a mission data library:

enables identification of emitters from detected signals

Emitter-to-platform associations improve situational awareness:

identified emitters may be correlated to known platforms or locations

Additional information can be associated to emitters to improve situational understanding


Mission data quality characteristics:

Accuracy

Currency

Relevance

All share a common dependence

on time

All are interdependent


Mission data libraries are typically specified to suit an Area of Operations: libraries should be updated as emitters enter or

leave the specified region

Mission data is typically based on observed emitter parameters: libraries should be updated as emitter

parameters change

particularly important as we enter the era of software-defined radio

Consequences

• No identification

• Misidentification

• Ambiguous

identification

Consequences

No identification:

no threat warning

countermeasures not initiated

platform loss

Misidentification or ambiguous identification:

false alarm

wrong countermeasures initiated

mission abort

EW Operational Support

AimTo update libraries before they become ineffective or

unsuitable for operational use

ProblemThe electromagnetic environment may change significantly

on a daily basis

ChallengeTo complete post-mission analysis, update library

requirements, develop an updated library, and verify and validate that library to a suitable extent… all before the

current library becomes an operational liability!

EW Operational Support

ANALYSE

REPROGRAM

OPERATE

Pre-MissionUpload

Post-MissionDownload

MissionReplay

ResolveAnomalies

RecommendChanges

DesignVerifyDevelop

Validate

Verification & Validation

Australian Technical Airworthiness regulations treat mission data libraries as software (not as data):

similar regulations are applied in the Australian maritime and land domains

Software engineering standards like DO-178B are applied to routine library reprogramming:

threat warning sensors are mission critical systems

libraries may be assessed to need high Software Assurance Levels

Assurance of mission critical software demands time-consuming Verification and Validation


VerificationThe evaluation of whether or not a product complies with a requirement specification“The library was developed correctly”

ValidationThe assurance that a product meets the needs

of the customer“The correct library was developed”

[IEEE 1490-2011]


Verification and Validation are important to

provide assurance that mission data

requirements are satisfied by a library...

...but those mission data requirements change

whilst library Verification and Validation is

being conducted

An optimal balance must be found between

rigour and responsiveness

Finding the Balance

1. Programming errors

found and resolved

2. System performance

optimisation

3. Mission data currency

degradation

4. Mission data

relevance degradation

Verification

Software-only testing:

basic error checking

basic performance estimation

“Was the library implemented as designed?”

Hardware-in-the-loop testing:

advanced error checking

advanced performance estimation

assess emitter interactions

“Does the library design satisfy its requirements?”

Limitations of Verification

The same mission data is used to develop and

verify the library

The performance of a library is fundamentally

limited by the quality of the available mission

data

Verification predicts library performance

assuming mission data is correct

Library verification provides no assurance of

underlying mission data quality

Validation

Field testing:

“Were the library requirements fit for purpose?”

Options:

controlled or mission conditions?

generic test scenario or mission rehearsal?

emitters of interest only, or “background

emissions” as well?

Limitations of Validation

High cost

If emitter emulators are used, same fundamental limitation as verification:

predicts library performance assuming emitter emulation is correct

If authentic emitters are used, still not a true representation of the operational environment:

no two radars emit exactly the same signal

when does “close enough” become “good enough”?

A New Approach

Verification and Validation are activities

performed as part of the library reprogramming

process

Library reprogramming is an activity performed

within the EWOS cycle

What if we treated the entire EWOS cycle as a

holistic Validation activity?

Can rapid, continuous execution of the EWOS

cycle fulfil the aim of Validation?

EWOS as Validation

Collect “test data” from operations: true mission conditions emitters of interest background emitters

Assess library performance in the operational environment through post-mission analysis

Assess the ongoing suitability of library requirements: “Were yesterday's library requirements still valid

during today's mission?” “If not, what needs to change?” “And how else do we expect tomorrow's

mission to change those library requirements?”

EWOS as Validation

Does rapid, continuous execution of the EWOS cycle mitigate the need for a formal Verification and Validation process?

A low quality library can still cause mission failure or the loss of life and materiel:

focussed testing of high risk emitters

Potential to compound errors:

initial library version must be of reasonable quality

Similar principles to the Spiral Model for software development [Boehm, 1988]

EWOS as Validation

ANALYSE

REPROGRAM

OPERATE

[Boehm, 1988]

Improving Library Performance

Implementation

Holistic consideration of Verification, Validation

and the entire EWOS cycle

Organisations, personnel and infrastructure

capable of sustaining highly responsive EWOS

Expertise to identify mission risks within

libraries and tailor Verification and Validation

processes to specifically address those risks:

instead of blanket-application of generic

software engineering standards to EWOS

Into the Future

Mission data library development processes for Electronic Support systems are currently similar to the Waterfall Model [Royce, 1970]

Rapid, continuous execution of the EWOS cycle could be considered similar to the Spiral Model [Boehm, 1988]

Many new software development models have emerged in the last few decades

Is it time for EWOS to move into the twenty-first century?

Conclusion

Libraries of accurate, current and relevant

mission data are essential to enable the

performance of Electronic Support systems

Low quality mission data libraries can result in

mission failure and other serious

consequences

Assurance of library quality is conventionally

achieved through application of software

engineering rigour

Conclusion

Conducting extensive Verification and

Validation processes to assure library

quality can actually reduce performance

The EWOS cycle must be responsive to

keep pace with an evolving

electromagnetic environment

A better balance must be found between

rigour and responsiveness

References

D.C. Schleher (1999), Electronic Warfare in the Information Age

R.G. Wiley (2006), Electronic Intelligence: The Interception and Analysis of Radar Signals

DO-178B, Software Considerations in Airborne Systems and Equipment Certification

IEEE 1490-2011, A Guide to the Project Management Body of Knowledge

W. Royce (1970), “Managing the Development of Large Software Systems”

B. Boehm (1988), “A Spiral Model of Software Development and Enhancement”

Verification and Validation ofMission Data Libraries for


Flight Lieutenant Charles WinsorRoyal Australian Air Force

Dr Thomas MillhouseNova Systems, Australia

verification and validation of mission data libraries for electronic...

Documents