situation/threatfeedforward/backwardfw788gw6072/fw788gw6072.pdf · j.llinas calspan corporation,p....

J. LLINASCalspan Corporation, P. O. Box 400,

Buffalo,

NY 14225, USAEmail: llinasQcalspan.com

andRICHARD T. ANTONY

US Army CECOM Signals Warfare DirectorateVint Hills Farms

Station,

Warrenton, VA 22186-5100, USA

Data fusion has been definedas aprocess dealing with the association,correlation,and combination of data and information from multiple sources to achieve refinedposition and identity estimates for entities, and complete and timely assessments ofrelated situations and threats, and theirsignificance. This process (sometimeslabeleda "technology")is pervasive, i.e. capableofbroad, multi-domainapplication. Indeed,data fusion has found extensive application in the commercial/industrial sector aswell, in areas such as robotics and process control, and for numerous applicationsrequiring intelligent, autonomous processes and capabilities. One of the purposes ofthis paper is to describe the evolving standard descriptionof the data fusion processascribedtoby the U.S. JointDirectors ofLaboratories (JDL) Data Fusion Subpanel (aDepartment of Defense organization),as well as components ofthe attendant lexiconand taxonomy.

While the specific definitions of a "situation assessment (SA)" and a "threat as-sessment (TA)" have proven to be problem-dependent for most defense applications,these notions generally encompass a large quantity of knowledge which reflect the(dynamic)constituency-dependencyrelationships among objects of various classes aswell as events and activities ofinterest. Formulationofhypotheses aboutsituationsand threats is a process having the following properties:

" it employs many types of knowledge" it must consider multiple, asynchronous activities" multiple types of dynamic and static data must be processed" numerous sub-networks of interest in the situation/threat picture (numerousconstituency-dependencyrelationships)exist —this leads tofeedforward/backwardinferencing requirements" information-processing strategies are required to produce estimates of aggregatedforce structures (given individual unit positions and identities), as well as aggre-

gated behaviors (given individual events or activities)

" the situational or threat state is often ephemeral and thus temporal reasoningcapabilitiesmust be part ofthe processThe paper expandson theprocesses and techniquesinvolved in SA and TA anal-

ysis, and describes, from various points of view, why the blackboard paradigm isproperly applicable to problems of SA and TA analysis. This assessment includesvarious trade-offfactors (features,

benefits,

and disadvantages or complexities) inapplying blackboardconcepts to data fusion related reasoning processes.

Specific research and development by the authors and synthesis of the resultsof a survey on data fusion applications (shown within) has led to the formulationof a recommended generic, ideal blackboard architecture for these defenseproblemsdescribedin the paper.Keywords: Blackboards, data

fusion,

multisensor

fusion,

architecture.285

International Journal

of

Pattern Recognition and

Artificial

Intelligence Vol. 7 No. 2 (1993) 285-308©World Scientific PublishingCompany

286 J. LLINAS & R. T. ANTONY

1. CHARACTERIZATION OF THE DATA FUSION PROCESSThe goal of tactical data fusion is the development of a complete, accurate, con-cise, and timely picture of a tactical environment based on sensors that provideonly limited observables, coverage, resolution, and accuracy. The environment canbe very complex, consisting of potentially large numbers of many classes of bothstationary and moving entities. Since the analysis of individual sensor reports canlead to ambiguous, inconsistent, and highly local interpretations, the fusion of mul-tiple sensor data tends to enhance the situation understanding process. Althougha spatially-distributed network of heterogeneous sensors can increase the total in-formation available, the non-deterministic nature of the domain and the largelyexpectation-based character of thereasoning process effectively guaranteesa degreeof uncertainty in the fusion product. This uncertainty can be minimized by syn-ergistically utilizing all available sensor-derived information and relevant a prioridomain knowledge; the former provides dynamic situation information, while thelatter supports real world, context-sensitive-reasoning.

An abstract, three-level functional model of the tactical data fusion process isshown in Fig. 1. Level 1 fusion represents predominantly information extractionrelated to detection, association, classification, and attribute refinement, normallyassociated with single entities, based on the analysis of single sensor or multiplesensor measurements. This information results from the application of a series ofindividual algorithms to produce object detections (e.g. statistical hypothesis test-ing), position and tracking estimates (e.g. Kalman filtering methods), and identityestimates (e.g. digital image processing and pattern recognition methods). WhereasLevel 1 fusion products are generatedby numerical methods andreasoning based, forthe most part, on direct observables, Level 2 fusion is based primarily on the currentsituation description, available a priori(static) domainknowledge and expectation-based world models. These world models are generally implemented by knowledge-based reasoning paradigms such as expert systems and strategies based on thetheory of plans. Level 2 fusion extends and enhances the completeness, consistency,and level-of-abstraction of the situation description produced byLevel 1 to (1) com-pensate for the information-deficient and errorful measurementspace, (2) resolveambiguity in the Level 1 products and (3) develop higher level interpretations ofthe current situation based on reasoning in context.

Level 1 processing is largely numeric in character since the measurements aremetric in nature and amenable to the application of closed-form algorithms (typi-cally statistical and estimation/optimization methods). The products of Level 1processing are often described as a "labeled set", i.e. a set of points in space rep-resenting hypothesized objects, having "labels" such as position, velocity, identity,reflecting the results of Level 1 estimation processes. Generally, this is a ratherabstract information product, devoid of context, although sometimes "contextualaugmentation" techniques are applied as part of the estimation process for the[kinematic + identity] attributes.

BLACKBOARD CONCEPTS FOR DATA FUSION APPLICATIONS 287

Fig. 1. Model of the data fusion domain produced by the Joint Directors of Laboratories DataFusion Subpanel. From Ref. 1.

The first stage of Level 2 fusion performs situation abstraction which includesboth situation generalization and situation specialization. Situation generalizationallows bottom-up abstraction of entities or events that are either not directly mea-surable or not observed, to be inferred; alternatively, the sensor network could betasked to provide critical missing information that supports such inference. Situa-tion specialization is a form of top-down reasoning where subordinate elementsare deduced or inferred. Situation generalization and specialization develop thestructural, organizational, and functional relationships among domain elements andsupport development of a consistent, complete, and higher level-of-abstractionsituation description. Situation assessment provides a higher-level interpretationof the evolving dynamic situation description in terms of the relationship amongperceived events and the status of entities and organizations. Thus, Level 2 fu-sion processes result in a contextual interpretation of the rather abstract Level 1data, and provide insight into the numerous constituency-dependency relationshipsthat may exist among the various entities (vehicles, weapons, communication nodes,etc.) observed or deduced. Contextual interpretation is also supported by behav-ioral analyses ofevents and activities, and accomplishment ofsuch analyses requiresthe application of temporal reasoning. Finally, Level 2 processes also achieve a levelof entity aggregation, wherein theentities are linked via reasoning methods derivedin part from doctrinal deployment patterns determined a priori from intelligenceinformation; defense analysts call this type of analysis "order-of-battle" estimationwherein equipment-to-organization relationships are estimated. Level 2 process-ing is distinguished from Level 1 processing by a significant shift in emphasis tosymbolic rather than numeric processing.


Threat assessment (Level 3) develops friendly-force vulnerabilities based on in-terpretation of the dynamic situation with respect to both friendly and hostile forcedispositions. Thus, threat assessment represents a specialization of the currentsituation assessment describing both near term and long term threats to friendlyforces. Level 3 processing produces an important triplet of information: (1) lethalityestimates result from assessmentof hostile force capability toproduce friendly forcelosses; this is usually achieved by counting and reasoning functions relative to hos-tile force weaponry, (2) intent estimates result from intelligence data, operationalreadiness assessments, observation ofkey events or objects, etc.; this is a very diffi-cult estimation process since it is predictive and also influenced by many intangibles,and (3) opportunity estimates, sometimes called "course-of-action" estimates, whichresult from assessments of strong and weak points in force dispositions, logisticalfactors which may support or deny certain routes, etc. Fusion at this level, as forLevel 2, is the result ofhierarchical, multi-perspective reasoning applied to both thetotal set of observables but also based on extensive a priori data bases.

Ideally, the data fusion process should be an adaptive, feedback control process.In such cases, not always realizable in practice, there is another important functionin the process usually called "sensor or source management", and sometimes con-sidered as afourth level of the process. Some degree of research has been conductedon the application of concepts of intelligent control to the data fusion process butclosed-loop designs of this type have to date been rather limited.

Once a new target is detected, the fusion system characterized by the multiplelevel model just described attempts to verify the target's presence,refine its locationand trajectory, its measurable attributes, its probable classification, its associationwith a particular military unit, the association between that unit and others, thecurrent enemy organization and the perceived situation, and finally, the potentialthreat to friendly forces. Thus, the overall fusion process consists of numerousdependent and independent functional processes at multiple levels of abstraction.Since individual sensors may observe the same target at different times and havevery different processing and reporting delays, and since estimates of the situationdescription are asynchronous with respect to the sensor acquisition process, fusionprocesses are often inherently asynchronous. Both the sensor systems and the pro-cessing nodes can be spatially distributed. Real time performance is mandated bythe time-critical nature of the tactical environment. The dynamic character of thedomain and the limited information content of sensor reports insures a degree ofuncertainty in the situation development process. Table 1 summarizes thesefunda-mental characteristics of the tactical data fusion domain, and shows how some ofthe features of a blackboard (BB) system satisfy various requirements of the datafusion process.

In general, datafusion is supported by three underlying reasoning classes: spatial,hierarchical, and temporal. Spatial reasoning deals with the spatial relationshipsamong entities (e.g. distance metrics, relative locations, doctrinal patterns, appar-ent goal states), as well as their association with geographic and cultural domainfeatures (e.g. supportability, mobility, visibility, and communicability). Hierarchical


Table 1. Characteristics of the tactical datafusion problem and corresponding blackboard (BB)paradigm problem solving support.

Fusion system character Problem-solving strategy BB Paradigm implication

Complex, large scale problem Task decomposition BB partitioningDistributedsensors & nodes Spatially distributed processing Communication and controlReal time requirements Distributed processing Communication and control

ParallelizationEfficientprocessingMultiple levels of abstraction Task decompositionDynamic situation Dynamic process mc

HierarchicalreasoningTemporalreasoningDynamic process model

Uncertainty (data & decisions) Multiple hypothesis management Uncertainty management

reasoning is required due to (1) the predominantly vertical organization of militaryentities (e.g. vehicles, companies, battalions, regiments), (2) the multiple level ofabstraction nature of the reasoning process (e.g. local strategies vs. more globalstrategies) and (3) the inherent efficiency of top-down problem solving. Becauseof the dynamic nature of the tactical domain and the asynchronous nature of thesensor reports, temporal reasoning underlies the entire fusion process.

It is important to appreciate the complexities of the various reasoning problemspresented by typical defense-type applications. Much of the difficultyresults fromthe temporal aspects of these problems, which produce demandsfor non-monotonicreasoning and possibly default or plausible (or other heuristic) techniques, due tothe numerous imperfections in the data, even for the multi-source case, and in themany qualification criteria associated with numerous reasoning processes.

Clearly, thefusion ofmultiple sensor (e.g. radar and infrared) information to en-hance situation understanding in a tactical environment is metaphorically similarto the fusion of multiple sensory (e.g. sight and sound) information for perceptionenhancement in biological systems. Based on an extension of the human memorymetaphor, theprimary elements of the fusion process can be associated with shortterm, medium term and long term memory.2 Shprtfterm memory holds highly tran-sient short termknowledge, medium term memory holds dynamic, but less transientmedium term knowledge and long term holds relatively static long termknowledge. Long termknowledge in biological systems, as well as data fusion sys-tems, represents factual and interpretive reasoning knowledge. If sensory data isassociated with a biological system's short term knowledge, sensor data representsa fusion system's short term knowledge.

A primary objective of an automated tactical situation understanding system isthe development of the current relevant perception of the environment that is main-tained in medium term memory. Thus, the dynamic situation description (Fig. 1)is the fusion system's metaphorical medium term memory element. For the auto-mated fusion system, current emphasizes the character of the dynamically changingscene under observation, as well as the time-evolving interaction among a network ofdistributed processes. Because of memory limitations and the critical role mediumterm memory can play in system survivability, only relevant states are maintained


by tactical fusion systems. The notion of relevance proves to be crucially impor-tant to supporting decision-making under stress, and is a challenge to the fusionsystem designer to develop systems which prevent degradation in reasoning andsub-optimal stress-related coping patterns. Perception emphasizes the uncertaintyin the interpretation process due to the limited information content of the mea-surements, the non-deterministic natureof the domain, and the uncertainties in thereasoning process.

2. GENERAL APPLICABILITY OF THE BLACKBOARD (BB) PROBLEMSOLVING PARADIGM

The requirements of the tactical data fusion process were summarized in column1 of Table 1; the second column suggests strategies for dealing with each of thefusion problem characteristics. In this section, the capabilities of a generalizedblackboard-based paradigm are shown to be ideally matched to tactical fusion sys-temrequirements as suggested by column 3 of Table 1.

The opportunistic collaboration of a group of human experts from multiple dis-ciplines seated in front of a chalkboard provides the problem solving metaphor forthe BB paradigm. As their expertise permits, individual experts participate in theevolutionary development of the problem solution that is maintained on the black-board. The true human metaphor is opportunistic; however, it has proven difficultto achieve this explicit behavior in computer-based systems due to concurrency,consistency, and control factors. Thus, in most cases, the computational 88, tovarying degrees dependent on the detailed control paradigm, reflects a "moderated"problem-solving paradigm. In the computational analog, a conceptual frameworkfor communication and result-sharing permits a group of independent processes(specialized knowledge sources) under some degree of centralized control (the mod-erator) to cooperatively and/or competitively interact with the evolving problemsolution state (held by the BB).

lional-level fusion model of Fig. 1 supports arbitrary implementationArchitectures in software and hardware consisting, for example, of potentially largenumbers of homogeneous and/or non-homogeneous spatially-distributed processingnodes. The individual processes can represent a mix of technical disciplines andproblems solving approaches. Sophisticated control of these complex structuresmay be required due to the inter-dependencies of these processes operating acrossmultiple levels of abstraction.

In the following, weelaborate on each of thefactors associated with BB applica-bility as shown in the third column ofTable 1.2.1. Problem Complexity and Distributed ProcessingThe application of data fusion processing is often employed in the face of com-plex information-handling requirements. Uses range from those involving single,autonomous systems (e.g. fusion within a single aircraft's avonics/sensor system)to force-level systems involving a hierarchy of fusion centers. Usual approaches to


the solution of such problems involve task decomposition. The BB paradigm ishospitable as a basis for implementation of such solutions since it is amendable topartitioning, albeit at some expense of processing and coordination complexity.

Thesesamefeatures permit the BB structure to be applied to distributed systemsprocessing. Other than for single-platform or single-site applications, data fusionsystems are almost always distributed, and require efficient communication andcontrol protocols for effective implementation. In these systems, the "world state"of interest is often partitioned on a spatial (regional, azimuthal, volumetric, etc.)basis permitting corresponding separate BB structures to be maintained for eachsubdomain. There areoften significantprocessing difficulties associated with spatialand temporal biases and temporal latencies but these are handled by special fusionprocessing algorithms separate from the BB structure.

2.2. Real-time RequirementsWhile data fusion processing has found frequent application in peacetime surveil-lance and various defense and non-defense applications not involving time-criticalprocessing, real time processing is nevertheless a frequent requirement in defensesystems. While the BB paradigm has proven to be a flexible framework for complexproblem solution, time-critical processing has proven generally difficult to achieve(e.g. see Refs. 3-5 ).

2.2.1. Distributed problem-solvingApart from the inherent spatially-based distributed character of many data fusionsystems, real time processing demands are in part often dealt with by employ-ing techniques of distributed problem-solving (e.g. Refs. 6-8). These are formalproblem-solving methods which offer strategies for achieving processing efficiencyand consistency in the global solution derived from partial solutions developed bydistributed processing nodes. Such techniques can be implemented within the BBframework but may impose special functional or processing requirements on specificprocessing elements of the 88.

2.2.2. Problem-solving efficiencyBecause of the typically time-critical requirements for fusion products and limitedcomputational resources in the tactical environment, highly efficient fusion process-ing is generally required in most applications. Since even an intrinsically efficientalgorithm can become I/O bound, efficient problem-solving requires both efficientalgorithms and efficient BB access. Efficient algorithms imply efficient support forspatial, hierarchical, and temporal reasoning. Although the BB is again hospitableto creative strategies to achieve efficient processing, implementations are highlyspecialized to particular problem domains.


2.3. Multiple Levels of Abstraction and Dynamic SituationsIn most defense applications, data fusion processing tends to be hierarchical in na-ture due to the inherent hierarchies built into defenseorganizations and operations.As a result, the fusion process also progresses through a hierarchical series of in-ferences at varying levels of abstraction; this characteristic is depicted in Fig. 2,which also suggests the iterative, continuous nature of these inference processesdriven by the temporal character of the usual defense problem. Blackboards arealso hospitable to hierarchical partitionings but the implementation of reliable tem-poral reasoning strategies, while a difficult issue, is separate from the questionof the appropriateness of the BB architecture. That is, the BB architecture is aproper frameworkwithin which to dealwith hierarchical and temporalrequirements;achieving correct and efficient hierarchical and temporal reasoning capabilities is adifficult but separatematter.

where what when who why how

physical objects

Fig. 2. Multi-level/multi-perspectiveinferencing.

friendly vulnerabilities & mission —^^options needs J/ friendl^ssets

specific

&

global

individual

2.4. Uncertainty ManagementAs noted previously, the existence of uncertainty permeates the entire datafusionprocess. Uncertainty exists not only because of sensor characteristics and poten-tial inconsistencies betweenreal world behaviors and formal assumptions built intoalgorithms (e.g. statistical independence) but even the reasoning processes may beuncertain. Formal methods for both uncertainty representation and propagationhave been the subjects of extensive research and data fusion system designers arefaced with difficult choices in this area. The BB structure, however, remains as an

Aw organizations "\^^7\^ events

ys^speclfic aggregated£. .^ terrain

&

enemy tactics "^s/ -v-.local global .^/

A^V-^enemy doctrine

&

objectives -«^-w.specific global


attractive formulation within which to implement various uncertainty managementtechniques.

2.5. Issues in Blackboard Managementand Evidence Propagation

Due to the dynamic, evolving nature of the tactical environment, time-late infor-mation, limited sensor-derived information, deliberate deception, ignorance of anadversary's intent and imperfect reasoning knowledge, uncertainty in perception,as noted in Sec. 2.4, is inevitable. The dynamic, sequential nature of the deci-sion process leads inevitably to the maintenance ofmultiple hypotheses which, overtime, can generate a combinatorial explosion of hypotheses. Thus, in order to sup-port the tactical fusion process, the BB must (1) hold the confidence in all existingBB hypotheses (or states) (2) maintain multiple conflicting hypotheses, (3) sup-port multiple views and (4) permit efficient retraction of belief and all associatedbacktracking operations.

The hypothesis explosion problem can be minimized by (1) powerful, context-sensitive algorithms that establish only relevant hypotheses in the first place and(2) representations that support efficient pruning of low confidence hypotheses, aswell as straightforward hypothesisretraction. Reasoning in contextdepends heavilyon the association among entities and events and with respect to extensive naturaland cultural domain feature databases. The most plausible hypotheses tend to be"close" along spatial, organizational or other natural problem-solving dimensions.Hypothesis pruning occurs as additional information becomes available that reducestheoverall situation uncertainty. Thus, a datarepresentation that supports efficient,highly localized spatial and semantic search facilitates both complex contextualreasoning and the development and maintenance of multiple hypotheses.

Three principle classes of fusion processes that interact with the BB can be de-fined. Refining processes do not change the level of abstraction of the data, butmerely refine the situation description. Aggregation processes attempt to producehigher level ofabstraction products from lower level of abstraction products. Subor-dination processes attempt to develop and refine lower level databased on a higherlevel of abstraction perspective. The first process class supports local reasoning,the second bottom-up reasoning, while the third supports top-down reasoning.

There are four ways new evidence can affect the BB; see Table 2. New evidencethat supports an old eventcan update (e.g. based on additional, newor more currentinformation) the status of the old (or existing) event (Class 1). Thus, Class 1represents a strong constructive form of BB interaction (e.g. track file update).Alternatively, if new evidence refutes an old event (Class 2), that event must bere-evaluated in light of theevidence. Thus, Class 2 represents a strong destructiveprocess (e.g. the existence of a staging area in a certain region is refuted by themovement of all units out of that region). If new evidence supports a new event(Class 3), an event is posted to the 88. Ifnewevidence refutes a newevent (Class 4),this knowledge must be preserved on theblackboard in aform that can subsequentlybe utilized. (Although the generation of a non-event may be the simplest approach,


the hypothesis combinatorics that could result may make such an approach unde-sirable). In a real world fusion system, both the creation and destruction of eventsare critical operations.

While the first four classes of BB interaction support strong inferences, the nextfour support much weaker inferences. There are four ways in which the lack of newevidence about an old event or an anticipated event can affect the 88, as shown inthe bottom portion of Table 2.

In addition to an effect on a single event, evidence (or the lack of evidence) cansupport multiple classes of BB interaction, a particular nuance, and complexityin the application of BB's to data fusion processes. By refuting one hypothesis,new information can support another existing hypothesis or create an entirely newhypothesis. Information that establishes the new location of enemy force elementson the friendly side of a river provides both aforce element location update and arefutation of a BB state relating to a "possible aggressive river crossing operationin progress". The collisions between the eight processing classes tend to drive the88-based situation description to the most plausible interpretation of the availabledata. However, because of the time-varying nature of the tactical domain, whilesome hypotheses will stabilize over time, other BB states will continue to evolve.

These operations can cause significanthypothesis instantiation and pruning com-plexities in datafusion systems, and generally require efficient BB managementandevidence propagation strategies.

Table 2 summarizes the eight general classes of process interaction with the88. Combined with efficient spatial and semantic data representation techniques(e.g. object-oriented), Table 2 provides a foundation for the development of a pow-erful 88-based truth maintenance capability. In effect, BB update and maintenancerequires eight functional agents or demons: four constructive and four destructive.Four agents evaluate new evidencefrom either the sensors, theBB or the long termfactual andreasoning knowledge, whilefour agentsevaluate the lack of newevidence.In effect, all functional elements (or objects) of thefusion process must support eachof the eight hypothesis generating operations. Alternatively, the blackboard objectcan support a generic form of each agent. In the latter cases, individual functionalprocesses interact with the BB through these eight generic agents.

In summary, although the potential for a combinatoric explosion ofhypothesesis an inherent characteristic of multiple hypothesis systems, the impact can beminimized by (1) powerful, context-sensitive algorithms that establish only relevanthypotheses and (2) representations that support efficient pruning of low confidencehypotheses and straightforward hypothesis retraction. Data representations thatsupport efficient, highly localizedspatial and semantic search facilitate both complexcontextualreasoning and the development and maintenance of multiple hypotheses.

3. SURVEY OF DATA FUSION APPLICATIONSThe blackboard paradigm has indeed proven to be a very flexible model for dataprocessing in a wide variety of both defense and non-defense applications (see,


Table 2. The eight principle BB interaction classes.

BB Data BB Effect Inference*

Class

Input

1 New Evidence Supports Old Event2 New Evidence Refutes Old Event3 New Evidence Supports New Event

4 New Evidence Refutes New Event5 No New Evidence Supports Old Event6 No New Evidence Refutes Old Event7 No New Evidence Supports Anticipated Event Special8 No New Evidence Refutes Anticipated Event Special

Update Old Event S. ConstructiveUpdate Old Event

S.

DestructiveCreate Event S. ConstructiveCreate Non-event S. DestructiveDecay/Special W. DestructiveMaintain/Special W. Constructive

VW. DestructiveVW. Constructive

S = Strong, W = Weak, VW = Very Weak

for example, Ref. 3 and also Refs. 9 and 10). In data fusion applications it hasbeen applied to intelligence, surveillance, pilot-aiding systems, and critical noderecognition problems, among others. Table 3 shows some 13 representative datafusion and defense applications of BB architecture, derived from a literature reviewof materials presented in the last five years. The purpose of the review was inpart to assess the breadth ofsuch applications but most importantly to accumulatea view of lessons learned in such applications. In particular, it was desired toexamine all the negativeaspects encountered in these cases, and from those aspectsto attempt to derive apicture ofan "optimal" BB architecture,at least conceptuallyor functionally, for a wide range of defense problems; see Sec. 5 on this.

A historical examination of the (generalized) BB paradigm and its evolutionyields a list of the general advantages and disadvantages of BB systems; thesequalities are shown in Table 4. Study of the cases presented in Table 3 revealedsome new aspects of BB implementation and additional detailed understanding ofthe features in Table 4, and reiterates or expands on some of the ideas in Sees. 1and 2.

For defense applications, one of the most crucial and recurrent issues is achievingreal time or near real time performance in systems that employ a wide range ofknowledge sources (KS's) and process a large amount of input data. Real systemsof this type thus face combinatoric problems in applying a large knowledge base toa large dynamic data set. Given that, as described in Sec. 2, the BB paradigm doesnot truly implement the opportunistic, interrupt-driven, human problem-solvingprocess, optimal parallel scheduling and application of the KS's in the computersystem becomes a major challenge. Such parallelization efforts are frequent in thedefense literature; strategies on thesoftware side have ranged from everfiner-grained(e.g. down to the rule level) KS partitioning to the use of a BB "manager" forBB I/O control, to the use of "private" (i.e. local KS only) and "public" (all KS's)databases, in efforts to localize and parallelize the solution components. Alternatestrategies employ "demon-based" software designs (as in our previous commentsin Sec. 2.5) and the use of sophisticated queuing models in conducting real timeperformance analyses in the design effort. There are various difficulties with thesedesign choices. On the one hand, Amdahl's rule5 prevails over all choices— this

Table 3. Representative defense applications of blackboard architecture.toCOOiSystem Domain Task

of

BB systemBB datastructure

Control

Knowledgesources

Knowledgerepresentation

Key

elements,

Lessons learnedparadigm

TRICERO

9

Aircraft

Fusion

of

ELINTand

GOMINT

Hierarchicalevent list

Priority rule-based approach " ELINT

KS

"

COMINT KS

Rule- based

One of

the

first

dis-tributed processingarchitecture

for

a BBsystem

surveillance

Ground

Surveil-lance Robot

Sensor fusion for

surveillance robot(Marine Corps)

Optimal sensor

control,

filtering

of

multisensordata

Class

tree

for

objectNot described

Sensor KSControl KS

Planner

KS

Not described Describes

fundamen-

tal aspects

of

sensordata

fusion

and im-pacts on data repre-sentation

GSR8

identification

Live testing in ar-mored personnelcarrier

BB

for

sensorcontrol 11

Multisensormanagement

Management

of informationflow

in multi-sensor data

Distributedlist-type

Data driven

KS

activationTopology and net-work monitorInternode negotia-tion broker

Generally, all arerule- based

Achieving indepen-dence

of

KS's to min-imize contention/consistency issuesLogically indepen-dent

KS

partitioning

may

limit speed-upachievable

Message prioritizer

Comma interface

Track DB

managerSensor

mgmtandcontrol

fusion

system

Navy

C

2

Connectionism

and RT ExpertSystem

for

Tacti-cal DF 12

Tactical picturecompilation from

fused

data

Multi-level Rule-basedscheduler

Not described Not describedbut tried bothrules and seman-tic networks

Used real Naval exer-cise data which high-lighted need

for

largeprocessing power toachieve RT perfor-

hypothesis tree

manceExperience here re-sulted in change

from

rules to semantic net-works

for

knowledgerepresentation

for

RTperformance coupledto connectionist-typeHW

Generic

ExpertSystem Tool

GEST

13

Development tool Implementation

of

several com-municating ex-pert systems

Frame tree and

facts

listMeta-knowl-edge sourcewith "private"rule and

frame

tree

Several

allowed: eachis represented on

Four schemessupported:- production

Each

KS

has privateKB to prevent BBaccess

from

becominga bottleneck

BB by a "descriptor

frame",

each is anindependent

GEST

environment

rules

framesfacts

procedures

c_

rri—

>"zHO

)'

I

i ':

" Three methods

of

uncertainty represen-tation

" Providesexplanation

ES for

data Spacefusion based on surveillanceBB 14

Classification

Event list andand ID

of

targets target classes/based on

fused

ID's

Rule-based "

Interface KS

control

of

an " Speed/accelerationevents list " AltitudeFacts and rules " Argue

for

importance

of

uncertainty repre-sentation and input:

data Radar cross section use Dempster-Shafer

" Doppler spectrum" Polarizationmethod

" Partition

KS's

into"negative"

KS's

which eliminate hy-potheses and "pos-itive"

KS's

whichnominate hypotheses

" Employ a develop-ment tool

CAGE

and

POLIGON

5Electronic intelli- ELINT data

CAGE:

global

gence

(ELINT)

fushion

list

POLIGON:

CAGE:

User Not describedspecified rule

CAGE:

Rule- "

Goals of

researchbased were to study RTwere to study

RT,

POLIGON: concurrent designFrame- based and " Direct paralleliza-

set

for

concur-

rency

acrossFrame-

based,

distributed hi-erarchical con-trol

for

the BBsystem

and within rules tion

of

conventionalBB systems will onlyresult in small speed-up

factors:

that

is,

central

control,

nomatter how

manyKS's

may

run in par-

allel,

drastically lim-its speed-up (in

ef-fect,

Amdahl's rules)(CAGE)

" Many special pro-gramming techniquesare required to dealwith concurrency

KS's,

and with-in rule sets

POLIGON:

Fine-grained(rule level) dis-tributed con-trol—no cen-tral schedules.Basis is ademon controlsystem

control and to as- toCOsure data consistency

(CAGE)

DOr>oCDo>:SoozQitshotcrt13OsoO

l§IeOz>E.o'5iOiZ

Table 3. Cont'd

; System Domain Task

of

BB systemBB data

Control

paradigmKnowledge

sourcesKnowledge

representationKey

elements,

Lessons learnedstructure

"

Six

techniques tocontrol conflictresolution

i

I

Table 3. (Cont'd).


Key

elements,

Lessons learned toCO00

"

Shared memory

multi-processor H/Wcannot eliminate theBB as a bottleneck(CAGE)

" Distributed approachin

POLIGON

raisesquestion about con-

vergence

to a solu-

tion;

this led tohierarchical controlapproach supportedby expectation-basedreasoning

" Primary difficultyin fine-grained ap-proach to paralleliza-tion is assurance

ofconvergence

and non-deterministic behav-ior (POLIGON)

Enemy

Contact

ThreatReport Expert assessmentSystem

ECRES

15

Fusion

of

multi- Multi-dimen- Centralized; " (Message) collec- Generally rule- " Multiple hypothesissource data

for

sional parti- details not tion Mgr based representation

of

order

of

battle tioned data described " Zone

mgr

resultsand threat area, compris- " Data

fusion

" Sophisticated terrainassessment ing "plates" " Force organizer knowledgeand "arcs" " Threat assessor representation" Military knowledge " Employ a BB "man-ager"

for

data con-trol/consistency

" Describes limitations

of

shared

memory

HW and lockout/consistency issues

" Describes rqmt

for

scenario generator

for

BB developmenttesting

REACT

16 Pilot aiding

for

Optimal threat Frame tree Numerical pri- Four cooperating

KS:

Frame/rule-base/ " Uses

GEST

threat response response ority scheme "

Global

planner/

fact

combination " Multiple

pgmg

lan-strategies routing

guages

and integra-t Local planner tion

of

symbolic and

" Threat assessment numeric processes" Ownship

C-l

rri—iz>

SO

>zOz

ystem "omam Task

of

BB systemBB datastructure

Control

paradigmKnowledge

sources

I

Table 3. (Cont'd).

Domain Task

of

BB datastructure

Control

Knowledgesources


Key

elements,

Lessons learnedSystem

BB system paradigm

" Object-orientedapproach" Demon

KS's for

data

BlackboardObjects17

Air surveillance Support radarcontroller withtrack mgmt

Not described Not described Not described Not described

purging and control

" Public and privatevariables

for

control

of

data

flow

BB (bot-tleneck prevention)

Identification

of

C

2 Operation

Nodes, ICON

18

Critical

combatnode

Multisource Multilevel Not describedobject network " Message reader/extractor Rule based " Integration

of

numeric/symbolicprocessing

" Employment

of

adevelopment tool

" Object-orientedapproach

message-baseddata

fusion for

node

identification

" Emissionsanalyzers"

C

3 node

identifieridentification

Radar/ImageBlackboard 4

Sensor

fusion

for

space robotObject trackingbased on

fused

data

BB imple. Via "BB man-mented as an ager" part

of

object the BB object

Target informationTrack formationTrack merging

Not described " Develops argumentto applicability

of

object-oriented

pgmg

as natural

for

data

fusion,

as well as BBswhich embody the

OO

approach

" Argues that

OO

ap-proach with BB asan object is manda-tory

for

RTapplications

" Describes need for

iI:

adaptive

KS's

" Employ queueingmodels

for

RT analy-sis and designsupport

" Importance

of

expla-nation

featureDACORR

II19 Critical combatnode

Land battle- Hierarchical,

Control KS

frame-based

maintainingtemplates multiplehy-

pothesis withnumerical

" Object

frame KS

" BB

KS Declarative,

rule-based and

frame-

based templatesbased multisensordata fusion

for

order

of

battle

identification

" Templatesanalysis

weightingscheme

for

to

CO

COcompetinghypotheses

DOr>DOo>S3OOOZOPITlHvi**1Osoo

ca>

►—

Ioz>r>—


rule simply says the length of time it takes to run with parallel processes is thatof the longest single process plus some overhead to implement the parallel sys-tem. On the other hand, the alternative which also is a fine-grained approach alsohas elements of difficulty; fine-grained systems are often tightly-coupled, requir-ing extensive inter-process communications, and thus higher overhead, and suchapproaches can exhibit instability in convergence to a desired goal state and non-deterministic behavior. In general, this aspect of applying BB architectures to datafusion problems demands great care in problem decomposition, e.g. if analysis canproduce loosely coupled'or decoupledsub-problems, then multiplicative speed-up fac-tors through sub-problem parallelization can potentially be achieved. Thus, thekeyBB design aspects are associated with the partitioning and allocation steps in theclassical system engineering method, and in the use of moderate-grainedparalleliza-tion within the de-coupled functions in order to achieve balance between speed-up,reliable behaviors, and goal convergence.

Table 4. Advantages and disadvantages ofBB systems.

There are some other summary observations on architectural issues that derivefrom an analysis of the survey data:

A typical system employs many types of knowledge.Data lockout, consistency, and control issues are very important —these issuesare magnified with parallelization since the BB system is then analogous to ashared, distributed data base system.

Advantages Disadvantages

" Flexible structure, inherent modularityfor broad range of applications, especiallyill-structuredproblems

" Scheduling or control can become verycomplex and is generally domain de-pendent

" Can be used for both static and dynamic " Robust generic developmenttools justaspects of a problem now maturing" With proper design can achieve grace-ful degradation in blackboard network

hierarchy" Limited (blackboard-based) communica-tion among KSs

" Easier overall knowledge base control and " Truth maintenance system oftenrequiredvalidation through KS partitioning as KS's operate autonomously" Expansion and growth relatively easy byadding KSs " All solution state knowledge on black-board causes high blackboard-KS input-

output as potentialbottleneck

" Simple KS-KS communicationprotocol " Can be expensive to build and run(no message-passing protocolsrequired)" Useful paradigm for exploratoryresearch,rapid prototyping, and incremental deve- " Incorrect problem decomposition, oftendiscovered late,requires system restruc-

lopment turing

" Speed-up strategies employing fine-grainparallelizationimperils assured conver-gence to a solution


These survey results suggest the following design factors: maximum data de-coupling and information-hiding techniques are required as well as private and publicdata sets and the use of an intelligent BB data manager.

Choices of hardware architecture are also very important.These survey results suggest the following design factors: in general, high pro-

cessing power is categorically desired. Shared-memory multiprocessors should beavoided; in Ref. 20 it is suggested that, depending on the nature of the problem,process-structured array, pipe, and pyramid machine architectures aremost applica-ble for arange ofapplications, followed by augmented treesystems andreconfiguringnetworks; all of this hardware is, however, in the category ofhigh-risk, leading edgetechnology.

Other design features and system qualities would include:an object-oriented approach,an explanation feature (see especially Ref. 21),a truth maintenancesystem to accomodate non-monotonic reasoning processes,alternative conflict resolution strategies; at least for those serial portions of theprocessing,some type of uncertainty propagation and managementapproach, andintelligent use of demon-concepts for process control and data managementIt is important to understand that the composite effect of all these design and

implementation strategies may not be adequate to achieve the desiredrun-time per-formance for specific, stressing problems. In such cases, an incremental step towardimproving run-time performance is to locally restructure the reasoning processes (ifpossible) into a rapidly executable strategy (e.g. decision tree) —or to re-examinethe knowledge representation concepts (see, e.g. Ref. 12).

4. THE APPLICABILITY OF DISTRIBUTED PROBLEM-SOLVINGTECHNIQUES

As noted in Sec. 2.2.1, and as we have noted in conducting our survey, specializeddistributed problem-solving techniquesare applicable to many datafusion problems,and have in fact been employed in various studies. With the modern emphasison joint service operations (army -navy -air force) in the military, these conceptsbecome evenmore important as abasis for deriving and sharing among such servicesa consistent, maximally accurate tactical picture of the situation and threat states.

Several distributed coordination and problem-solving frameworks have been de-veloped in these research areas. Each uses different ways to divide control, re-sponsibility, and knowledge, and share partial results. These include contracting,multiagent-planning, integrative, and open-system frameworks. Various of thesestrategies may be applicable to distributed data fusion applications, and wouldrequire specific modifications to the general BB framework for implementation.

In contracting or negotiatingframeworks, problem solvers use bidding, contract-ing, and information-exchangeprotocols to allocate work orresolve conflicts. These


include e.g. the Contract Net,22 and the Multistage Negotiations.23 Multiagent-planning frameworks use a single agent or a group ofagents to form a coherent planfor solving a multiagent problem. Dependencies and conflicts among the actionsand knowledge of different agents are identified in advance. Communication andsynchronization acts are inserted into each agent's plan to prevent conflicts whenthe plan is executed.

Open-system frameworks provide theoretical and practical models of flexible,self-defining aggregation, communication, and coordination frameworks. Theseframeworks may include locally reconfigurable communities of agents whose in-teractions are based on modeling each other, not on external global or hierarchicalsynchronizationmechanisms (see, e.g. Ref. 6).

Still other paradigms have been suggested. For example, Ref. 24 describes a"process assembly network" and both hierarchic and "anarchic" derivatives for thesituation assessment task. Lesser and Corkill 25 have proposed a "functionally accu-rate, cooperative" distributed system concept suited to applications where the datanecessary to achieve a solution cannot be partitioned in such a way that a node cancomplete a task without seeing the intermediate state of task processing at othernodes.

All of thesestrategies require some intelligent approach to control of the system;this control problem is sometimes called the "coordination problem" , e.g. see Ref. 7.The coordination problem is the question ofhow to allocate work to a collectionof agents over time to maximize the values of some performance criteria. At leastthree-levels to this problem exist (not tobe confused with data fusion process levels):the domain-level problem of deciding what work is to be done and allocating Itamong agents (Level 1) and two metaproblems of settling which agents will do thedeciding and how (Level 2), and how to construct representations and interactionor analytical frameworks that allow for stating and solving any problems, includingthe coordination problem (Level 3). Each of these metaproblems can be treated asa multiagent problem.

Four approaches to coordination are presented in Table 5; see Ref. 7. Thesecoordination approaches vary according to how much and what type of uncertaintythey can accommodate and how much they rely on the assumption that metaprob-lems have been solved. As noted above, these types of control may be required inparticular data fusion applications, and would require appropriate modification ofthe control function in the BB system. Any of these may be employed at any levelof decision, but each presupposesthat some prior questions of context and represen-tation have been solved. On reflection, the difficulty of locating the boundaries ofaproblem or collection ofproblem solvers should become clear according to Ref. 7,from which these comments are derived. Stating, dividing, or allocating a problemrequires that other contextual problems (for example, of description or method) besolved first; but solutions to the contextual problems determine the nature of andthe possible solutions for the domain problem. It becomes easier to see why opensystems approaches are so important.


Table 5. Four coordination approaches(after Ref. 7).

Type of Control Features

Explicit control: (Procedure calls or master/slave)Explicit synchronization and communication:(Semaphores, monitors, and so on)Functionally accurate/cooperativeapproach:(Triangulationand convergence on usefulresults)Reasoned control: (Agents use knowledge ofselves and others to build andrevise coordi-nation frameworks)

Explicit constraints; centralized; minimallyadaptiveInteractionconstraints: semi-centralized;adaptiveto temporal uncertaintyOpportunistic control; fixed interactions;adaptiveto some semantic and temporaluncertainty; semi-centralizedPredictions and adaptive interaction;adaptiveto more semantic, temporal, andinteractionaluncertainty; minimal sharing;decentralized

5. REPRESENTATIVE ARCHITECTURES FOR DATA FUSIONWe have described some of the many reasons why the blackboard approach is espe-cially applicable to the data fusion problem. For military applications, data fusionfor situation assessment may be partitioned in a number of ways, e.g. by warfaretype, by command level, by type of information processing, by geographicalregion,or any of a variety of other ways. Figure 3 illustrates one concept of a blackboardapproach to situation assessment. At the bottom of the figure, a tactical problemis partitioned by type of warfare (e.g. surface, sub-surface, air-to-air, etc.). Foreach warfare type, a separate knowledge base and logical knowledge agent oper-ates on observational data to detect energy (infer object detection), establish theposition and velocity of objects, estimate attributes of platforms or emitters, andestablish the identity of low level entities or objects. For each warfare type, a Level1 fusion knowledge based process is applied. The result of the individual Level 1processes is an evolving order of battle. These data can be shared via a sharedmemory mechanism, although in certain cases a shared memory architecture canbe suboptimal.

At a higher inference level, a situation assessment/refinement process may beperformed. At this level, more abstract inferences are sought; the who, what,where, when, and why of the military situation (top of Fig. 3). At this level, theanalysis seeks to aggregate objects and entities into larger military units, determinesequences of events and activities,establish the meaning of objects and events in thecontextof the environment, predict future events, and interpret the meanings behindan evolving situation. Figure 3 also shows a second set of knowledge agents andknowledge bases used to address this higher level problem. In Fig. 3, the situationassessment problem is partitioned into four subproblems, viz. object processing,contextual processing, prediction, andinterpretation. Each subproblemis addressedby a separate inference process and knowledge base. Again, information is sharedvia a shared memory mechanism.


Fig. 3. Example ofblackboard data fusion architecture.

Figure 4 shows a conceptually optimal BB architecture for data fusion applica-tions (note that this is not a process-flow diagram —it is only conceptual). Thisconcept diagram attempts to respond to all the "design factors" relative to BB sys-tems that were denoted at the end ofSec. 3. One important aspect is in the approachtocontrol, which is conceptually partitioned into a "refinement data" controller, andan "intelligent BB manager", both of which only operate on "public" data emanat-ing from the knowledge agents (KAs). Further modifications in the approach tocontrol, following the ideas of Sec. 4 and in Table 5, may be required for specific ap-plications. This partitions refinement processing (considered simpler and involvingdata having less uncertainty) from aggregation and subordination processing, andfrom the process of truth maintenance for the multi-hypothesis blackboard. Notetoo that both BB controllers interface to the multisource/multisensor manager sothat data/information requests can be accommodated, and so that the total BBarchitecture comprises a closed-loop feedback system.

6. SUMMARY

An overview of the data fusion process as currently defined by the Joint Directorsof Laboratories Data Fusion Subpanel (a U.S. Defense Department organization)has been provided. As noted, fusion processing is a complex, multidimensional,and multi-perspective process. Further details, viewpoints, and solution techniquesare described in Refs. 27-32; Ref. 31 includes some 500 additional references forthose interested in the fine points of data fusion processing. Specialized aspectsof the reasoning and data processing flows for data fusion and their relation to


t

OptimalDataRepresentation Moderate - Grained Parallelization

Allocate toProcess-

Structured

Array,Pipe,PyramidProcessorHardware

Loosely CoupledSub-Problems,Agents, KB's

Fig. 4. Conceptually optimal blackboard architecture for data fusion applications

the BB model were also discussed. In particular, problem-solving efficiency, BBmanagement and control issues were expanded upon.

The research survey conducted in formulating this paper certainly corroboratesour a priori intuition that the BB paradigm is well-suited to datafusion applications;various other researchers have developed similar conclusions based on their own


research (see e.g. Ref. 26). However, the added complexities of many real time,combinatorially stressful datafusion problems still result in difficulties in designingBB systems that have all thefunctional qualities desired as well as are able to satisfyperformance requirements. This paper has attempted to identify and examine suchdesign issues asreflected in recent research on BB systems, and to offer aconceptualBB architecture that exploits many of the architectural lessons learned from thesurveyed research.

Finally, it is noted that while the BB paradigm has an inherent symmetry withdata fusion and C2processes, specialized concepts for BB control, drawn largelyfrom the distributed problem-solving research area, are also applicable.

REFERENCES1. F. E. White, "A model for datafusion", presented 29 April 1988.2. R. T. Antony, "A framework for automated tactical data fusion", in Proc. 1987 Tri-

Service Data Fusion Symposium, Johns Hopkins University Applied Physics Labora-tory, Laurel, MD, June 1987.

3. V. Jagannathan,R. Dodhiawala and L. S. Baum (eds.), Blackboard Architectures andApplications, Academic Press, Boston, 1989.

4. D. Johnson et al., "Real-timeblackboards for sensor fusion" , in Sensor Fusion, Proc.SPIE, Vol. 1100, 1989,pp. 61-72.

5. H. P. Nii, N. Aiello and J. Rice, "Framework of concurrent problem solving: A re-port on CAGE and POLIGON", in Blackboard Systems, Eds. R. S. Engelmore andT. Morgan, Addison- Wesley, Workingham, UK, 1988, pp. 475-501 (chp. 25).

6. L.

Gasser,

"Distribution and coordination of tasks among intelligentagents", in Proc.First Scandinavian Conf. on AI, Amsterdam, The Netherlands, 1988.

7. L.

Gasser,

"Distributedartificial intelligence",AIExpert4, 7 (1989).8. S. Y. Harmon, G. L. Bianchini and B. E. Pinz, "Sensor data fusion through a dis-

tributed blackboard", in Proc. 1986 lEEE Int. Conf. on Robotics and Automation,San Francisco,

CA,

April 1986, pp. 1449-1454.9. H. P. Nii, "Blackboard systems: Blackboard applicationsystems, blackboard systems

from aknowledge engineeringperspective(Part Two)", AIMag. Aug. (1986) 82-106.10. H. P. Nii, "Blackboard systems: The blackboard model of problem-solving and the

evolution of blackboard architectures (Part One)", AI Mag. Summer (1986) 38-53.11. E. R. Addison and B. D. Leon, "A blackboard architecture for cooperating expert

systems in managing a distributed sensor network", in Proc. 1987 Tri-Service DataFusion Symposium, Laurel, MD, June 1987,pp. 671-675.

12. J. R. Macßae and C. D. Byrne, "Connectionism applied to areal time expert systemfor tactical datafusion", in Proc. Third Annual Expert Systems in Government Conf.,Washington, DC, Oct. 1987, pp. 66-71.

13. S. D. Taylor, S. P. Roth and J. F. Gilmore, "Implementationof a genericblackboardarchitecture", in Application of Artificial Intelligence V, Proc. SPIE, Vol. 786, 1987,pp. 116-124.

14. A. Brogi et al., "An expert systemfor datafusion based on ablackboard architecture",in Proc. Bth Int. Workshop on Expert Systems and theirApplications, Avignon, France,May 1988, pp. 147-165.

15. R. T. Naylor et al., "Battlefield data fusion", in Application of Artificial Intelligenceto Command and Control Systems, Ed. C. J. Harris, Peter Peregrinus Press, London,1988, pp. 212-233 (chp. 3.3).


16. J. P. Rosenking and S. P. Roth,5"REACT: Cooperating expert systems via a black-

board architecture", in Applications of Artificial Intelligence VI, Proc. SPIE, Vol. 937,1988, pp. 143-150.

17. R. Zanconato, "BLOBS —An object-oriented blackboard system framework for rea-soning in time", in BlackboardSystems,Eds. R. S. Engelmoreand T. Morgan,Addison-Wesley, Workingham, UK, 1988,pp. 335-345 (chp. 16).

18. S. W. Barth, S. A. Barrett and K. H. Gates, "A blackboard architecture for identi-fication of command and control operations nodes", in Proc. 1989 Tri-Service DataFusion Symposium, Laurel, MD, May 1989,pp. 385-392.

19. J. R. Miller and J. H. Swaffield, "DACORR II: An operational user, hierarchical,knowledge-basedtactical situation support system", in Proc. 1990 JointService DataFusion Symposium, Laurel, MD, May 1990,pp. 511-518.

20. L. Uhr, Multi-Computer Architecturefor Artificial Intelligence, John Wiley and Sons,New York, 1987.

21. J. Montgomery, "Providingexplanationfacilities for a real-time blackboard system",in Proc. UK Intelligent TechnologyConf., London, 1988,pp. 15—18.

22. R. Davis and R. G. Smith, "Negotiationas a metaphor for distributedproblem solv-ing", Artif. Intell. 20, 1 (1983).

23. S. Conry, R. Meyer and V. R. Lesser, "Multistage negotiationsin distributedplan-ning", in Readings in Artificial Intelligence, Eds. A. H. Bond and L. Gasser, Morgan

Kaufmann,

San Mateo,

CA,

1988.24. R. Wesson et al., "Network structures for distributed situation assessment", lEEE

Trans. Syst. Man Cybern. 11, 1 (1981).25. V. R. Lesser and D. D. Corkill, "Functionally accurate, cooperative distributedsys-

tems", lEEE Trans. Syst. Man Cybern. 11, 1 (1981).26. B. C. Moxon, "A multiprocessor-basedsensorfusion softwarearchitecture", Automated

Inspection and High Speed Vision Architectures, Proc. SPIE, Vol. 849, 1987.27. R. T. Antony, "Eight canonical forms of fusion: A proposed model of the datafusion

process" , Proc. 1991 JointService Data Fusion Symposium, Johns Hopkins UniversityApplied Physics Laboratory, Laurel, MD, Oct. 1991.

28. R. T. Antony, "A hybrid spatial/object-oriented DBMS to support automated spatial,hierarchical and temporal reasoning", Advances in SpatialReasoning, Vol. 1, Ed. SuShing Chen, Ablex Press, NJ, 1990, chp. 3.

29. R. T. Antony, "A generalizeddatafusion process model and database implications",in Proc. 1989 Tri-Service Data Fusion Symposium, Johns Hopkins UniversityAppliedPhysics Laboratory, Laurel, MD, May 1989.

30. J. Llinas, "Assessing theperformance of multisensor fusion systems", presented at theSPIE Sensor Fusion Conference, Boston, MA, Nov. 1991.

31. J. Llinas and E. Waltz, Multisensor Data Fusion, Artech House Publishers, Norwood,MA, Sept. 1990.

32. J. Llinas, "Datafusion technology forecast for C3and MIS systems", presented at theThird lEE Int. Conf. on C 3MIS Systems, Bournemouth, England, May 1989.

Received January 1991; revised June 1992.


J. Llinas works bothas a regular employee ofCalspan Corporation, adefenseresearch organi-zation in the USA andas a free-lance lecturerand consultant to de-fense and technology-oriented corporationsaround the world.

In his role as Executive Engineer atCalspan, Dr. Llinas both develops andmanages programs in datafusion and relatedtechnologies (e.g. artificial intelligence, opti-mization techniques, etc.). Hebrings approxi-mately 20 years of experience in data fusiontechnology to his work activities. He is aninternationally-recognized expert in data fu-sion, has recently authored what is generallyagreed to be the first integrated book on thesubject, and has lectured internationally forabout eight years onthis topic. He is a Tech-nical Advisor to the U.S. DoD/Joint Directorsof Laboratories DataFusion Panel, a positionhe has held for eight years. His experienceincludes applying this technology to differentproblemareasranging fromSIGINT to NCTIto ATR and to broad C3applications. Hehasalso served as Technical Advisor to the USAFAWACS SPO on data fusion technology in-sertion to the E-3A, and provided oversightsupport to various E-3A data fusion studiesover a periodof three years.

His research interests are currently focusedon methods of multisensor management, au-tomatedreasoning and spatialreasoning tech-niques, hidden Markovmethods, and NATOdata fusion and C3applications.

Dr. Tilinas is currently managing two AirForce datafusionprograms, one involving ex-ploratoryresearchon ground-based, multisen-sor detection of low-observable air targets,and the other involving the fusionofon-boardand off-boardinformation tosupport trackingand identification functions.

Richard Antony hasbeen involved in datafusion research since theearly 1980s. His pri-Mary interests are (1)hierarchical, temporaland spatial reasoning,(2) the design of highperformance hybrid spa-tial/semantic oriented

databases, and (3) the fundamental under-pinnings of the data fusion process. He wasthe Army member of the Joint Directors ofLaboratories Data Fusion Subpanel for fiveyears. He has recently prepareda monographon data fusionas a Research Fellow at the C3ICenter of George Mason University in Fair-

fax,

VA and is currently a research computerscientist at the U.S. Army CECOM Intelli-gence and Electronic WarfareDirectorate. Hereceived his B.S. in electrical engineering in1968 and his M.S. in communications theoryin 1972 from the University of Maryland.

situation/threatfeedforward/backwardfw788gw6072/fw788gw6072.pdf · j.llinas calspan corporation,p....

Documents