a practical approach to the development of ontology-based information fusion systems juan...

A practical approach to the development of ontology-based information fusion

systems

Juan Gómez-Romero, Miguel A. Serrano, Jesús García, Miguel Á. Patricio, José M. Molina

Applied Artificial Intelligence GroupUniversity Carlos III of Madrid, Spain

Summary

• Ontologies– What are ontologies?– Why should we care?– How can they be exploited?– Are there any successful experience?– How can we contribute?

Outline

• Motivation• Knowledge representation and

reasoning with ontologies• Ontologies for HLIF in the maritime

domain• Proposed architecture• Implementation• Discussion, conclusions and future

work

Outline

> Motivation•Knowledge representation and reasoning with ontologies•Ontologies for HLIF in the maritime domain•Proposed architecture•Implementation•Discussion, conclusions and future work

1. Motivation

• Information Fusion– "theories and methods to effectively

combine data from multiple sensors and related information to achieve more specific inferences that could be achieved by using a single, independent sensor." (Llinas and Hall, 2009)

1. Motivation

• Low-level data fusion– To process multi-sensor signals to

estimate objects properties• Tracking: data acquisition, collection,

spatial and temporal alignment• Video-based tracking: estimate the kinetics

of scene objects in a video sequence– Surveillance

1. Motivation

• High-level information fusion– To obtain a symbolic

description of the qualitative relations between the objects in the scenario• Actions, intentions, threats, etc. >

Situation assessment

1. Motivation

• High-level information fusion– Understand the scene, evaluate

threats, support decision making– Purely numerical techniques are

insufficient• Cognitive abilities• Complex and unpredictable world behavior

1. Motivation

• High-level information fusion– Flexible and dynamic situation models– Context exploitation

– Symbolic formalisms to represent and reason with high-level information: abstract scene objects and relations

Outline

• Motivation> Knowledge representation and

reasoning with ontologies• Ontologies for HLIF in the maritime

domain• Proposed architecture• Implementation• Discussion, conclusions and future

work

2. Knowledge representation and reasoning with

ontologies• Ontologies

– Knowledge model that describes the objects in a domain by means of a language that can be automatically processed• Description Logics (DLs) representation• Proposed to be the language for

metadata representation in the Semantic Web

2. Knowledge representation and

reasoning with ontologies• Advantages– Interpretability

• High level symbolic description– Interoperability

• Agreed representation of fusion information– Scalability

• Promote extension and reuse– Formal

• Reasoning with logic-based formalisms– Tools

• OWL standard, reasoning engines, programming interfaces, …

2. Knowledge representation and

reasoning with ontologies• Representation primitives– Concepts

• Vessel, HarborZone– Relations

• hasFlag, insideOf– Instances

• vessel_1, nearShoreZone– Axioms

• Vessel and (hasFlag some (Flag and (belongsTo some AlliedCountry)) subclassOf FriendVessel

• transitive(insideOf)• (vessel_1, nearShoreZone: insideOf)

2. Knowledge representation and

reasoning with ontologies

2. Knowledge representation and

reasoning with ontologies• Reasoning– Satisfiability / consistency

• An axiom is satisfiable if it is not a contradiction to the remaining axioms

– Subsumption• A (super-) concept includes a (sub-) concept

– Equivalence• Two concepts include the same instances

– Disjointness• Two concepts do not have any common instance

– Instance checking• An instance belongs to a class

Outline

• Motivation• Knowledge representation and

reasoning with ontologies> Ontologies for HLIF in the

maritime domain• Proposed architecture• Implementation• Discussion, conclusions and future

work

3. Ontologies for HLIF in the Maritime Domain

• Situation and threat assessment in the harbor surveillance scenario– Detected and estimated vessel information from

VTS• Position, AIS identification

– Context knowledge• Restrictions to the fusion process

– Is the situation plausible?• Enrich available information

– Link to external information sources

– Normalcy models• Harbor navigation restrictions• Expert knowledge about threats

3. Ontologies for HLIF in the Maritime Domain

• Knowledge representation – Terminological knowledge base to describe

harbor elements, actors and context• Concepts, relations, axioms (GCIs)

– Geometrical elements of harbor– Vessel classification– Rules of operation

– Assertional knowledge base to represent current instances of the harbor entities and relevant contextual information

• Instance axioms– Static– Dynamic

3. Ontologies for HLIF in the Maritime DomainTerminological Knowledge Base: Harbor areas

divided_into

extends_from

extends_to

isA

isA isA isA

isA isA isA

delimited_byadjacent_to

partially_overlaps

3. Ontologies for HLIF in the Maritime Domain

• Qualitative spatial knowledge management– Zone boundaries and vessel position are

represented according to their relative situation– RCC (Region Connection Calculus)

• Logic theory for qualitative spatial knowledge representation and reasoning

– disconnected, externally connected, partial overlap, tangential inner part, etc.

• Cannot be fully represented with ontologies• Supported by reasoning engines

3. Ontologies for HLIF in the Maritime Domain

• Assertional knowledge base (factual)– Individuals representing current instantiation

of the terminological model

• Scene interpretation is a model-construction procedure

• Instances representing more abstract entities are inferred from instances representing concrete measures

> From sensor-based data to situation assessment

3. Ontologies for HLIF in the Maritime DomainIndividual: vessel1

Types:

PowerDrivenVessel

Facts:

inside_of middle_harbour,

has_property vessel_size

Individual: vessel1_size

Types:

Size

Facts:

size vessel_size_value

Individual: vessel_size_value

Types:

AbsoluteFloatValue

Facts:

val 25.0f

3. Ontologies for HLIF in the Maritime Domain

• Rules of operation• Any vessel inside an area with restricted

speed moving at a speed under the speed limit is normal

– Classes describing the normal behavior– Instance checking can be used to

classify a vessel as threatening or non-threatening according to the “normal behavior” classes

3. Ontologies for HLIF in the Maritime Domain

3. Ontologies for HLIF in the Maritime Domain

• Rules of operation– Vessels that do not accomplish any normalcy

rule are not classified as non-threatening• It is easy to describe normal scenarios according to

harbor rules• Better supported by ontologies that abnormalcy

models– Open World Assumption: the knowledge in an ontology is

incomplete» Default reasoning is not performed

• Any entity not classified as normal remains in an unknown state

3. Ontologies for HLIF in the Maritime Domain

Individual: vessel1

Types:

PowerDrivenVessel

Facts:

inside_of middle_harbour,

has_property vessel_size

has_property vessel_speed

Individual: vessel1_speed

Types:

Speed

Facts:

speed vessel_speed_value

Individual: vessel_speed_value

Types:

AbsoluteFloatValue

Facts:

val 4.0f

Individual: vessel1

Types:

PowerDrivenVessel,

NoSpeedViolation, SafeVessel

Facts:

inside_of middle_harbour,

has_property vessel_size

has_property vessel_speed

Individual: vessel1_speed

Types:

Speed

Facts:

speed vessel_speed_value

Individual: vessel_speed_value

Types:

AbsoluteFloatValue

Facts:

val 4.0f

reasoner

3. Ontologies for HLIF in the Maritime Domain

Individual: vessel1

Types:

PowerDrivenVessel

Facts:

inside_of middle_harbour,

has_property vessel_size

has_property vessel_speed

Individual: vessel1_speed

Types:

Speed

Facts:

speed vessel_speed_value

Individual: vessel_speed_value

Types:

AbsoluteFloatValue

Facts:

val 6.0f

Individual: vessel1

Types:

PowerDrivenVessel,

owl:Thing

Facts:

inside_of middle_harbour,

has_property vessel_size

has_property vessel_speed

Individual: vessel1_speed

Types:

Speed

Facts:

speed vessel_speed_value

Individual: vessel_speed_value

Types:

AbsoluteFloatValue

Facts:

val 6.0f

reasoner

Trigger abductive reasoning

3. Ontologies for HLIF in the Maritime Domain

• Abductive reasoning– Takes a set of facts as inputs and finds a

suitable hypothesis that explains them• See whether inconsistency is result of low-quality

observations, or this vessel exhibits a possible threatening behavior

– Increase threat level

– Not directly supported by ontologies• Monotonic formalisms –do not allow adding or

retracting knowledge while reasoning– Reasoning engines include extensions to allow

abductive rules (not uncertain)

3. Ontologies for HLIF in the Maritime Domain

• Reasoning under uncertainty– Additional reasoning layer

• BAS (Belief Argumentation System)– Combination of symbolic logic and belief theory

» Compute beliefs supporting or rejecting a hypothesis (e.g., vessel features or spatio-temporal relations with other vessels and zones)

• Probabilistic ontologies– Reduction to Bayesian inference

Outline

• Motivation• Knowledge representation and

reasoning with ontologies• Ontologies for HLIF in the maritime

domain> Proposed architecture• Implementation• Discussion, conclusions and future

work

4. Architecture

Multi-camera Radar ...

Multi-source tracking AIS

Spatial Reasoning

...

Tracking and Sensor Data

Scene Objects

Activities and Situations

Threats and assessment

Reco

mm

enda

tions

Heuristics Common-sense knowledge Expert knowledge

OntologyReasoning

High-level decision- ready knowledge

recommendationsin

terpretatio

ns

Scene model

Outline

• Motivation• Knowledge representation and

reasoning with ontologies• Ontologies for HLIF in the maritime

domain• Proposed architecture> Implementation• Discussion, conclusions and future

work

5. Implementation

• Two layers– General tracking layer

• Numerical measures (Desnsity functions, movement vectors)

– Ontology-based contextual layer• Tracking representation• Contextual data• Symbolic reasoning

5. Implementation

• Tracking layer– Four modules that run in sequence– Each module has a set of interchangable

algorithms– Input: Frames– Output: Track and features

5. Implementation

• Association module– Fuzzy Region Assignment (FRA)

• Low level granularity, image segmentation operations

• Medium level granularity, smoothness criteria on target features

• High level granularity, constraints on tracking continuity

5. Implementation

– Fuzzy Region Assignment (FRA)• Bayesian formulation to determine when a blob is

related with a track• Heuristic function to update the track situation

and dimensions• Fuzzy rules derived from experimentation to

define the final group

5. Implementation

• Context layer– Set of ontologies organized according to the JDL model

• Tracking entities (TREN) • Scene object (SCOB) e.g. Restricted area, Person, Vessel• Activities (ACTV) e.g. Threatening behaviours• Situation assessment (IMPC) e.g Emergency

– Inputs managed through the OWL 2 API• Context knowledge given by users or previous executions• Sensor data (Video tracking)

– Ontologies are instanced in the RACER reasoner• Abductive nRQL rules and independent RCC implementation

– Scene interpretation and recommendations as output

5. Implementation

• Context layer and mass storage– Timestamps / snapshots allow temporal

dimension• “A vessel stopped in a restricted area during the last

ten intervals”

– Delays in the overall system• Query search through a larger number of axioms

– Compromise between data storage and query performance

– Temporal window

5. Implementation

• Scalability - Dynamic RCC– Aims

• Representation and reasoning with qualitative spatial properties

• Efficient update of the spatial properties of the objects

– Architecture• Knowledge base (SCOB spatial features)• Optimized geometric model: Geometric model (JTS with

OpenGIS) and a data structure• RCC implementation to store the qualitative spatial

relationships (RACER substrate)

5. Implementation

• What is the problem?– A full topological analysis has a

quadratic complexity– Choose only candidate

geometries that could modify the spatial relations

• How?– Querying the auxiliary data

structure

• Advantages– Topological relations of a

geometry is obtained by checking only a few geometries

5. Implementation• Video examples

– Scene annotation

– Object identification

– Tracking enhancement

– Scene recognition

http://www.giaa.inf.uc3m.es/miembros/jgomez/et/

Outline

• Motivation• Knowledge representation and reasoning

with ontologies• Ontologies for HLIF in the maritime domain• Proposed architecture• Implementation> Discussion, conclusions and future

work

6. Discussion, conclusions and future work

• Ontologies for high-level fusion– Cognitive model

• Symbolic representation of the world– Formal knowledge model

• Representation• Reasoning

• Ontologies in the maritime domain– Heterogeneous information

• Vessel Traffic Systems, AIS• Security protocols• Harbor areas and navigation restrictions

6. Discussion, conclusions and future work

• Ontological model– Categorization of interesting entities

• Vessels (Temporal properties)• Harbor areas (Spatial features)

– Normalcy models• Normal categories of behaviors

– Abduction and uncertainty management• Extended rule-based reasoning• Belief-based argumentation (BAS), Bayesian

networks

6. Discussion, conclusions and future work

• Architecture and implementation– Low-level fusion layer

• Tracker– High-level fusion layer

• Cognitive scene model– Reasoner– Spatial module

Video-surveillance applications

6. Discussion, conclusions and future work

• Future work– Practical implementation in real domains (harbor!)

• Multiple information sources– Acquisition– Integration

• Expert knowledge– Acquisition– Representation

• Real-time demands• High reliability

– Specific features• Uncertain and imprecise knowledge• Interfacing with human users

Questions, comments?

Juan Gómez-Romero, Miguel A. Serrano, Jesús García, Miguel Á. Patricio, José M. Molina

Applied Artificial Intelligence GroupUniversity Carlos III of Madrid, Spain