a practical approach to the development of ontology-based information fusion systems juan...
TRANSCRIPT
A practical approach to the development of ontology-based information fusion
systems
Juan Gómez-Romero, Miguel A. Serrano, Jesús García, Miguel Á. Patricio, José M. Molina
Applied Artificial Intelligence GroupUniversity Carlos III of Madrid, Spain
Summary
• Ontologies– What are ontologies?– Why should we care?– How can they be exploited?– Are there any successful experience?– How can we contribute?
Outline
• Motivation• Knowledge representation and
reasoning with ontologies• Ontologies for HLIF in the maritime
domain• Proposed architecture• Implementation• Discussion, conclusions and future
work
Outline
> Motivation•Knowledge representation and reasoning with ontologies•Ontologies for HLIF in the maritime domain•Proposed architecture•Implementation•Discussion, conclusions and future work
1. Motivation
• Information Fusion– "theories and methods to effectively
combine data from multiple sensors and related information to achieve more specific inferences that could be achieved by using a single, independent sensor." (Llinas and Hall, 2009)
1. Motivation
• Low-level data fusion– To process multi-sensor signals to
estimate objects properties• Tracking: data acquisition, collection,
spatial and temporal alignment• Video-based tracking: estimate the kinetics
of scene objects in a video sequence– Surveillance
1. Motivation
• High-level information fusion– To obtain a symbolic
description of the qualitative relations between the objects in the scenario• Actions, intentions, threats, etc. >
Situation assessment
1. Motivation
• High-level information fusion– Understand the scene, evaluate
threats, support decision making– Purely numerical techniques are
insufficient• Cognitive abilities• Complex and unpredictable world behavior
1. Motivation
• High-level information fusion– Flexible and dynamic situation models– Context exploitation
– Symbolic formalisms to represent and reason with high-level information: abstract scene objects and relations
Outline
• Motivation> Knowledge representation and
reasoning with ontologies• Ontologies for HLIF in the maritime
domain• Proposed architecture• Implementation• Discussion, conclusions and future
work
2. Knowledge representation and reasoning with
ontologies• Ontologies
– Knowledge model that describes the objects in a domain by means of a language that can be automatically processed• Description Logics (DLs) representation• Proposed to be the language for
metadata representation in the Semantic Web
2. Knowledge representation and
reasoning with ontologies• Advantages– Interpretability
• High level symbolic description– Interoperability
• Agreed representation of fusion information– Scalability
• Promote extension and reuse– Formal
• Reasoning with logic-based formalisms– Tools
• OWL standard, reasoning engines, programming interfaces, …
2. Knowledge representation and
reasoning with ontologies• Representation primitives– Concepts
• Vessel, HarborZone– Relations
• hasFlag, insideOf– Instances
• vessel_1, nearShoreZone– Axioms
• Vessel and (hasFlag some (Flag and (belongsTo some AlliedCountry)) subclassOf FriendVessel
• transitive(insideOf)• (vessel_1, nearShoreZone: insideOf)
2. Knowledge representation and
reasoning with ontologies
2. Knowledge representation and
reasoning with ontologies• Reasoning– Satisfiability / consistency
• An axiom is satisfiable if it is not a contradiction to the remaining axioms
– Subsumption• A (super-) concept includes a (sub-) concept
– Equivalence• Two concepts include the same instances
– Disjointness• Two concepts do not have any common instance
– Instance checking• An instance belongs to a class
Outline
• Motivation• Knowledge representation and
reasoning with ontologies> Ontologies for HLIF in the
maritime domain• Proposed architecture• Implementation• Discussion, conclusions and future
work
3. Ontologies for HLIF in the Maritime Domain
• Situation and threat assessment in the harbor surveillance scenario– Detected and estimated vessel information from
VTS• Position, AIS identification
– Context knowledge• Restrictions to the fusion process
– Is the situation plausible?• Enrich available information
– Link to external information sources
– Normalcy models• Harbor navigation restrictions• Expert knowledge about threats
3. Ontologies for HLIF in the Maritime Domain
• Knowledge representation – Terminological knowledge base to describe
harbor elements, actors and context• Concepts, relations, axioms (GCIs)
– Geometrical elements of harbor– Vessel classification– Rules of operation
– Assertional knowledge base to represent current instances of the harbor entities and relevant contextual information
• Instance axioms– Static– Dynamic
3. Ontologies for HLIF in the Maritime DomainTerminological Knowledge Base: Harbor areas
divided_into
extends_from
extends_to
isA
isA isA isA
isA isA isA
delimited_byadjacent_to
partially_overlaps
3. Ontologies for HLIF in the Maritime Domain
• Qualitative spatial knowledge management– Zone boundaries and vessel position are
represented according to their relative situation– RCC (Region Connection Calculus)
• Logic theory for qualitative spatial knowledge representation and reasoning
– disconnected, externally connected, partial overlap, tangential inner part, etc.
• Cannot be fully represented with ontologies• Supported by reasoning engines
3. Ontologies for HLIF in the Maritime Domain
• Assertional knowledge base (factual)– Individuals representing current instantiation
of the terminological model
• Scene interpretation is a model-construction procedure
• Instances representing more abstract entities are inferred from instances representing concrete measures
> From sensor-based data to situation assessment
3. Ontologies for HLIF in the Maritime DomainIndividual: vessel1
Types:
PowerDrivenVessel
Facts:
inside_of middle_harbour,
has_property vessel_size
Individual: vessel1_size
Types:
Size
Facts:
size vessel_size_value
Individual: vessel_size_value
Types:
AbsoluteFloatValue
Facts:
val 25.0f
3. Ontologies for HLIF in the Maritime Domain
• Rules of operation• Any vessel inside an area with restricted
speed moving at a speed under the speed limit is normal
– Classes describing the normal behavior– Instance checking can be used to
classify a vessel as threatening or non-threatening according to the “normal behavior” classes
3. Ontologies for HLIF in the Maritime Domain
3. Ontologies for HLIF in the Maritime Domain
• Rules of operation– Vessels that do not accomplish any normalcy
rule are not classified as non-threatening• It is easy to describe normal scenarios according to
harbor rules• Better supported by ontologies that abnormalcy
models– Open World Assumption: the knowledge in an ontology is
incomplete» Default reasoning is not performed
• Any entity not classified as normal remains in an unknown state
3. Ontologies for HLIF in the Maritime Domain
Individual: vessel1
Types:
PowerDrivenVessel
Facts:
inside_of middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 4.0f
Individual: vessel1
Types:
PowerDrivenVessel,
NoSpeedViolation, SafeVessel
Facts:
inside_of middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 4.0f
reasoner
3. Ontologies for HLIF in the Maritime Domain
Individual: vessel1
Types:
PowerDrivenVessel
Facts:
inside_of middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 6.0f
Individual: vessel1
Types:
PowerDrivenVessel,
owl:Thing
Facts:
inside_of middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 6.0f
reasoner
Trigger abductive reasoning
3. Ontologies for HLIF in the Maritime Domain
• Abductive reasoning– Takes a set of facts as inputs and finds a
suitable hypothesis that explains them• See whether inconsistency is result of low-quality
observations, or this vessel exhibits a possible threatening behavior
– Increase threat level
– Not directly supported by ontologies• Monotonic formalisms –do not allow adding or
retracting knowledge while reasoning– Reasoning engines include extensions to allow
abductive rules (not uncertain)
3. Ontologies for HLIF in the Maritime Domain
• Reasoning under uncertainty– Additional reasoning layer
• BAS (Belief Argumentation System)– Combination of symbolic logic and belief theory
» Compute beliefs supporting or rejecting a hypothesis (e.g., vessel features or spatio-temporal relations with other vessels and zones)
• Probabilistic ontologies– Reduction to Bayesian inference
Outline
• Motivation• Knowledge representation and
reasoning with ontologies• Ontologies for HLIF in the maritime
domain> Proposed architecture• Implementation• Discussion, conclusions and future
work
4. Architecture
Multi-camera Radar ...
Multi-source tracking AIS
Spatial Reasoning
...
Tracking and Sensor Data
Scene Objects
Activities and Situations
Threats and assessment
Reco
mm
enda
tions
Heuristics Common-sense knowledge Expert knowledge
OntologyReasoning
High-level decision- ready knowledge
recommendationsin
terpretatio
ns
Scene model
Outline
• Motivation• Knowledge representation and
reasoning with ontologies• Ontologies for HLIF in the maritime
domain• Proposed architecture> Implementation• Discussion, conclusions and future
work
5. Implementation
• Two layers– General tracking layer
• Numerical measures (Desnsity functions, movement vectors)
– Ontology-based contextual layer• Tracking representation• Contextual data• Symbolic reasoning
5. Implementation
• Tracking layer– Four modules that run in sequence– Each module has a set of interchangable
algorithms– Input: Frames– Output: Track and features
5. Implementation
• Association module– Fuzzy Region Assignment (FRA)
• Low level granularity, image segmentation operations
• Medium level granularity, smoothness criteria on target features
• High level granularity, constraints on tracking continuity
5. Implementation
– Fuzzy Region Assignment (FRA)• Bayesian formulation to determine when a blob is
related with a track• Heuristic function to update the track situation
and dimensions• Fuzzy rules derived from experimentation to
define the final group
5. Implementation
• Context layer– Set of ontologies organized according to the JDL model
• Tracking entities (TREN) • Scene object (SCOB) e.g. Restricted area, Person, Vessel• Activities (ACTV) e.g. Threatening behaviours• Situation assessment (IMPC) e.g Emergency
– Inputs managed through the OWL 2 API• Context knowledge given by users or previous executions• Sensor data (Video tracking)
– Ontologies are instanced in the RACER reasoner• Abductive nRQL rules and independent RCC implementation
– Scene interpretation and recommendations as output
5. Implementation
• Context layer and mass storage– Timestamps / snapshots allow temporal
dimension• “A vessel stopped in a restricted area during the last
ten intervals”
– Delays in the overall system• Query search through a larger number of axioms
– Compromise between data storage and query performance
– Temporal window
5. Implementation
• Scalability - Dynamic RCC– Aims
• Representation and reasoning with qualitative spatial properties
• Efficient update of the spatial properties of the objects
– Architecture• Knowledge base (SCOB spatial features)• Optimized geometric model: Geometric model (JTS with
OpenGIS) and a data structure• RCC implementation to store the qualitative spatial
relationships (RACER substrate)
5. Implementation
• What is the problem?– A full topological analysis has a
quadratic complexity– Choose only candidate
geometries that could modify the spatial relations
• How?– Querying the auxiliary data
structure
• Advantages– Topological relations of a
geometry is obtained by checking only a few geometries
5. Implementation• Video examples
– Scene annotation
– Object identification
– Tracking enhancement
– Scene recognition
http://www.giaa.inf.uc3m.es/miembros/jgomez/et/
Outline
• Motivation• Knowledge representation and reasoning
with ontologies• Ontologies for HLIF in the maritime domain• Proposed architecture• Implementation> Discussion, conclusions and future
work
6. Discussion, conclusions and future work
• Ontologies for high-level fusion– Cognitive model
• Symbolic representation of the world– Formal knowledge model
• Representation• Reasoning
• Ontologies in the maritime domain– Heterogeneous information
• Vessel Traffic Systems, AIS• Security protocols• Harbor areas and navigation restrictions
6. Discussion, conclusions and future work
• Ontological model– Categorization of interesting entities
• Vessels (Temporal properties)• Harbor areas (Spatial features)
– Normalcy models• Normal categories of behaviors
– Abduction and uncertainty management• Extended rule-based reasoning• Belief-based argumentation (BAS), Bayesian
networks
6. Discussion, conclusions and future work
• Architecture and implementation– Low-level fusion layer
• Tracker– High-level fusion layer
• Cognitive scene model– Reasoner– Spatial module
Video-surveillance applications
6. Discussion, conclusions and future work
• Future work– Practical implementation in real domains (harbor!)
• Multiple information sources– Acquisition– Integration
• Expert knowledge– Acquisition– Representation
• Real-time demands• High reliability
– Specific features• Uncertain and imprecise knowledge• Interfacing with human users
Questions, comments?
Juan Gómez-Romero, Miguel A. Serrano, Jesús García, Miguel Á. Patricio, José M. Molina
Applied Artificial Intelligence GroupUniversity Carlos III of Madrid, Spain