ONTOLOGICAL REPRESENTATION OF CONTEXT KNOWLEDGE FOR VISUAL DATA FUSION

Juan Gómez RomeroMiguel A. PatricioJesús GarcíaJosé M. Molina

Applied Artificial Intelligence Research Group (GIAA)University Carlos III of Madrid

Objective

Semantic representation of visual information, both perceived and contextual,

to facilitate fusion of hard and soft entriesin surveillance applications

To formalize the heuristics

Sensor-based data vs. Contextual and common-sense knowledge

Outline1. Context in Visual Data Fusion

2. Architecture & Contents of our Model

3. Conclusions and Future Work

Definition of context

Context is “any information (either implicit or explicit) that can be used to characterize the situation of an entity” [1]

Computer Vision Additional information about the scene entities [2]

Scene environment Parameters of the recording Previously computed information

User-provided information (soft entries!)[1] A. Dey, G. Abowd. “Towards a Better Understanding of Context and Context-Awareness,” CHI Workshop on

the What, Who, Where, When, and How of Context-Awareness, The Hague, Netherlands, 2000.

[2] F. Bremond, and M. Thonnat, “A context representation for surveillance systems,” ECCV Workshop on Conceptual Descriptions from Images, Cambridge, UK, 1996.

Necessity of Context Knowledge for High-Level Information Fusion

Track 008pos ()vel ()

Track 010pos ()vel()

Tracking L1 L2-L3

PersonEntry> EnteringMirror> ReflectionColumn

Person 1 is(Entering through Entry 2) and(Reflected by Mirror 1)

Interpretation

User-Provided ContextRepresentation & Reasoning

Proposal

Use of ontologies to represent context knowledge for visual data fusion

Ontologies for Context Management

Ontologies: “Formal, explicit specifications of a shared conceptualization” [3] An ontology is a knowledge model which describes from a

common perspective the objects in a common domain using a language that can be processed automatically.

Based on Description Logics (DLs) DLs are a family of logics to represent structured knowledge Inferences can be performed: consistency, subsumption,

membership, etc.

Basic constructs: Concepts, Relations, Individuals, Axioms

Standard: The Web Ontology Language (OWL)[3] R. Studer, V. R. Benjamins, & D. Fensel. “Knowledge engineering: principles and methods”. In: Data

Knowledge Engineering 25.1-2 (1998). Pp. 161–197.

Proposal

Use of ontologies to represent context knowledge for visual data fusion

Logic-based representation of fusion information Associated reasoning procedures

Abstract description of the scenes Better interpretability and easier interaction with users

Extensible and Reusable: New elements can be easily added to the model The model can be reused (particularly, by generalization &

specialization) in different domains Standard languages and tools

Less effort to deal with the models

Contribution

Ontology-based model to manage contextual and sensorial data in visual fusion systems

Outline1. Context in Visual Data Fusion

2. Architecture & Contents of the Model

3. Conclusions and Future Work

JDL-based architecture

Ontological Model• Descriptive Knowledge (TBox): Definition of concepts, relations, etc.• Intensive Knowledge (ABox): Instantiation for a concrete scene

JDL-based architecture: Inputs (I)

Hard Inputs: Sensor Data1. Tracking data obtained by a (classical) tracking algorithm 2. Identification data3. Non visual sensor data

JDL-based architecture: Inputs (II)

Soft Inputs: Human-generated Data1. Contextual information2. Context-based rules

JDL-based architecture: Outputs

Outputs1. Situation Assessment

2. Impact Assessment3. Visualization of the interpreted situation

JDL-based architecture

From Data to Information: Abductive Reasoning1. Tracking: Moving entities2. Correspondence: Association between possible objects and tracks3. Recognition: Activity identification4. Evaluation: Computation of the impact of an activity

JDL-based architecture: TREN ontology

L1 – TRACKING ENTITES ONTOLOGY (TREND) Ontological representation of low-level data from the tracking algorithm: frames, tracks and track properties Temporal evolution of the tracks: tracks have associated track snapshots Flexible representation of properties: qualia spaces (DOLCE ontology)

JDL-based architecture: SCOB ontology

L1-L ½ -- SCENE OBJECTS DESCRIPTION ONTOLOGY (SCOB) Objects of the scene: entry, exit, person, column, etc. Static (contextual) and Dynamic (tracked) objects Object properties (change in time)

JDL-based architecture: ACTV ontology

L2 – ACTIVITY DESCRIPTION ONTOLOGY (ACTV) Activities of the scene and connections with the objects involved: grouping activity + grouped objects Activities taxonomy largely based on: C. Fernández, and J. González, “Ontology for Semantic Integration in a Cognitive Surveillance System,” 2nd Int. Conf. on Semantic and Digital Media Technologies, Genoa, Italy, 2007, pp. 260-263.

JDL-based architecture: IMPC ontology

L3 -- IMPACT DESCRIPTION ONTOLOGY (IMPC) Abstract description of the impact of activities Impact concept (reification of the hasImpact relation) Impact taxonomies or restrictions according to context could be implemented

Overview of the In-Use Ontological Model

Specific Model Specialization of the template concepts provided in the General Knowledge Model PETS2002 sequence

Abductive Rules Rules with ontological terms to infer information of a higher level from information of a lower level Example:If the distance between two people is decreasing, then they are grouping

Outline1. Context in Visual Data Fusion

2. Architecture & Contents of the Model

3. Conclusions and Future Work

Summary

Ontological model for representing contextual and perceived data for visual data fusion

Formal description of scenes and reasoning, from low-level to high-level (intra-level reasoning)

Logic-based mechanisms (rules) to infer high-level information from low-level data (inter-level reasoning)

Extensible to different applications (e.g. surveillance)

Temporal evolution of the scenes

Future work

Full integration with tracking software Adaptation (simplification) of representation and reasoning

when response time is constrained

Incorporation of different data sources, not only visual

Test and validate results in different application areas Development of ontologies and rule bases

Feedback to the low-level algorithms from the high-level How tracking errors can be detected (or predicted) and

solved when the situation has been identified?

THANK YOU!

QUESTIONS…