e c representation and exploitation of contextual knowledge in maritime surveillance galina rogova...
TRANSCRIPT
EC
Representation and Exploitation of Contextual
Knowledge in Maritime Surveillance
Galina RogovaEncompass Consulting
Jesús GarcíaUniversity Carlos III of Madrid
EC
Outline1. Introduction2. Piracy Problem3. Maritime surveillance4. Fusion in Maritime Surveillance
– Level 1– HLF
5. Context in Information Fusion6. Reasoning and uncertainty7. Ontological representation
– Ontology categorization– Knowledge representation
• Conclusions
EC
Piracy Threat
Immanent Threat(Tri-partite Whole)
Opportunity* :• limitless range of vulnerable targets• enormous volume of commercial ships• the need for ships to pass through congested (and ambush-prone) maritime choke points
•vast territorial waters• skeleton crews• limited resources for monitoring territorial waters and ports
•lack of international laws
Intent: Goals (profit) Objectives Directives
*Rand Corporation testimony, 2009, A. Dali, Piracy attacks in the Malaaca strait.
Opportunity
Inte
nt
Cap
ability
Potential Threat(two-part whole)
Inte
nt
Cap
ability
Opportunity
EC
Piracy Threat* (cont’d)Opportunity (cont’d):•lax naval and coastal security•corruption and easily compromised judicial structures
• situation in Somalia• ready willingness of ship owners to pay ransom• limited inter-government cooperationCapability:
• supply of automatic weapons available thanks to the proliferation of small arms•well organized crime/terrorist rings possesseing:
ability to manufacture false identity papers for the crew and vessel and fake cargo documentsa broker network to sell the stolen goods general poverty
*Rand Corporation testimony, 2009, A. Dali, Piracy attacks in the Malaaca strait.
Immanent Threat(Tri-partite Whole)
Opportunity
Inte
nt
Cap
ability
Potential Threat(two-part whole)
Inte
nt
Cap
ability
Opportunity
EC
Piracy and Terrorism: Differences
•Different goals
–Piracy is an economically driven phenomenon
–Terrorism aims at undermining the oceanic environment to secure political, ideological or religious imperatives
• Piracy specific consequences: undermining and weakening government legitimacy by encouraging corruption among elected officials and bureaucrats.
• Pirates are not martyrs, which affects the attack implementation and means, e.g. no suicide attacks, no bomb building capabilities
• May require specific surveillance means
EC
Piracy and Terrorism: Commonality
• Possible common consequences:– a direct threat to the lives and welfare– direct economic impact in terms of fraud, stolen cargos and
delayed trips, which could undermine a maritime state’s trading ability
– major environmental catastrophe• Similar opportunities due to similar limitless range of vulnerable
targets:– enormous volume of commercial ships– necessity to pass through congested (and ambush-prone)
maritime choke points
*A Dali
EC
Piracy and Terrorism: Commonality (cont’d)
• Common tactics are possible, e.g. terrorists could hijack ships carrying huge loads of highly flammable natural gas to undertake a suicide mission or to ram a hijacked vessels against the cruise center, the container terminal or an oil refinery
• Piracy and terrorist attacks are carefully planed and orchestrated, both types require significant resources and well organized crime/terrorist rings.
• Willingness to risk their lives although often with different goals
EC
Piracy and Maritime security
Piracy problem is to a great degree similar to the terrorism problem and represents a part of the general problem of maritime security.
Surveillance technology developed in the maritime security domain can be adopted to fight piracy
EC
Maritime Surveillance: Why and How
“In a world where small water craft can be turned into weapons against navy destroyers and pirates can hold ships for ransom, surveillance of the sea is of increasing importance.”Potential capability monitoring requirements: Small arm trafficking (capability monitoring)Pirate organization network (intent monitoring)Immanent threat monitoring:Monitoring of high vulnerability areas (ports, navigation routes)Realized threatSecondary underlying threat
*http://www.nurc.nato.int/research/msa.htm
EC
Maritime Surveillance
“Surveillance resources are inadequate to monitor the world’s shipping channels tools that help maritime surveillance analysts identify suspicious activity are extremely valuable.” *
*http://www.nurc.nato.int/research/msa.htm
Overall goal:
Alert decision makers on vessel behaviors of interest with minimal false alarm
EC
Data & Information sources(examples)
– Vessel Traffic Systems (VTS) – Port Traffic Management System (PTMS)
• Special sensors and systems for oil and gas ports increase the safety of vessel transit and berthing.
– AIS– Civilian and military sensors– Coastal patrols– Unmanned aerial vehicles (UAVs), – Unmanned surface vehicles (USVs)– Spot reports– Visual sightings– General communication reports from coastal patrols.
• Ex: shore-based emergency communications, search and rescue service, etc– Documents, procedure– Open source information (web-based, radio,…)
– Ref.
EC
Data and Information Categories
• Observables: obtained by persistent surveillance, i.e. continuous tracking and tracing of vessels with observations systems Ex.: statements about the size of vessels and ships (large,
small), speed (slow, high), and track behavior (loitering, stopped, and continuously ahead).
• A priori knowledge including vessel characteristics (size, speed…), current practices and trade activities about information about recent activities of groups and persons and events .
• Learned knowledge• Contextual information.
12
EC
Maritime Surveillance (Challenges)
•Large and heterogeneous areas•Enormous volume of commercial ships •Diverse operations and Decision makers •Huge amount of Information of variable quality• Dynamic and unpredictable environment•Heterogeneous sources of information (sensors, human reports)•Delay in data transmission
13
EC
Fusion and Surviellance
L2,3: Situation & threat assessment:Where, what, who
Context
L1: continuous detection, ID, tracking and tracing of vessels with observation systems
Observables by persistent surveillance•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports
Knowledge base: RulesA priori beliefs…
EC
Situation and Threat assessment
• Higher Level Fusion (situation and threat assessment) produces knowledge about the state of the environment by evaluating relations between entities, their aggregations, characteristics and behavior within a specific context
• HLF provides multiple decision makers with answers to the questions such as: • What is going on• Is anything unexpected or suspicious going on?• Where?• What are the possible explanations (what does it mean)?• What is the impact (what can be expected in the future)?
EC
Challenges: Situation and Threat Assessment
How to transform data and information into knowledge and deal with:– complex distributed system modeling
– scalability problem
– time constraints and time delays
– communication within and between systems, decision makers, ,various agencies, countries,…
– secure information exchange
– heterogeneous data, e.g. structured (databases) and unstructured data (text messages)
– designing a mechanism for inferencing from a given state of knowledge to a possible explanation for a hypothesized situation
– Taking into account information quality (reliability, trust, relevance)
– Uncertain and unknown context.
EC
Context
The problem“Given an entity of interest (a physical object, a situational item, and an event) what context or a sequence of contexts can be formed , such that a task about this entity can be accomplished.”*
The Merriam-Webster dictionary: “The events or circumstances that form, or influence, the environment, within which something exists or takes place.”
L. Gong, 2005
EC
Context
18
Context*
“Context-of X”a situation of interest containing a set of entities and relations between them and providing constraints and expectations about X
“Context-for X”items externally related to X and selected to help better understand and manage a given situation.
X (“reference items”) represents any physical or conceptual entity and event to be considered, e.g. a boat moving towards a cruse ship.
* L. Gong, 2005 Steinberg & Rogova (2008)
How to choose context-of X?
How to represent context-for X?
EC
“Context Of” (CO)
•Guides response to a situation (“what can be done given a certain context” ).• Provides information for detection of possible underlying threat.• Offers dynamics of the situation (in the context of he willingness to pay ransom)• Facilitates effective communications between actors by constraining the meanings of messages.•Constrains goals, objective, functions, actions of the situation responders Example: the geophysical and geopolitical world situation such as situation in Somalia, relations between different countries in Malacca Strait.
CO is characterized by “problem variables” (entities, relationships, and activities).
CO is either given, estimated, or discovered
EC
“Context For” (CF)• CF is used for:
– Action optimization– Better situation understanding
• CF is characterized by “contextual variables” (auxiliary)
• May be static e.g. maps, or dynamic e.g. weather or geopolitical situation.
• CF is either selected or obtained by direct observations
EC
Context in Fusion and Situation managements
•Re-planning•Response•Prevention•Resource management
Situation Management
FUSION PROCESSES
•Objects•Relations•Vulnerabilities•Situations•Threats
L2
L1
L3
L4
CONTEXTS
•Contextual variables•Constrains for:
–Objects–Processes,–Preferences–Hypothesis
•Actions•Decisions•Situational
items •Predicted potential threat•Immanent threat•Realized Threat
Context:•Discovered •Estimated•Predicted
Operational knowledge•Goals, •Objectives•Functions•Plans, Actions•Operational requirements
Domain ontology
Observations•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports
Expected critical situations• Characteristics• Planned actions• Required resources
Knowledge base: •Rules•A priori beliefs…
Constrains for:•Goals•objectives•Functions•Actions•Message Meanings•Plans
Constrains for:•Ontology•Observations•Expected situational items, characteristics
Rogova, 2009
EC
Harbor Context
• Multiple areas and operations• Congestion
EC
Malacca Strait Context
http://www.?????? 23
* A. Dali
Piracy and international issues*
•dense trafficking, between neighboring countries with possible tensions between countries and conflicts in the past:
• smuggling of weapons,
•presence of international peace forces,…
•Lack of international piracy laws
•Numerous vessels with ‘normal’ missions
EC
Harbor traffic rules as context for maritime
surveillance• Harbor general rules
– Identification: ships entering/leaving the harbor must have a permission of Harbor Authority: destination, arrival/departure times, passengers/cargo, ...
– Speed limit: The speed limit usually is defined for areas, lower in inner parts and higher outer. Typical values may be 5-10 knots within the harbor areas
– Navigation: there are predefined limited areas for different categories of vessels. Crossing generally forbidden
• Procedures for special vessels
– Some ships are obliged to use pilotage service: overloaded, carrying cargo potentially dangerous or pollution (oil derivatives, chemicals, etc)
– Depending on maneuvering capability, there may be specifications of towage: entering the port, mooring positions, docking places, etc., with a number of tug boats depending on length and draught of vessel.
24
EC
Approach
Framework for contextual
ontology-based reasoning
Anomaly? ORInsufficient Quality of: • Contextual knowledge?• Observations• Fusion processes?
ObjectsRelations
L2L1
L4 L2
Fusion Processes
Domain ontology
Current Context
(CO)
Environment monitoring•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports
Is the belief in anomaly justified?
Expected:• objects• situational items• characteristics• behavior
Assessed:• objects•relations• situational items• characteristics• behavior
no
Constraints for:•Objects•Processes•relations
Consistent?
StatisticsRulesArguments
Constrains for:•Objects•Processes,•Beliefs•Preferences•Hypothesis•Arguments
CF
EC
Dealing with Inconsistency
How and whether to change the state of the knowledge (e.g.
threat/no threat) due to the arrival of new information if new
information contradicts to prior knowledge in the context
under consideration.
Involves:
• methods for detecting inconsistency
• abduction: a means to determine the sources of the contradiction (e.g., whether this contradiction is because of insufficient quality of contextual knowledge, observations, fusion processes)
How and whether to change the state of the knowledge (e.g.
threat/no threat) due to the arrival of new information if new
information contradicts to prior knowledge in the context
under consideration.
Involves:
• methods for detecting inconsistency
• abduction: a means to determine the sources of the contradiction (e.g., whether this contradiction is because of insufficient quality of contextual knowledge, observations, fusion processes)
Anomaly? ORInsufficient Quality of: • Contextual knowledge• Observations• Fusion processes?
Requires reasoning under uncertainty
EC
Inconsistency Detection• May be based on explicit “normalcy” or “anomaly” models by using:
– Values of characteristics or behavior of situational items obtained from databases.
– Rules, e.g. presence or absence of certain characteristics.– Statistical information obtained from databases (e.g. hypothesis
testing)
Problems• Scalability: There are too many patterns of normal behavior • There are less abnormal patterns than normal but the problem with the
“black swan.”
What is normal depends on context (geophysical, vessel type,…)
EC
Inconsistency Detection (cont’d)
Incremental learning operating on evolving data (“on-line stream classification problem”)
–Accumulated statistics
–Neural networks
– Case-based reasoning
Advantages:
Can identify unseen earlier patterns of behavior or characteristics
Drawbacks
A pattern that might be considered as an anomaly, could become normal with time when more information is available.
Anomaly detection triggers the process of believe update to find the source of inconsistency, which requires reasoning under uncertainty
EC
Reasoning Under Uncertainty
Qualitative methods(Uncertainty is handled by manipulation of symbols)
Quantitative methods(Uncertainty encoded by numbers)
• Probability theory–Classical probability (chance)–Bayesian (subjective belief)
• Possibility theory (incompleteness)• Fuzzy theory (vague information)• Evidence theory (ignorance, ambiguity)
–Dempster-Shafer–Transferable belief• …
• Default logic(incompleteness) • Argumentation (inconsistency)• …
Hybrid Methods
Selection of a single formalism of a hybrid system depends on a particular problem
EC
Belief Based Argumentation
Belief Based Argumentation* is an approach to non-monotonic reasoning
under uncertainty combining symbolic logic with belief theories for judging
hypotheses about the unknown world by utilizing given knowledge.
Quantitative partComputes and combines beliefs
that hypotheses and the
arguments, which bear on them,
are valid.
Quantitative partComputes and combines beliefs
that hypotheses and the
arguments, which bear on them,
are valid.
Belief Based Argumentation
Qualitative partFinds arguments in favor and
against a hypothesis about a
possible cause consistent with
observations.
Qualitative partFinds arguments in favor and
against a hypothesis about a
possible cause consistent with
observations.
* Rogova et al (2004,2005) based on R. Haenni, et al, (2000)
* Rogova et al (2004,2005) ,R. Haenni, et al, (2000)
EC
Why Belief Based Argumentation
Allows for:
• contextual reasoning with harbour rules and regulations
• dealing with uncertainty
• reasonng under the open world assumption (incomplet information and non-exhaustive hypotheses)
• incorporating subjective knowlegdeinclude both numeric and symbolic information
EC
Example
Discovery of possible threat from a boat based on:• Boat features (speed, direction, type, flag, etc.)
• Spatio-temporal relations between the suspicious boat and others, or relations between the boat and harbor zones
• Beliefs assigned to assumptions are based on the observed spatio-temporal relations and the correspondence of the boat behaviour to the rules and regulation
32
An argument pro hypothesis “treat” for a vessel under a flag of convenience built as a conjunction of the following uncertain assumptions:
A1: the suspicious boat is too close to a vessel sailing in the opposite direction
A3: The vessel following in the opposite direction is a big cruise ship
A4: The suspicious boat is increasing its speed
EC
Fusion and Surveillance
L2,3: Situation & threat assessment:Where, what, who
Context
L1: continuous detection, ID, tracking and tracing of vessels with observation systems
Observables by persistent surveillance•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reportsKnowledge base: RulesA priori beliefs…
EC
Fusion Challenges: Level l
• Tracking: accurate detection and estimation of targets dynamics • Design aspects: Analysis of appropriate fusion architectures and algorithms in
this environment• Experimental analysis: demonstrate through an operative prototype
– Focus in efficiency, point to the highest computation load to allow acceptable performance in real time.
– Use of contextual information for adjustment to adapt the algorithms to operating conditions
34
EC
Fusion Challenges: Level l
35
SENSOR INPUTS
MULTI-ALGORITHM TRACKING SYSTEM TRACKS
EC
Sensor Fusion in MD• Example: AIS-radar Fusion system
– Distributed architecture with specific logic for each source– Output: global tracks: ID, reference time, WGS84 location,
speed/course over ground
EC
Sample Scenario
• Simulated scenarios and real data analysis with Cape Verde Islands, 7 available sesors (4 RADAR, 3 AIS Stations)
EC
Sensor Fusion in MD• Specific sensor processing
– Radar with clutter, noise, detection– AIS: ID, speed processing, aperiodic
EC
L1 Algorithm design• Local Tracks Management
Cycle plots acquisition
Temporal Spatial Association
Unassociated plots
Local track / plot
associations
Deletion checks
Local tracks creation
Local tracks update
Gating
Plott type ID Match
radar
AIS
EC
L1 Algorithm Design
• Global Tracks Management
Local processing
Inconsistency check
Redundancy check
Empty global tracks check
Splitting process
Global tracks fusing process
Global track deletion
Updated local tracks acquisition
Previously associated local
tracks
New local tracksGating
AssociationGlobal track creation
Global track update
Local to Global track fusion
EC
Geometric context• Parameter tuning
– Initializtaion areas– Speed threshold– Maeuvering index– Management cycles
Plane tessellation
Nº tile: 1 2 3 4 5 ...
Dock area
Low -speed initialization area
Ground areas 2
(xp, yp)
2
EC
Context-aided tracking
• Context-aided tracking: adaptive estimation of target dynamics accordingly to configuration and uncertainty– Ex.: use of known geometrical conditions and sensor performance– Use of expected behaviours
• Examples: radar data processing in airport surface traffic monitorization
42
EC
Context-aided tracking
• Rule-based tracking: rule-based video processing system: split/merge effect– Examples: video in airport surface traffic monitorization, sports, parking
surveillance, …
43
EC
Fusion Challenges: Level l
• Adaptable tracking processes– Data association/correlation (video)– Track initialization / management logic– Appropriate tracking filters (ex. EKF/IMM tuning)– Object dynamics characterization (ex. manueverability)– Cycle management (efficiency)
• Scenario Context – Maps– Regulation (speed, size, …)– Restricted areas– Trafficability maps
44
EC
Knowledge representation: ontologies
Ontology is the study of the kinds of things that exist.Provides formally structured context-dependent information about dynamic reality capable of capturing situational entities and the wealth of relations between them.
Formal upper level ontological analysis to represent Real World in general
Formal upper level ontological analysis to represent Real World in general
Formal Ontology of a
Specific Domain
Domain-specific ontological analysis to represent domain specific characteristics of Real World
Informs Constrains
SituationOntology
Dynamic Environment
Uncertainty
EC
Types of Domain specific Ontologies
• Domain Specific ontologies: elements in the application domain, such as known types of vessels and ports– To acquire data from information sources in a meaningful and consistent
manner
• Situation ontologies. capture a situation or series of states in the application space using concepts from the content ontologies. – Situation ontologies are input to search patterns of interest, e.g. the
specific intents that we want the system to recognize. – Ex.: a search pattern for smuggle might: rendez-vous with another
vessel mid-sea or the vessel type does fit the current location and time.
[] Ref: A. van den Broek FUSION 2011
46
EC
Use of Ontologies for Knowledge Representation in Fusion
– capture domain knowledge to represent objects, objet attributes, and relations between them
– capture domain knowledge not only about about entities and status, but also domain semantics (concepts, relationships, etc).
– Explicit representation of formal semantics such as Description Logics enables automated reasoning
Ontology-based systems usually have their logical basis in any type of classical logic (extensions of rule systems)
47
EC
Representation
• A priori specialized knowledge (terminological axioms)– Geometrical elements of harbor: moving lines, buoys, restricted areas,
etc.– Vessel classification: vessel types and properties– Rules of operation: margin speeds, procedures (crossing only
orthogonal), rules of priority
• Instantiation (extensional axioms)– Properties of actual entities of the current scenario
• Static (e.g., delimitation of harbor zones)• Dynamic (e.g., vessel a position)
48
EC
Ex.: Harbor traffic rules• Harbor general rules
– Identification: ships entering/leaving the harbor must have a permission of Harbor Authority: destiny, arrival/departure times, passengers/cargo, ...
– Speed limit: The speed limit usually is defined for areas, lower in inner parts and higher outer. Typical values may be 5-10 knots within the harbor areas
– Navigation: there are predefined limited areas for different categories of vessels. Crossing generally forbidden
• Procedures for especial vessels
– Some ships are obliged to use pilotage service: overloaded, carrying cargo potentially dangerous or pollution (oil derivatives, chemicals, etc)
– There may be specifications of towage: entering the port, mooring positions, docking places, etc., with a number of tug boats depending on length and draught of vessel
• External procedures– Outer channels: Merchant ships have to proceed following a “safe speed”
avoiding waves to small boats/vessels– Zones considered dangerous for maritime traffic due to military exercises
49
EC
divided_into
extends_from
extends_to
isA
isA isA isA
isA isA isA
delimited_byadjacent_to
partially_overlaps
Representing areas (static)RCC: Region Connection
Calculus
50
EC
Represent vessels (dynamic)
has_track_snapshotis_related_to_track
has_property
isA isA isA
position
isA
isA
has_value
51
EC
Reasoning: Approach
Framework for ontology-based
reasoning Anomaly? ORInsufficient Quality of: • Contextual knowledge?• Observations• Fusion processes?
ObjectsRelations
L2L1
L4 L2
Fusion Processes
Domain ontology
Current Context
(CO)
Environment monitoring•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports
Is the belief in anomaly justified?
Expected:• objects• situational items• characteristics• behavior
Assessed:• objects•relations• situational items• characteristics• behavior
no
Constraints for:•Objects•Processes•relations
Consistent?
StatisticsRulesArgumentsContext
exploitation in specific domain
Constrains for:•Objects•Processes,•Beliefs•Preferences•Hypothesis•Arguments
EC
RACER as reasoning support
• RACER inference engine– Commercial reasoner to reason with Description
Logics ontologies
• Features– Supports standard OWL DL
– RCC-based spatial reasoning (semantics for topological reasoning: inclusion, adjacency, overlapping, etc.)
– nRQL extension• Instance querying• Rule definition
– Deductive rules– Abductive rules
Ontology model
53
ReasoningSupport
EC
Rule Manipulation
• Abductive rules– Include additional individuals in the consequent, which are created as
new instances of the ontology Scene recognition• Interpret what is happening from the basic tracking data• Create upper-level instances from lower-level data: objects from tracks
(correspondence), activities from objects (recognition), impacts from activities (evaluation)
– Tracking feedback• From the interpretation of the scene (instances of the CL model), feedback
rules create instances of recommendations to the tracker
EC55
Information Flow with Ontology Reasoning
EC
Example
• Video scene representation (PETS 2002)– Static and dynamic entities to remove reflecting elements
56
(170, 180)
(165, 105)
(415, 60) (485, 55)
(490, 65)
(485, 140)
(475, 145)
(410, 130)
(445, 100)mirror
mirrored area
mirror1 is a mirrormirrored_area1 is the area of influence of a mirror1mirror1 is in (445, 100)mirror1 area is delimited by points: (415, 60), (490, 65), (475, 145), (410, 143)...position & area of mirror1, mirroredArea1, etc.
EC
Example
• Video scene representation (PETS 2002)– Static and dynamic entities to remove reflecting elements
57
(360, 105)
(25x45) (25x40)
(425, 100)
001 002
track
track
EC
Summary
• General requirement: Modern fusion systems must be adaptive and context-sensitive, customized to application knowledge
• Highlighted some open issues: A general approach to consider context exploitation and uncertainty-based reasoning, related with the specific needs in maritime analysis
• Explained an overall process based on an ontological representation, with threat recognition and abductive reasoning– Support for reasoning with rule manipulation engine
58
EC
Piracy Domain Specific Challenges
• Context challenges: – characterization of variables: completeness vs complexity, plausibility,
relevance, etc– Context discovery– Formal representation– Context dynamics
• Piracy domain specific ontology– incorporating specific rule and regulations– wide areas: interoperability among different countries– specific methods for ontology management (scale)
• Anomaly (normalcy) model: semantic problem
59
EC
Specific Technology Challenges
• Completely integrate the Context Layer and the Input Sensor Layer (Tracking)
• Performance issues• Multi-modal multi-sensor context extension• Incorporating uncertain/imprecise knowledge (both in the input and
the outputs)• Algorithms and architectures, new sensors, knowledge extraction,
user interaction, etc.• Scalability
60
EC
Representation and Exploitation of Contextual Knowledge in
Maritime Surveillance
Galina RogovaEncompass Consulting
Jesús GarcíaUniversity Carlos III of Madrid
MANY THANKS FOR YOUR ATTENTION!!