sami: situational awareness from multi-modal input naveen ashish

SAMI: Situational Awareness from Multi-

modal Input

Naveen Ashish

Talk Organization Why are we at RESCUE interested ? Situational Awareness (SA)

– Introduction System architecture Research challenges Expected outcomes and artifacts Extraction system demonstration

Team

Naveen AshishSharad MehrotraNalini VenkatasubramanianUtz WestermannDmitry KalashnikovStella ChenVibhav GogatePriya GovindarajanRam HariharanJohn HutchinsonYiming MaDawit SeidJay LickfettChris DavisionQuent Cassen

Bhaskar RaoMohan TrivediRajesh HegdeSangho ParkShankar Shivappa

Ron EguchiMike Mio

Jacob Green

Information from Various Sources

People/Victims at disaster

Emergency responders

News, video, audio footage

GIS, satellite imagery, maps

Pushing “Human-as-sensor”

More Data ≠ More Information

SA

Where are the fire personnel ?Have all medical supplies reached ?What areas should we start evacuating first ?

Situational Awareness Wide variety of fields

– Beginning in mid-80s, accelerating thru 90s– Fighter aircraft, ATM, Power plants,

Manufacturing Definitions

– "the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future"

– "the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take"

Situational awareness and decision making Areas

– Cognitive science– Information processing– Human factors

Knowing what is going on

Abstraction of Information

Multimodal Input: Text, Audio, Video

Events

Awareness

First-cut Architecture

EVENT BASE

Qu

ery

ing

an

d

A

na

lysi

s

Graph View

VISUALIZATION and USER INTERFACES

SpatialIndexingPDF Histogram

KNOWLEDGE: ONTOLOGIES

Text

Audio

Video

Internet

RAW DATA

EVENT EXTRACTION

REFINEMENTDisambiguationLocation

Centered around EVENTS as fundamental abstractions

Research Areas

Event ModelingEvent Extraction

Disambiguation

Location Uncertainty

Graph Analysis

GIS Querying

Event Modeling What is an event ? Event Representation

RELIABILITY

PEOPLE EVACUATION

LOCATION

TIME

REPORT

TYPE

NAME LOCATION

AGENCY

FROM TO

OPERATION

NUMBER

Domain Knowledge

Captured as Ontologies

ROAD EVACUATION

EVACUATION

AIREVACUATION

IS-A

IS-A

THAILAND

SOUTHERN REGION

…….

PHUKETPHUKET,CHANGWAT

Event Extraction Long history of information extraction

– IR (MUC efforts)– Web data extraction

DARPA ACE– Entities, Relations, Events– Events in 2004

Event extraction accuracy is still low SA Domain

– Stream of information– Duplicated, ambiguous– Reliability– Conversations

Modalities– Text

Semantics Driven Approach

Semantics Driven Challenges

– Framework– Ontologies

What semantics required for event extraction ? Application

With NLP, ML techniques Performance

– SA specific Duplicates, reconciliation, temporal,

conversations …..

Disambiguation

– point-location

in terms of landmarks

uncertain, not (x,y)

– reasoning on such data support all types of queries

Uncertainty is a Challenge

Report 1: “... a massive accident involving a hazmat truck on I5-N between Sand Canyon and Alton Pkwy ...”

Report 2: “... a strange chemical smell on Rt133 between I405

and Irvine Blvd ...”

Report 1 R

epor

t 2

Information and Computer Science

There are many challenges when dealing with text.One of them is uncertainty.Often people describe locations in terms of landmarks, they do not give you the exact (x,y) coordinates of the events. Those descriptions are uncertain.For example, the first report says that the accident is somewhere on thisroad and the second report tells that there is a strange smell somewhere on this road.In reality, these two report might refer to the same accident that happenedsay here.So our goal is to be able to reason on top of such data.

Implications of Uncertainty in Text How to model uncertainty?

– probabilistic model– P(location | report)

e.g. report says “near building A”

Queries– cannot be answered exactly...

use probabilistic queries all events: P(location R | report)

> 0– SA requirements

triaging capabilities fast response

– top-k – threshold: P(location R | report) >

-RQ, k-RQ, k -RQ

How to map text to probabilities?– use spatial ontologies

AB

R

Information and Computer Science

There are several implications that arise from uncertainty in text.First, we need to decide how to model spatial uncertainty. We usea probabilistic model for that. Specifically, we are interested inthis conditional probability: given a report about an event, what the location of that event is. A report can describe an event location, for instance, as “near building A”.Finally, observe that queries cannot be answered exactly (...)The solution is to use probabilistic queries. For example a probabilistic version of a range query is to retrieve all the objectsthat have non zero probability to be inside a given range.In addition, SA applications for crisis response have specific requirement for queries. Namely, triagingcapability should be provided for filtering out onlyimportant information. Also the solution should scaleto large dataset and be very efficient in terms of query response time.To support the desired functionality we will considertwo enhancements of probabilistic queries: top-k andthreshold. The top-k enhancement of a spatial query returns only at most k elements from the result setthat have the highest probability to be in the result set.The threshold enhancement returns onl ...OK, probabilistic model is fine, but how do we map text into probabilitiesin the first place. Our solution is to use spatial ontologies for that.

Graph Analysis

GAAL Inherent spatio-

temporal properties Graphs are powerful

for querying and analysis

Current FGDC Search

GIS Search

Progressive Refinement of Data

GIS Search

Deliverables, Outcomes, Artifacts “Vertical” thrusts

– Event extraction system (TEXT)– Disambiguation system– GIS search system

Overall system demonstration ? “By-products”

– Ontologies Computer science research areas

DatabasesSemantic-Web

Information Retrieval

Intelligent Agents (AI)

Thank you !

http://sami.ics.uci.edu

sami: situational awareness from multi-modal input naveen ashish

Documents

disambiguation slide

changwat slide

video events awareness

event extraction accuracy

operation number slide

information sa

events events

location agency