CoGenTex, Inc.

Ontology-based Multimodal User Interface

in MOQA

AQUAINT 18-Month WorkshopSan Diego, California

Tanya KorelskyBenoit LavoieTed Caldwell

CoGenTex, Inc.

Goals of MOQA Interface

• To enable the analyst in ergonomic browsing of an information-rich, ontology based repository

• To facilitate justification of search results and production of session reports

CoGenTex, Inc.

Interface Features

• Support for both natural language (NL) queries and structured queries used for validation of NL queries

• Intuitive web-based multimodal interface for answers (search results)

• Tables, text, maps, timeline, and social network graphs interconnected by hyperlinks

• Natural language generation used in answers, answer justification, and for query validation

• Implemented using XML-based technology

CoGenTex, Inc.

MOQA User Interface: Accomplishments since December 2002 Workshop

• Implemented all features shown in mockup at December 2002 Workshop

• Incorporated all the feedback given at Workshop

• All the modalities in place, including integrated third party software, MapObjects from ESRI and Graph Editor Toolkit from Tom Sawyer Software.

• Fully integrated with the MOQA Fact Repository and working as a live system over the Internet and at the sites

CoGenTex, Inc.

Benefits of Textual Summaries

• Textual descriptions complement graphics and tables by– highlighting salient facts and events– clustering data in meaningful ways

• Hypertext enables web-based data exploration via “drilldown”

• Text plays a crucial role in – search results justification when search involves

inference– session report production

CoGenTex, Inc.

CoGenTex’s NLG Expertise

• R&D for MOQA builds on and extends capabilities developed in previous efforts:– Web-based multimodal presentations which

include automatically generated summaries – Generic text plans which support portability

between subject domains– Explanation of expert system reasoning

CoGenTex, Inc.

Query Interface

• Integrates with the NL question processor, which builds Text Meaning Representations (TMRs).

• Produces a corresponding structured query, to validate the NL question, and automatically paraphrases it in a disambiguated form of language.

• Paraphrases are a crucial feature of the planned ontology-based query clarification and repair dialogs

CoGenTex, Inc.

Search Page with NL Query:

CoGenTex, Inc.

(REQUEST-INFO-1007 (THEME (VALUE (COMMON MEET-WITH-1007.THEME))) (INSTANCE-OF (VALUE (COMMON REQUEST-INFO))))

(MEET-WITH-1007 (TIME (VALUE (COMMON DATE-1007 (< (FIND-ANCHOR-TIME *PAST-EVENT*))))) (AGENT (VALUE (COMMON HUMAN-1007))) (INSTANCE-OF (VALUE (COMMON MEET-WITH))))

(DATE-1007 (MONTH (VALUE (COMMON 1))) (YEAR (VALUE (COMMON 2003))) (INSTANCE-OF (VALUE (COMMON DATE))))

(HUMAN-1007 (AGENT-OF (VALUE (COMMON MEET-WITH-1007))) (OBJECT-NAME (VALUE (COMMON "Jalal Talabani"))) (INSTANCE-OF (VALUE (COMMON HUMAN))))

TMR for NL Query

CoGenTex, Inc.

Underlying XML Query

• Use standard XML technology (browser, parsers, etc.)

• Support for :– Modularity: XML

representation is viewable and exchangeable between subsystems

– Automatic validation of query instances using a query class hierarchy described by XML schemas

– Complex queries via use of logical expressions in XML

CoGenTex, Inc.

Validation of NL Query

CoGenTex, Inc.

Results Page Highlights

• Multimodal display with table, timeline, text, map, and relationship diagram

• Hyperlinks for drilldown

• Textual summaries highlight patterns in results (prompts for sorting/ filtering table)

CoGenTex, Inc.

Query Results Page for Previous Query

CoGenTex, Inc.

Person Details Display

CoGenTex, Inc.

Expanded Relationship Graph

CoGenTex, Inc.

Details Page with Arabic Alias

CoGenTex, Inc.

Excerpt from Arabic Source

CoGenTex, Inc.

Source Arabic Document

CoGenTex, Inc.

Session History

• Captures sequence of queries and results

• Analyst can reorder/delete items to retain most important info

• Reports can be exported to text editor

CoGenTex, Inc.

Session History Page

CoGenTex, Inc.

R&D Plans for next 6 months

• Presentation of ontology-based inference for result justification

• Intelligent maintenance of session history and retrieved material to facilitate report production

• Presentation of search results for partially understood natural language queries

• Exploration of interface portability between subject domains based on automatic generation of interface core from the subject domain ontology

CoGenTex, Inc.

CoGenTex, Inc.

MOQA User Interface: Validation

• Validation by domain experts is crucial for producing an intelligent and ergonomic question answering interface– Highly interconnected multimodality offers

new views on data and methods of browsing– Feedback is crucial

• for the search results justification• for session report production

CoGenTex, Inc.

Thank you!