multimodal research in linköping

Arne Jönsson 1

Multimodal research in Linköping

Arne JönssonNLPLAB/HCS/IDA

Arne Jönsson 2

NLPLAB

Upper row:Pernilla,Sonia,Lena,Annika,Micke,Magnus

Bottom row:Håkan,Lars A,Robert,Mustapha,Arne,Lise,Anders,Lars D

Missing:Nils, Bertil, Pontus, Gen

Arne Jönsson 3

Current research

• (Multimodal) information access- Natural language interaction- Text and document analysis

• Usability of multimodal interfaces• Open-Source and development methods

Arne Jönsson 4

Dialogue system

Dialoguemanager

Generator

Interpreter

StructuredInformation

Dialogue system

Arne Jönsson 5

Ex TV programme guide

• Search for TV programs• Mediaterminal• Structured database • Spoken input• Developed from framework and method

for iterative development• With Nokia Home Communication

Arne Jönsson 6

Demo

Arne Jönsson 11

Text and document analysis

Language analysis

StructuredInformation

Unstructured documents

Q&A

MT

Arne Jönsson 12

Ex. AskRSV

• Information extraction from text document

• Simple Q&A-system• With Ida Infront

Arne Jönsson 13

Ex. taxation brochure

24 pages unstructured text

Arne Jönsson 14

Domain ontology in ASK-RSV

• Taxonomy of a limited set of objects, events, properties and relations in the tax domain

• Only IS-A and synonymy relations

OBJECT

THING

ACTOREVENT

PAPER DATE TAXATION TAX EVENT ORG.PEOPLE

TAX-RETURN ASSET DEBT INCOME TRANSACTIONCOMP. PUBL

Arne Jönsson 18

Combined system

StructuredInformation

Unstructured documents

Language analysis

Dialoguesystem

Lexicon Grammar Ontology

Arne Jönsson 19

Shared knowledge sources

• Ontology, lexicon and grammar- Different demands from IE and DS

• Design• Development

- Corpus analysis, user - Background system, database

• Automatic update

Arne Jönsson 20

BirdQuest

• Bird encyclopaedia- Tagged, templates

• Corpus with user questions- Analysed

• Based on dialogue systems framework

Arne Jönsson 26

Ötraf

• Time tables from local bus company• Multimodal system

- Dialogue- Handling of spatial and temporal

expressions

Arne Jönsson 28

Studies of multimodal interaction

• Wizard of Oz-simulation • Feedback study

- User’s experience- Interviews, questionnaires- Various dimensions: control, cooperation, ease of use,

efficiency, anxiety, hability etc- Three conditions

- No spoken feedback- Limited spoken feedback- Full spoken feedback

- 30 subjects

Arne Jönsson 29

Results

• All or nothing• More like a human in the full feedback

condition- Before the study subjects wanted to have a

timetable for the task, after they preferred a human• No difference in experienced control and

cooperation• The speech synthesis better in full feedback

mode- Same voice

Arne Jönsson 30

Open source

• Share research results between research groups- Stimulate exchange and co-operation

• Language technology readily available for industry- Increased use and insights- The companies do their own prototype

development• Robust code• nlpFarm

http://sourceforge.net/projects/nlpfarm/

Arne Jönsson 31

http://sourceforge.net/projects/nlpfarm/