ACL, ECCAI and the Verbmobil/SmartKom Consortia

German Research Center for Artificial IntelligenceStuhlsatzenhausweg 3, Geb. 43.1

66123 SaarbrückenTel.: (0681) 302-5252/4162

Fax: (0681) 302-5341E-mail: wahlster@dfki.de

WWW: http://www.dfki.de/~wahlster

Wolfgang Wahlster

ACL Workshop, Hong Kong 7 October 2000

Hosted by DFKI (Thierry Declerck)

http://registry.dfki.de

130 links to software systems (shareware, freeware, open source, trial versions)

all registered software packages are classified

software from academic and industrial groups

Classification according to NSF/EU „State of the Art in Language Technology“

Cooperation with ELRA, LDC, ELSNET, CLASS, TELRI, BAS

The International NL Software Registry of ACL

UkrainADUIS

ItalyAIIA

FranceAFIA

PortugalAPPIA

BulgariaBAIA

U.K.BCS-SGES

SSAISBCzech

RepublikCSKIDenmark

DAISFinlandFAIS

GreeceEETN

GermanyGI/KI

IsraelIAAI

LatviaLANO

LithuaniaLIKS-AIS

NorwayNAIS

HungaryNJSZT

Belgium/ The Netherlands

BNVKI

AustriaÖGAI

RussiaRAAI

SwedenSAIS

SwitzerlandSGAICOSlovenia

SLAIS

Slovak RepublicSSKI SAV Spain

AEPIA

CataloniaACIA

ECCAI MEMBERS

Founded in 1982, currently 26 member societies

Represents more than 6000 AI researchers

The world‘s largest AI society

Web site: WWW.ECCAI.ORG

Every two years: ECAI conference and ACAISummer school

ETAI electronic journal, neo-classical reviewing Hardcopy distributed by Royal Swedish Academy of Sciences

ECCAI: European Coordination Committeeon Artificial Intelligence

UNIVERSITÄT DESSAARLANDES

RUHR-UNIVERSITÄTBOCHUM

Phase 2

UNIVERSITÄTHAMBURG

UNIVERSITÄTKARLSRUHE

UNIVERSITÄTBIELEFELD

TECHNISCHEUNIVERSITÄT

MÜNCHEN

FRIEDRICH-ALEXANDER-UNIVERSITÄT

ERLANGEN-NÜRNBERG UNIVERSITÄTSTUTTGART

RHEINISCHE FRIEDRICHWILHELMS-UNIVERSITÄT

BONNLUDWIG

MAXIMILIANSUNIVERSITÄT

MÜNCHEN

TU-BRAUNSCHWEIG

EBERHARDT-KARLSUNIVERSITÄT

TÜBINGEN W. Wahlster, DFKI

The Verbmobil Consortium

Multilingualand Mobile

CommunicationAssistants

Multimodal Interfaces

SmartKom

Speech-based Web Access to Multilingual

Web pages

WAP Phones WebTV

Multilingual Audio Retrieval

and Audio Mining

Discussions Lecture Notes Organizers

MultilingualIndexing andAnnotation of

Videos

Video Archives News Archives

Call CentersECommerce Mobile Travel Assistance Telephone Translations

Verbmobil

Dialog Translation

International Research Trends in Multilingual Systems

Multilingual Language Technology Speech Recognition, Language Understanding, Language Generation,

and Speech Synthesis

Multilingual Language Technology Speech Recognition, Language Understanding, Language Generation,

and Speech Synthesis

Spontaneous Speech, Robust Processing and Translation, Semantic and Pragmatic Understanding

Verbmobil‘s Massive Data Collection Effort

Transliteration Variant 1Transliteration Variant 2 Lexical OrthographyCanonical PronounciationManual Phonological Segmentation

Automatic Phonological SegmentationWord SegmentationProsodic SegmentationDialog ActsNoises

Superimposed SpeechSyntactic CategoryWord CategorySyntactic FunctionProsodic Boundaries

The so-called Partitur (German word for musical score)orchestrates fifteen strata of annotations

3,200 dialogs (182 hours)with 1,658 speakers79,562 turnsdistributed on56 CDs, 21.5 GB

Treebank: 85,000 Trees

79,562 turns

1,520,000 words

76,210 dialog acts

Collection Sites: All over Germany

American English: CMU

Japanese: ATR

Mandarine: Philips Taiwan

The Verbmobil Corpora

Machine Learningfor the Integration of Statistical Properties into

Symbolic Models for Speech Recognition, Parsing,Dialog Processing, Translation

TranscribedSpeech Data

SegmentedSpeech

with ProsodicLabels

AnnotatedDialogs withDialog Acts

Treebanks &Predicate-ArgumentStructures

AlignedBilingualCorpora

HiddenMarkovModels

Neural Nets,MultilayeredPerceptrons

ProbabilisticAutomata

ProbabilisticGrammars

ProbabilisticTransfer

Rules

Extracting Statistical Properties from Large Corpora

MediaInterfaceEuropean Media LabUinv. Of

MunichUniv. ofStuttgart

Saarbrücken

Aachen

Dresden Berkeley

Stuttgart

MunichUniv. ofErlangen

Heidelberg

Main ContractorProject Management

TestbedSoftware Integration

DFKISaarbrücken

The SmartKom Consortium:

Project Budget: $ 34 MProject Duration: 4 years

DAIMLERCHRYSLERUlm

SmartKom: Intuitive Multimodal Interaction

W. Wahlster, DFKI

User(s)

MediaAnalysis

Design

Media Fusion

OutputRendering

Representation and Inference

UserModel

DiscourseModel

DomainModel

TaskModel

MediaModels

InteractionManagement

MediaAnalysis

InputProcessing

Information

Applications

People Intention

RecognitionMediaDesign

Ap

pli

cati

on

In

terf

aceDiscourseModeling

UserModelin

g

PresentationDesign

Language

Graphics

Gesture

Biometrics

Language

Graphics

Gesture

AnimatedPresentation

Agent

The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)

W. Wahlster, DFKI

SmartKom-Home/Office:A Versatile Agent-based Interface

SmartKom-Public: A Multimodal

CommunicationBooth

SmartKom-Mobile: A Handheld

CommunicationAssistant

Media Analysis

Kernel ofSmartKomInterface

Agent

Interaction Management

ApplicationManage-

ment

MediaDesign

SmartKom: A Transportable and Transmutable Interface Agent

W. Wahlster, DFKI

Smartcard/ Credit Cardfor authentication and billing

Docking stationfor PDA/Notebook/Camcorderhigh speed and broadbandwidth Internet connectivity

High-resolution scanner

Loudspeaker

Room microphone

Face-tracking camera

Virtual touchscreenprotected against vandalism

Multipoint video conferencing

SmartKom-Public:A MultimodalCommunication Booth

W. Wahlster, DFKI

SmartKom Domains: EPG and Cinema Selection

Multimodal SelectionOf Cinema

Multimodal Selection of Seats

Camera

GPS

Microphone

Loudspeaker

Stylus-Activated Sketch Pad

WearableComputeServer

Docking Stationfor Car PC

Biosensorfor Authentication

& Emotional Feedback

GSM for Telephone,Fax, Internet Connectivity

SmartKom-Mobile: A Handheld Communication Assistant

W. Wahlster, DFKI

SmartKom‘s Data Collection of Multimodal Dialogs

User

Side-viewCamera

Face-tackingCamera withMicrophone

EnvironmentalNoise

MicrophoneArray

Screen

ProjectedWebpage

Face-tackingCamera

LoudspeakerMicrophone

Array

User

Bird’s-eyeCamera LCD

Beamer

SIVIT-Camera

BAS in Munich (Prof. Tillmann, Dr. Schiel)

68 GB of transliterated speech data 590 hours of speech, 2500 different speakers

works for Verbmobil and SmartKom consortium

treebanks, gesture and dialog act annotations

very large corpora of GSM phone dialogs multi-channel recordings of spontaneous dialogs WOZ data

one year exclusive use for German labs and industry

after one year distribution by ELRA and then by LDC

The German Center for Language Resources