theban stanley human and systems engineering center for advanced vehicular systems enhancements to...

18
Theban Stanley Human and Systems Engineering Center for Advanced Vehicular Systems Enhancements to the DARPA Communicator Architecture

Upload: alisha-walsh

Post on 31-Dec-2015

215 views

Category:

Documents


2 download

TRANSCRIPT

Theban StanleyHuman and Systems Engineering

Center for Advanced Vehicular Systems

Enhancements to the DARPA Communicator Architecture

Page 2 of 20Human Language Technology

Abstract

Monolithic Systems Natural lan..

Speech Rec..

Informati..Distributed Systems

Problem Statement: The distributed framework has enabled the development of client-server applications with complex inter-process communication which has resulted in the reduction of the overall system robustness to failure.

Some of the notable projects:

Advanced Language Engineering Platform (ALEP)

General Architecture for TEXT Engineering (GATE)

TIPSTER project

DARPA Communicator program: An open source architecture for spoken language applications.

Speech Recognition

Natural Language Understanding

Information Retrieval

Page 3 of 20Human Language Technology

Introduction

Advantages of the DARPA architecture:

• The Communicator has a flexible “hub and spoke” architecture with a programmable hub which allows flexible control of interaction among servers.

• Open source architecture.

• Reduced prototype development time.

• Enables sharing of components across different sites. Standard platform for evaluation of systems developed by different laboratories.

Initial Prototype system:

• Dialog System application

• Speaker Verification application

Page 4 of 20Human Language Technology

Original Architecture - Disadvantages

Deadlocks in the communication between servers:

• The two main reasons for these deadlocks were inter-process communication error and misfiring of user interface events.

• Need for more organized mechanism for logging all the communication between servers.

• These issues were anticipated to grow in number and complexity as we moved to a multi user/application platform.

Need for automated monitoring of servers:

• When a process fails, the user has to manually track this development and restart the process.

• Even though the Galaxy’s Process Monitor provides a good interface to start and terminate the servers, it needs manual monitoring of the servers.

• Need for modules with the intelligence to start servers, check on their status and terminate these processes when the application is closed.

Page 5 of 20Human Language Technology

Original Architecture - Disadvantages

Inability to automatically handle multiple users:

• DARPA communicator provided no mechanism to handle multiple users.

• The servers needs to be manually started for each user.

• In a multi user environment, port allocation needs to be handled with care.

• Need for modules that can keep track of the client-server association and manage port allocation.

Need for a common user interface for all the applications:

• Supporting multiple applications required a common interface that allows the user to choose from a list of applications.

• The module should be able to start the user interface for the requested application and coordinate the creation of the appropriate servers.

Page 6 of 20Human Language Technology

Architectural Enhancements – Automated Server Management

• Automated server management becomes critical when the system has to handle multiple users.

• Process Manager module was designed to automatically manage and control all server processes in the prototype system architecture.

• The Process Manager exploits the powerful functionality of Java’s Process Object.

• Process Manager the capability to create a process, wait on a process, perform input/output on a process and also check the exit status of a process.

• How it works:

When the user chooses a specific application, the Process Manager starts the required servers.

The Process Manager has the knowledge to start the appropriate servers for the chosen application.

The Process Manager allocates ports.

keeps track of the client-server association.

Page 7 of 20Human Language Technology

Architectural Enhancements – Process Manager

ProcessManager

Client Side Server Side

Speech Analysis

Hub

Signal Detector

Data RecorderSpeech

Recognition

Hub

Signal DetectorData Recorder

Speech Recognition

Page 8 of 20Human Language Technology

Architectural Enhancements – Common Application Interface

The Demo Selector module:

• Provides desired interface

• Coordinates with the Process Manager module to start the required servers.

• The user selects one of the application

• The Demo Selector loads and displays the appropriate user interface (UI).

• Directs the Process Manager to start the appropriate servers.

• The configuration menus which allow the user to change the default port and IP settings.

Page 9 of 20Human Language Technology

Architectural Enhancements – Robustness Improvements

Improving system robustness was the primary focus of these enhancements.

State Machine architecture:

• Redesign of the servers as state machines.

• The server anticipates a particular message.

• Any inter-process communication error leading to a server failure or a deadlock can be trapped by this architecture.

Handshaking: Handshaking was implemented by using the implicit capabilities of the Communicator frame.

Wait_for_Audio_

Ready state

Data_Transfer

State

End_Of_Utterance

State

Initializationstate

Audio Ready

state

Data_Transfer

State

End_Of_Utterance

State

Initializationstate

Audio_Ready_Ack

state

Audio_Ready

Audio_Ready_

Ack

Data

Data_Ack

End_Points

End_Of_Utterance

End_Of_Utterance

Ack

Speech Analysis Client Signal Detector

Page 10 of 20Human Language Technology

Results and Analysis – Quantitative Analysis

Pilot Corpus Experiment:

About 4% of the system failures were caused due to server errors and deadlocks.

Page 11 of 20Human Language Technology

General Usage Scenarios Experiment:

Procedure:

Five users were asked to engage in 24 usage scenarios using the original and the enhanced architecture. After a 10-minute practice to get familiar with the functionality of the system, the user performed the scenarios. The entire experiment took approximately 1 hour 30 minutes. The user was asked to cease testing if there was a system failure or he/she exceeded the allotted time of 30 minutes.

Scenario sample:

(Dialog system Application) Imagine you are in a big city to attend a conference. Once the conference proceedings are over for the day, you want to visit some sites of interest. You don’t have a map with you and have no idea about the layout of the city. Use the system to plan your trip.

Results and Analysis – Quantitative Analysis

Page 12 of 20Human Language Technology

* E stands for enhanced architecture; O stands for original architecture.

Results:

Results and Analysis – Quantitative Analysis

Page 13 of 20Human Language Technology

Dialog System – Speech mode Experiment:

Procedure:

Five users were asked to engage in 9 usage scenarios restricted to the Dialog system: speech mode category. The scenarios were based on the user’s visit to Starkville from her city of residence. The user performed the tasks on both original and the enhanced architecture. After a 10-minute practice session to get familiar with the functionality of the system, the user performed the scenarios. The entire experiment took approximately 1 hour. The user was directed to ask for assistance in restarting the system in case of a system failure.

Scenario sample:

Imagine you are working in a big city for quite a few years. You plan to make a visit to Starkville. So you start on a road trip from your city. You are almost near Starkville when you find that you are really low on gas. Use the system to make a decision on whether you can make it without filling gas.

Results and Analysis – Quantitative Analysis

Page 14 of 20Human Language Technology

Results:

Results and Analysis – Quantitative Analysis

Page 15 of 20Human Language Technology

Results and Analysis – Qualitative Analysis

ProcessManager

Client Side Server Side

Speech Analysis

Hub Data Recorder

Signal Detector

Initialization state

Wait_Audio_Ready

state

Data Transfer

state

End_of_Utterance

state

Audio_Ready

Audio_Ready_ack

Audio_Ready

The Signal Detector errors out

Page 16 of 20Human Language Technology

Results and Analysis – Qualitative Analysis

Debug Window:

Page 17 of 20Human Language Technology

Conclusion

• The DARPA Communicator architecture has significantly advanced human language technology and, has played a critical role in the design and development of human language technology applications at CAVS.

• In developing these applications, vulnerabilities in this architecture have been addressed through several important enhancements. The key Contributions include:

Improved Robustness by incorporating state machine architecture and basic handshaking between servers.

Automated Server Management

Common Application Interface

Experiments have shown a 7.2% improvement on the address querying task which is the most complex task in the HLT system.

Page 18 of 20Human Language Technology

Future Work

• Further experimentation with:

At least 20 additional users who have no familiarity with the system.

The system should be allowed to respond to user queries continuously for prolonged time periods.

The prolonged experiments must be carefully controlled using scenarios that properly exercise system functionality, so that meaningful data are collected.

• Further Wizard of Oz experiments can be performed to improve the grammar and the language model which will allow the dialog system to handle a wider range of user queries.

• A future enhancement would be to extend the dialog system to accommodate a statistical parser which can be trained on any data set.

• The enhanced system can be tested on ERC supercomputer clusters which will significantly improve the resources available to the system.