NPCEditor: Creating Virtual Human DialogueUsing Information Retrieval Techniques

Anton Leuski and David TraumInstitute for Creative Technologies

12015 Waterfront DrivePlaya Vista, CA 90094

AbstractNPCEditor is a system for building a natural languageprocessing component for virtual humans capable of en-gaging a user in spoken dialog on a limited domain.It uses statistical language classification technology formapping from a user’s text input to system responses.NPCEditor provides a user-friendly editor for creatingeffective virtual humans quickly. It has been deployedas a part of various virtual human systems in several ap-plications.

1. Virtual HumansImagine talking to a computer system that looks and acts al-most human—it converses, it understands, it can reason, andexhibit emotion. As an example, recall such computer char-acters created by Hollywood moviemakers as the librarian inTime Machine, the holographic professor in I Robot, and ofcourse, the holodeck characters in numerous Star Trek: TheNext Generation episodes. Once the realm of science fiction,limited, domain-specific versions of these kinds of charac-ters are now achievable using AI and computer graphicstechnology. Such simulations, called virtual humans (VH),open up new horizons for entertainment, teaching, and learn-ing. Virtual humans can serve as colleagues or adversaries intraining simulations helping a student to study language andculture (Johnson, Vilhjalmsson, and Marsella 2005) or honeher negotiation skills (Traum et al. 2005). They can help totrain physicians in treating psychological disorders (Kenny,Parsons, and Rizzo 2009). They work as virtual guides (Janet al. 2009), museum docents (Swartout et al. 2010), or evenengage the user in a gunfight (Hartholt et al. 2009). Also seethe insert box.

A typical VH system is rather complex and may consistof several components including speech recognition, ges-ture recognition, language understanding, dialogue manage-ment, emotion reasoning, planning, inference, verbal andnon-verbal output, body simulation, realistic graphics, andmixed reality displays (Gratch et al. 2002). So, before avirtual character is ready to interact, it has to be designedand VH system components have to be assembled and de-ployed. Therefore, we distinguish four types of VH technol-ogy users:

Copyright c© 2011, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Designers author the VH system using available tools.They create scenarios, develop the look and behavior ofthe agents, including the interactive dialogue behavior.

Administrators deploy the system, maintain it in workingorder so that others can interact and view the VHs.

Interactors talk to the VH. There are really two types ofinteractors, demoers who work with the Administratorsand are familiar with the system, and players who are not.Players are the primary target of the interaction. In thecase of Demoers, it is the audience that is the primary tar-get of interaction and demoers are presenting the systemto the audience.

Audience members observe others interact with the virtualhuman. As said above, when interactors are demoers thenthe audience is the primary target of the interaction, how-ever there may be an audience member acting as sec-ondary target even when the interactors are players.

For virtual human systems to become widespread thereare two main requirements. First, the advances in technol-ogy must reach the level of reliability and efficiency thatmakes the interactions with virtual humans seamless and re-alistic. Second, this technology has to be implemented andpackaged in software that is accessible to the training or en-tertainment system designers who are not technical expertsin the underlying AI technology.

In this paper we focus on the natural language process-ing (NLP) parts of a virtual human system, including nat-ural language understanding and generation, and dialoguemanagement. We describe NPCEditor1—an NLP systemthat supports all of the above user types. At the core of thesystem is a statistical text classification algorithm developedspecifically for the task of understanding the interactor’s lan-guage. In the paper we summarize the algorithm and someexperimental results that show its effectiveness. NPCEditorpackages the text classifier in a GUI-based application thatallows creation of useful VH systems with minimal train-ing. In the paper we outline the system design and charactercreation and deployment process. More details on the userinterface for the technology and how it is used can be foundelsewhere (Leuski and Traum 2010). NPCEditor has beenused extensively in a number of projects both internally in

1NPC stands for Non-Player Character

Twins Ada and Grace, the Museum GuidesA well-informed guide or interpreter can have a tremendousinfluence on the quality of a museum visitors experience. Thebest guides not only provide information but also engage thevisitor in an interactive exchange that can lead to deeper un-derstanding and promote excitement about museum content.Unfortunately, human museum guides are often in short sup-ply. Many studies have shown that people react to virtualhumans in much the same way that they react to real peo-ple (Reeves and Nass 1996; Gratch et al. 2007). Could vir-tual humans be used to create museum guides that can engagevisitors with museum content?

Ada and Grace are two virtual museum guides that we re-cently deployed at the Boston Museum of Science as a part ofthe InterFaces exhibit (Swartout et al. 2010). They are life-sized, photo-realistic characters that interact in natural lan-guage, complete with gestures, and other forms of non-verbalcommunication. Ada and Grace can answer questions on anumber of topics, including their personal background, gen-eral facts about virtual human technology, and the exhibitson computers, robotics, and communication in the CahnersComputerPlace exhibit hall of the museum.

Visitor: good morning

Ada: Hello!

Visitor: what are your names

Ada: My name’s Ada.

Grace: And I’m Grace. We’re your Virtual MuseumGuides. With your help, we can suggest exhibitsthat will get you thinking! Or answer questionsabout things you may have seen here.

Ada: What do you want to learn about?

Visitor: what can you tell me about?

Grace: Let’s see. What are you interested in? Do youwant to learn more about computers, robots, cellphones, hands-on activities...

Visitor: I’m interested in robots

Grace: Here at ComputerPlace you can command arobot to do a Mars mission, and program an-other one at Robot Park. You can remote con-trol a robot soldier, or figure out how robotsclimbs walls. You can even make a robotic dogdance, or use one to navigate a virtual world.And don’t forget the one that remembers meet-ing you. What do you want to do first?

Visitor: where is the mars yard

Ada: The Mars Yard is just to your right, and includesMission Control.

Visitor: can we colonize mars

Ada: Mars isn’t exactly paradise. It has little oxygenand no food for humans to survive. It can getextremely cold and has harmful radiation. But,Grace and I, being computers, we can visit justfine.

Visitor: where do you go to school

Ada: Even though we can’t go to school, we are theproduct of a lot of education.

Grace: The museum gathered up what they wanted to usto know about computer place and sent it overto the University of Southern California. Therethe Institute for Creative Technologies added VHuman knowledge to the stack, then uploaded itall into our database.

Ada: Not exactly a college education, but scientistsfrom both Boston and Southern California haveloaned us theirs, until we figure out how to joinall of you in the classroom.

our group and externally by other teams at the institute andoutside organizations. We describe some of the VH systemsthat use NPCEditor and have been deployed at training lo-cations, shows, virtual worlds, and museums.

2. Scientific Contribution: Cross-LanguageRetrieval for Dialogue Response Selection

There are many NLP technologies that might be applied tovirtual human language interaction. The choice depends inlarge part on the required capabilities: Does the VH have afirm agenda or is it more flexible? Does it lead the interac-tion or react? Does it need to perform deep inference on themeaning of what is said, or can it stay close to the surface?Will the responses need to be computed on the fly based oncurrent context, or can they be pre-computed or authored?

NPCEditor has been used primarily to construct question-answering characters. These characters play the role ofinterviewees and respond to questions in character. Thereare many kinds of interviews (doctor-patient, police-suspect,reporter-witness, information seeker-expert, and so forth)and thus question-answering characters have broad applica-bility. For example, imagine that you are playing the roleof a detective in the game of “Clue.”2 An owner of a largehouse has been murdered and you interrogate the guests ofthe house. The house guests and witnesses are played by vir-tual humans. Each character should be capable of answeringa number of questions on a limited set of topics that are po-tentially relevant to the event and it should be able to deflectall other questions.

A question answering virtual human is characterized bya collection of responses relevant to a particular topic. Thisapproach gives complete control over the virtual persona’sknowledge and expressions to the scriptwriter who createsthe responses. It allows the writer to specify the character ofthe virtual persona, what information it can deliver and theform of that delivery. When an interactor comes up to thevirtual character and asks it a question, the system drivingthe character analyzes the interactor’s question and selectsthe appropriate response from the collection.

This approach also simplifies the overall VH system: abare-bones system would consist of an ASR module forspeech processing, the NPCEditor system to process the in-teractor’s utterances and select the character response, and arendering engine, capable of presenting the animated char-acter on the screen and playing back prerecorded responses.

Automatic question answering has been studied exten-sively in recent years. It is generally defined as an informa-tion retrieval (IR) problem where a user places her request ina form of a question and expects a relevant and succinct re-sponse, e.g. “How tall is mount Everest?”—“Mount Everestis 29029 feet tall.” One example of such a system is STARTfrom MIT (Katz 1988). It uses well-defined informationdatabases and carefully crafted question parsing rules to findthe required answer. Web-based question answering systemsand systems studied in the context of the question-answeringtrack at the Text REtrieval Conference (TREC) attempt to

2“Clue” is official trademark of Hasbro Inc.

answer user’s questions by finding and parsing relevant para-graphs in large text collections (Voorhees 2003).

In contrast to the fact-based question answering scenariowhere the goal is to provide the most relevant answer, we fo-cus on the answer’s appropriateness. In our example aboutan investigation, an evasive, misleading, or an “honestly”wrong answer from a witness character would be appropri-ate but might not be relevant. Alternatively, different char-acters may have different knowledge about the event and re-spond differently to the same question. We try to highlightthat distinction by talking about question-answering char-acters as opposed to question-answering systems or agents.Another difference is that question-answering systems relyon the question text to be lexically and grammatically cor-rect and well-formed, while our system is primarily usedto reply to spontaneous spoken language, which is muchmore likely to include disfluencies and non-standard con-structions. A third difference is that the input for our systemcomes from an automatic speech recognition (ASR) mod-ule that sometimes introduces errors into the transcription.These errors can affect the interpretation performance sig-nificantly. A virtual human should be robust to both disflu-encies in conversational English and to the errors introducedby the ASR module.

Similar requirements exist for automatic phone reserva-tion and call routing systems (Gorin, Riccardi, and Wright1997). For example, Chu-Carroll and Carpenter describe asystem that answers a phone call, asks a caller some ques-tions, and routes the call to the appropriate destination (Chu-Carroll and Carpenter 1999). The system uses a vector-based text classification approach to analyze the caller’s re-sponses and map them to the destinations in the organiza-tion. Our NPCEditor system maps text of the question di-rectly to texts of the answers and uses a novel text classifi-cation approach based on statistical language modeling thatsignificantly outperforms vector-based approaches (Leuskiet al. 2006).

Text classification has been studied for several decades,and numerous approaches exist (Lewis et al. 1996). It isthe task of assigning pieces of text to one or more classesbased on the training data. The traditional text classifica-tion approach for our task is to define each answer as a classand define the corresponding questions as the training textpieces. Generally, a text string is represented as a featurevector where individual words serve as feature elements.When a new question arrives, it is tokenized, converted intoa feature vector representation, compared to the vectors ofthe known questions, and the answer corresponding to thebest matching group of questions is returned. The disadvan-tage of this approach is that it completely ignores the contentof an answer. The main difference between our text classi-fication and a traditional text classification approach is thatwe match a user’s question to known answers and not tothe known questions. Our experiments show that taking theanswer text into account during classification improves theeffectiveness of the classification significantly.

Let us explain the theory behind our classification ap-proach. Our task can be described as follows: given aquestion about a particular topic, find an answer about the

same topic. The key achievement of IR is an ability tomatch two strings of text based on content similarity. Thatis how a search system works—a text representation is com-puted for documents and a query, a matching algorithm isapplied, and the best match is returned to the person whoentered the query. One technique for text content represen-tation that has recently gained wide usage in IR (Ponte andCroft 1997) uses the notion of a statistical language model: aprobability distribution P (W ) over all possible word stringsW = w1, ..., wn. Here is how it works: a content topic canbe described by different text strings, some are more likelyto be used than others. For example, the prhase “green andround” is much more likely to be used when describing anapple than the phrase “blue and square.” So if we can de-fine the probability of observing each possible text string inconnection with a given topic, we will have a very detailedrepresentation for the topic. It is reasonable to assume thatlanguage models of similar topics will also be similar. Ifwe can estimate a language model from the question stringand another language model for an answer string, we cancompare the content of the question and the answer by com-paring the corresponding language models.

Before we describe the details of the method, we haveto make an observation: We cannot compare question andanswer language models directly because the former is theprobability distribution over questions and the latter is theprobability over answers. These distributions are likely to bedifferent even when matching to the same topic. For exam-ple, due to grammar rules, some strings (e.g. strings contain-ing “wh” words) are much more likely to appear as questionsthan answers. Moreover questions and answers are “gener-ated” by different entities—the interactor and the character(or, the scriptwriter) who may have different styles of ex-pression reflected in the bias of the language model. Thesedifferences allow us to talk about questions and answers assamples from two different languages.

Here is where some training data can be useful. If foreach answer in a character database we have some ques-tions that can be answered by that answer, we can train thesystem to “translate” from the language of questions to thelanguage of answers. When we see a new question Q weuse that training data to estimate the language model of theanswer to the question PQ(A) and then compare that lan-guage model to language models of the character answersand return the best match. This approach is very similar tothe cross-language information retrieval task, e.g., where asearch system has to find Chinese documents in response toan English query (Grefenstette 1998). The training data thatpairs sample questions with the answers serves as “parallelcorpora” and the translation rules are derived implicitly fromthat mapping.

There are different ways to compare two probability dis-tributions. NPCEditor uses the Kullback-Leibler (KL) di-vergence D(PQ(A)||P (A)) defined as

D(PQ(A)||P (A)) =∫A

PQ(A) logPQ(A)P (A)

(1)

which can be interpreted as the relative entropy between twodistributions. Note that the Kullback-Leibler divergence is a

dissimilarity measure, we use −D(PQ(A)||P (A)) to rankthe answers.

Normally a topic is represented by a single text string (W).It is impossible to determine the language model from sucha sample explicitly. The goal is to estimate the probabil-ity of such a string, P (W ), as accurately as possible. Theproblem of estimating the joint probability P (w1, ..., wn)of several words occurring together to form a string of textW has received a lot of attention in recent years among re-searchers in the IR community. The main challenge is totake into account the interdependencies that exist among theindividual words while still making the computation feasi-ble. Several different methods were suggested starting fromthe most trivial technique where all words are assumed to bedistributed identically and independently from each other—the unigram model:

P (w1, ..., wn) =n∏i=1

P (wi) (2)

Other approaches include Probabilistic Latent Semantic In-dexing (PLSI) (Hofmann 1999) and Latent Dirichlet Allo-cation (LDA) (Blei et al. 2003), where the authors modeltext collections by a finite set of k topics and the overall textprobability is viewed as a mixture of the individual topiclanguage models.

Lavrenko (Lavrenko 2004) suggests a more general ap-proach where the word interdependencies are defined by anunknown vector parameter θ and the words are taken asconditionally independent. This step allows him to relaxthe independence assumption of the unigram model so thatthe probability distribution depends on the co-occurrence ofwords. Lavrenko treats the vector θ as a random variableand suggests several techniques for estimating its probabil-ity distribution from experimental data. He calls it a Rele-vance Model approach and shows how PLSI and LDA canbe viewed as special cases of the approach. His experi-ments and studies conducted by other researches (e.g., (Weiand Croft 2006)) established the relevance model approachas one of the top performing techniques in information re-trieval.

While Lavrenko suggests several methods for estimatingthe probability distribution of vector θ, one of these tech-niques is most often used in the experiments. Its advantagesare the small number of parameters to estimate (i.e., one)and relative computational efficiency. Specifically, given acollection of known questions Q, the language model of anew question Q = q1, ..., qn is defined as follows:

P (q1, ..., qn) =1|Q|

∑s∈Q

n∏i=1

ps(qi) (3)

where |Q| is the number of questions in the database andps(qi) is the probability of observing word qi in string s.There are several ways of estimating the latter value. Weuse Maximum Likelihood Estimation (MLE) with Jelinek-Mercer smoothing (Bahl, Jelinek, and Mercer 1990):

ps(q) ∼= πs(q) = λπ#s(q)|s|

+ (1− λπ)∑s #s(q)∑s |s|

(4)

#s(q) is the number of times word q appears in string s,|s| is the number of words in string s, and λπ is a tunableparameter that can be determined from the training data.

An astute reader may notice that equation 3 is similar tothe unigram model: it is an average of unigram models ofindividual questions in the training data. It allows us to takeinto account the word co-occurrence in the training data andincorporate this into model estimation.

Equation 3 assumes that all words qi come from the samevocabulary. We can show that in the case of two different vo-cabularies, the conditional probability P (a|Q) of observinga word a in answer language given an interactor’s utteranceQ can be estimated as:

P (a|Q) =P (a, q1, ..., qn)P (q1, ..., qn)

(5)

=∑s πAs

(a)∏mi=1 πQs

(qi)∑s

∏mi=1 πQs

(qi)

The matching criteria in Equation 1 can be written as

D(PQ(A)||P (A)) =∑a

P (a|Q) logP (a|Q)πA(a)

(6)

In summary, given a character database {Qs,As} anda question Q, we use Equations 3, 4, and 5 to computeEquation 6 for each answer A in the database and re-turn the answer with the highest value −D(PQ(A)||P (A)).See (Leuski et al. 2006; Leuski and Traum 2008) for moredetails.

The final parameter is the classification threshold on theKL-divergence value: only answers that score above thethreshold value are returned from the classifier. The thresh-old is determined by tuning the classifier on a randomly cho-sen subset of the training data.

Non-lexical FeaturesSo far we have described how a textual answer is selectedin response to a textual question. There are several othercases in which the NPCEditor uses the same classificationalgorithm to go beyond this scenario. First, in some appli-cations we may use the cross-language information retrievalapproach to convert between text and a semantic language.In some systems, we use the NPCEditor to recognize fea-tures such as speech acts or impact on interpersonal vari-ables (Roque and Traum 2007), while in other systems, theNPCEditor can be used to interpret the meaning of an ut-terance in a semantic representation, rather than selectingthe answer to respond (Gandhe et al. 2008). Likewise, theNPCEditor can be used to translate a semantic representa-tion of a response into text (Leuski and Traum 2008).

In some applications additional context information mightbe available as well as text. For example, in the Gunslingersystem (Hartholt et al. 2009) the interactor meets with threedifferent virtual humans. The system uses NPCEditor, anASR module, and a vision component, which (among otherthings) detects where the interactor is looking. NPCEditorannotates the ASR output with a token corresponding to theinteractor’s gaze target. The classifier treats such annota-tions as words in a piece of text associated with the question

but separate from the actual question text. Thus a questionbecomes a multi-field data structure. One of these fields con-tains the original text, the other fields contain label tokens.These label tokens have special vocabulary different fromthe question text vocabulary, so a separate language modelis estimated for each field. The question-answer similarityscore becomes a weighted sum of similarities between theanswer language model and the language models for eachfield in the question data structure:

D(PQ(A)||P (A)) =∑

i

αi

∑w∈Vi

P (w|Qi) logP (w|Qi)

πAi(w)(7)

here Qi and Ai are the ith field of the question and theanswer, the outer summation goes over every field of in-terest, while the inner summation iterates over vocabularyfor the ith field. The parameters αi allow us to vary theimportance of different fields and can be determined fromthe training data. Thus NPCEditor can be trained to re-spond differently to the same question,—e.g., “What is yourname?”,—depending on who is the interactor is looking at.NPCEditor’s user interface allows the designer to define ar-bitrary annotation classes or categories and specify some ofthese categories as annotations to be used in classification.

Dialogue ManagementThe text classification algorithm returns a ranked list of ap-propriate answers for a given question. This list can beempty when the classifier believes that no known answeris appropriate to the question. Alternatively, this list maycontain multiple answers while only one answer has to bereturned. The dialogue manager is tasked with choosing theone response that is returned back to the user. NPCEditorcontains a rule-based dialogue manager that interacts withthe rest of the system via a simplified API:• The classifier selects multiple answers: the dialogue man-

ager returns the least recently used answer, breaking tiesby the classifier score.

• The classifier selects no answer: the classifier believesthat there is no appropriate response in the characterdatabase for the user’s question. We call such a question“off-topic”. The dialogue manager returns one of the an-swers that the designer specifies as an “off-topic” answer,e.g., “Say this again?” or “I do not know anything aboutit.”

• The system returns several off-topic responses in a row:the dialogue manager tries to bring the user back to do-main of conversation by prompting her with a question,e.g., “Why don’t you ask me about my technology?” Weencourage the character designers to add a variety of off-topic and prompt utterances to the character database.For simple question-answering characters, variations of

the above approach are sufficient to generate useful behav-ior. However, this approach is less suited to dialogues wherethe decision of what to say is based more on context and/orcharacter/scenario goals than reactions to a new question.NPCEditor can also support more complex dialogue phe-nomena, as long as the basic paradigm of selecting pre-authored outputs can be maintained. This “advanced” func-

CharacterCharacter

Classifier

ConversationConversation

Communication Module

Email IMJMS FlashLocal

Conversation

Classifier

Character Server

Dialogue Manager

Logging

Character Editor

Character Database

ResponseQuestion

Classifier Trainer

Character

(a) (b)

Figure 1: a) NPCEditor system design. b) Character editor screen.

tioning is still rather primitive, however, requiring the de-signer to program the dialogue manager rather than provid-ing graphical support and easy to follow design guidelines(see Section 6 for planned future directions).

An advanced character designer can create her own re-sponse handling strategies using the Groovy scripting lan-guage3. For example, in the Gunslinger project the inter-actor’s experience follows a predetermined script that goesthrough several states. There is a greeting and smalltalkstate, where the interactor meets two virtual characters andgets to know them. It is followed by another state, wherethe characters confide their problem to the interactor. It isfollowed by another state when the third character comes onstage, and so on. The interactor’s behavior and how she re-sponds to the characters determines the story development,the ordering of states, and the topic of conversations. Foreach dialogue state the Gunslinger designers specify a sub-set of the character lines that are appropriate for that state.They train a text classifier for each state and switch betweenthe classifiers as the interactor’s experience transitions be-tween states.

The dialogue manager also keeps a model of the interactorand updates it as the interaction progresses. For example, atthe beginning of the experience the characters are polite tothe interactor and would pause to listen when the interactortries to speak. Later on, if the interactor says something thatupsets them, the characters may become more hostile andhush the interactor if she tries to interrupt them.

3. Practical Contribution: System forCharacter Development and Deployment

While the cross-language information retrieval models de-scribed in the previous section have been shown to be ef-

3http://groovy.codehaus.org/

fective (see Section 4), it can still be daunting for a systemdeveloper to master the equations, create the requisite train-ing data, and train a classifier. It may also be a challengefor a system administrator to connect this module to otherparts of a VH system, so that interactors can successfullycommunicate with the virtual human. One of the goals ofNPCEditor is to preserve the effectiveness of the technicalapproach while hiding the complexity from the users. To thisend, NPCEditor creates a unified development and run-timeinterface, that allows easy character authoring and deploy-ment. To create a character, a designer only has to populatethe language database and push a single button to train theclassifier. To integrate the module into a VH system an ad-ministrator has to fill out a form as simple as an accountform in an email client.

NPCEditor is written in Java and should run on any plat-form that supports Java 6. It has been extensively tested onMicrosoft Windows and Mac OS X. The initial version wasdeveloped in 2004-2005 and subsequently maintained andextended by the first author.

Figure 1a shows the block diagram of the NPCEditor sys-tem. The character database stores information about thevirtual characters. A character designer can store multi-ple characters in the same database so the interactor(s) mayhave a conversation with several virtual humans at the sametime. Each virtual human character is associated with a setof responses it can produce. The designer enters samplequestions and links them to the responses. The classifiertrainer component generates classifiers using the methodsdescribed in Section 2 that map from the interactor’s ques-tions to the character’s responses. The designer also selectsone of the provided dialogue manager components. A dia-logue manager is a rule-based subsystem that uses the clas-sification results and the dialogue history to select the actualresponse. Finally, the character designer sets up the charac-

ter server by registering network identities for each characterwith the communication module and enabling conversationlogging. The character server monitors the network, ac-cepts incoming messages, processes the requests, and sendsout character responses. Multiple people can interact withthe virtual human at the same time. For this purpose theserver maintains a list of conversations.

The overall architecture is modular and extensible. Sev-eral parts of the NPCEditor functionality can be extendedvia external plugins. These include supported network pro-tocols, dialogue management, text pre-processing for classi-fication, classification optimization function, and supportedfile formats.

NPCEditor provides monitoring and control functionalityover the individual system components using a GUI charac-ter editor. An NPCEditor window consists of several tabbedpanels each corresponding to a particular function. Thesepanels are listed below, along with how they are used by theuser classes defined in Section 1.

1. Utterances: a character designer specifies answers andsample questions, assigns them to individual characters,and links them to each other.

2. Settings: a designer creates and modifies annotation cat-egories and labels, assigns colors to labels, and specifieswhether a category should be used as a non-lexical featurefor classification.

3. People: a designer specifies available characters and editsthe character properties. An administrator uses the panelto specify how the characters are connected to the net-work.

4. Classifier: a designer has to train the classifier param-eters after creating or modifying the character languagedatabase. The training process is as simple as pressing asingle button on the Classifier panel. However the panelalso provides some advanced classifier tuning capabilitiesfor expert designers.

5. Conversations: an administrator can monitor the existingconversations.

6. Chat: an interactor can pose arbitrary questions to thecharacters in the database and observe how the classifierranks the available responses. Also, the audience can seedetails of the system performance. An administrator anddesigner can debug the characters.

Figure 1b shows an NPCEditor window with the utter-ance editor panel selected. There are two main areas here:the question editor is on left and the answer editor is on theright. Both the question and the answer editors follow themaster-detail interface pattern: each lists all the utterancesin a table and provides controls for editing the selected ut-terance. Specifically, a developer can define the utterancetext, speaker, assign a text-based identifier and annotationlabels. To link a question to an answer, the developer selectsthe question and the answer in the corresponding lists andassigns the link value using the popup menu at the bottomof the window. More details about the interface and usesmay be found in (Leuski and Traum 2010).

4. EvaluationWe have evaluated NPCEditor in a number of off-line andon-line experiments. We have tested the classification ac-curacy, robustness to errors in the classifier input, and userengagement in interactions with a virtual human. In thissection we summarize some of these experiments. Moredetails about the experimental setup and the results can befound elsewhere (Leuski et al. 2006; Artstein et al. 2009;Leuski and Traum 2008; Kenny, Parsons, and Rizzo 2009).Evaluations of the performance of characters built using theNPCEditor are briefly described in Section 5.

Classification AccuracyIn the first set of experiments we have evaluated the classi-fication accuracy or how often the first answer returned bythe system was appropriate. As the baseline we used a textclassification approach based on Support Vector Machines(SVM). We represented questions as vectors of term featuresand the linked answers defined the question classes. Wetokenized the questions and stemmed the tokens using theKStem algorithm (Krovetz 1993) in exactly the same way aswe tokenize the text to compute language models. We used atf × idf weighting scheme to assign values to the individualterm features (Allan et al. 1998). Finally, we trained a multi-class SVM (SVMstruct) classifier with an exponential ker-nel (Tsochantaridis et al. 2004). We also experimented witha linear kernel function, various parameter values for the ex-ponential kernel, and different term weighting schemes. Thereported combination of the kernel and weighting schemeshowed the best classification performance. Such an ap-proach is well-known in the community and has been shownto work well in numerous applications (Joachims 1998). Webelieve it provides us with a strong baseline.

As the second baseline we used the language model ap-proach described in Section 2, but we compared questionsto questions instead of comparing them to answers. This isequivalent to single-language retrieval without the transla-tion effect of the question-answer mapping. Specifically, inEquation 5 we use the likelihood over question terms andsum over all sample questions. Given an input question, thistechnique retrieves the most similar sample question, and wereturn the answer linked to that question.

To evaluate the systems we used the language databasefrom the SGT Blackwell virtual human (see Section 5). Thedatabase contains 1,261 questions and 60 answer classes.We divided the collection of questions into training and test-ing subsets following the 10-fold cross-validation schemaand calculated the effectiveness of each approach.

accuracy impr. over SVM avg. prec.SVM 53.13SLM 57.80 8.78 63.88CLM 61.99 16.67 65.24

Table 1: Comparison of three different algorithms for an-swer selection on SGT Blackwell data. Each performancenumber is given in percentages.

We use two evaluation metrics to compare the systems’

performance. Firstly, we calculate the classification accu-racy or how often the first answer returned by the systemwas appropriate. Table 1 shows the accuracy numbers forthe two baselines (we call them “SVM” and “SLM”) andthe NPCEditor classification (“CLM”). The NPCEditor clas-sification approach is 17% more accurate than the SVMbaseline. It is also more accurate than the SLM baseline.The differences shown are statistical significant by t-test(p < 0.05).

Secondly, recall that both SLM and CLM methods areranking techniques. They order the answers in the databaseby their expected appropriateness. Both methods may re-turn several candidate answers depending on the thresholdvalue. Thus, an interactor may get a different response ifshe repeats the question (if there is more than one good an-swer). We want to measure the quality of the ranked list ofcandidate answers. As our second evaluation metric we useinterpolated average precision—a well-known IR measurethat for every appropriate answer in the list of candidatescomputes the proportion of appropriate answers among allpreceding answers and averages those values. We showthe average precision numbers for the SLM and CLM runs.The average precision scores are significantly higher for thecross-language approach than for the single-language ap-proach. It indicates that the CLM approach tends to rank ap-propriate answers above non-appropriate answers more of-ten than the SLM approach.

We have repeated the experiment on 7 other virtual char-acters with smaller language databases. We observed thatour system is more effective on problems with more answerclasses.

Classifier RobustnessIn the second set of experiments we evaluated the classifierrobustness to input errors. Recall that in a typical VH systemthe text input to the classifier comes from an ASR module,which may contain speech recognition errors. We thus ex-amined the impact of ASR quality on answer quality. We re-cruited 20 participants to interview the SGT Blackwell char-acter. Each participant asked 20 questions. We computedthe Word Error Rate (WER) for each ASR-transcribed ques-tion. A word error rate is the ratio of the total number ofword errors in a string (substitutions, deletions, and inser-tions) to the number of words in the correct string. Note thatWER can be greater than 100%. The average WER scorewas 37.33%.

We applied the NPCEditor classification approach to bothASR and human transcribed data and recorded the selectedanswers. We asked three human raters to judge the appropri-ateness of the selected responses using a 1-6 scale (Gandheet al. 2004). The Cronbach’s alpha value, measuring theinter-rater agreement, was above 0.91 indicating high con-sistency among the judges.

To judge the impact of ASR errors on classification ap-propriateness, we computed the cumulative average appro-priateness score (CAA) as a function of WER: for eachWER value t we average the appropriateness scores for allquestions-answer pairs with WER score less than or equalto t, as shown in equation 8, where p is a question-answer

pair, A(p) is the appropriateness score for p, and E(qp) isthe WER score for the ASR output of a spoken questionqp. CAA(t) is thus the expected value of the appropriatenessscore if the WER is at most t.

CAA(t) =1|S(t)|

∑p∈S(t)

A(p), S(t) = {p|E(qp) ≤ t} (8)

We computed two sets of CAA(t) values: one using theappropriateness scoresA for ASR-transcribed questions andthe other using the appropriateness score for the human tran-scribed questions. We examined the differences betweenthese scores at different values of WER. We observed thatthe differences are small and not statistically significant un-til WER reaches 60%. After that point the CAA score is sig-nificantly lower on the ASR transcribed data (by t-test withp < 0.05). We concluded that the classifier performance isnot significantly affected by the input errors if the amount oferror does not exceed 60%.

Interaction QualityKenny and his colleagues (Kenny, Parsons, and Rizzo 2009)study virtual humans for clinician training. They have builta virtual human using NPCEditor that plays a role of a pa-tient with a psychiatric problem and they wanted to assesswhether a virtual patient would respond to the clinician in-terview as a real patient would. They wanted to see if 1)clinicians could elicit proper responses from questions rel-evant for an interview from a virtual patient and 2) to eval-uate psychological variables such as openness and immer-sion of the participant and believability of the character asa patient. They have engaged 15 test subjects from a med-ical school including medical students, psychiatry residentsand fellows. Each subject conducted a 15 minute interviewwith the virtual patient trying to diagnose her condition andfilled out a set of questionnaires before and after the inter-view. The researchers analyzed the data from the interviewtranscripts and from the questionnaires and found that thesubjects were generally immersed in the interviews, they de-scribed the virtual patient character as believable and engag-ing, and they did ask and received responses covering allaspects of a typical patient interview. The study showed afeasibility of using virtual patients for training.

5. ApplicationsNPCEditor has been used as the language processing com-ponent for over a dozen virtual humans at ICT (some withmultiple versions), and several dozen elsewhere. Over adozen different developers have so far used the system tocreate or extend characters. The system has been deployedand administered in museums, in virtual worlds, at tradeshows and conferences, and in mobile vans by people notinvolved in their development. Thousands of people have in-teracted with these systems, and even more have seen themas audience to live interactions. In this section we describesome of the installations, highlighting their unique features.

SGT Blackwell Originally created as a showcase of VHtechnology, SGT Blackwell was introduced at the 2004Army Science Conference. He is a life-size 3D US Army

soldier projected onto a transparent screen in a mixed-realityenvironment. Conference attendees acted as audience withICT demoers who spoke to SGT Blackwell. The originaldomain had 83 responses on different topics covering hisidentity, origin, language and animation technology, designgoals, our university, the exhibition setup, and some mis-cellaneous topics, such as “what time is it?” and “wherecan I get my coffee?” After a lot of positive feedback fromthe attendees, several subsequent versions were built, usingthe NPCEditor. An extended version with additional domainitems was used for demos both at ICT and by the office ofthe US Army’s Director for Research and Laboratory Man-agement, with external administrators and demoers. It wasalso selected as a part of the Smithsonian’s National DesignTriennial, Design Life Now exhibit in 2006. The system wasinstalled at the Cooper-Hewitt Museum in New York fromDecember 2006 to July 2007 (and later at two other muse-ums), where SGT Blackwell was administrated by Museumstaff and interacted directly with over 100,000 visitors. Lim-ited versions of SGT Blackwell were also created speciallyfor the Director for Research and Laboratory Managementto interact with at the opening and closing of the 2006 ArmyScience conference, as well as a cameo appearance withSGT Star at the 2008 conference. An example of dialoguewith SGT Blackwell can be found in (Leuski et al. 2006).Preliminary evaluation of Blackwell in the Cooper-Hewittcan be found in (Robinson et al. 2008).

SGT Star Interactive SGT Star was funded by the USArmy Accessions Command, who wanted a mobile, life-sized, face to face version of their web character fromgoarmy.com4. SGT Star is like SGT Blackwell a life-sizerendering of an Army soldier that answers questions on top-ics including Army careers, training, education and moneyfor college5. He can also handle queries about the technol-ogy behind his development and explain how his creation fitsin with plans for future Army training environments. Thereare approximately 320 answers in his repertoire. The origi-nal version was used by Accessions command at trade showsand has since been ported to several “Army Adventure Vans”in which Army educational personnel interact with SGT Starabout Science and Army careers. The character databasewas constructed by a linguist in our lab, with consultationfrom scriptwriters and Army SMES and is administered andinteracted with by the vans’ Army staff. More about SGTStar, including a longitudinal evaluation at several conven-tions can be found in (Artstein et al. 2009).

Virtual Patients for Clinical Training Since 2006, theNPCEditor has been used to create virtual characters ex-hibiting psychological conditions who can interact verballyand non-verbally with a clinician in an effort to teach theclinician interpersonal skills such as interviewing and diag-nosis. Three virtual patient characters were developed bya separate team at the institute without the direct involve-

4The website version was developed by Next IT Corporationand does not share any technology with the ICT version.

5A video of an early version of the SGT Star character canbe found at our website: http://projects.ict.usc.edu/nld/group/videos/early-version-sgt-star.

ment of the NPCEditor creators. Each character databasecontained up to 200 responses. Users were medical and psy-chology students (Kenny, Parsons, and Rizzo 2009) .

Army Communication Skills Training Since 2006NPCEditor has been successfully used by the Program Ex-ecutive Office (PEO) Simulation, Training, and Instrumenta-tion (STRI), US Army, as a natural language understandingand processing component in a number of interactive train-ing systems that teach soldiers communication and culture-specific conversational skills. We have received very posi-tive feedback about NPCEditor from designers and develop-ers of the training systems. These systems have been fieldedin 19 training installations. As of this writing, more than3,000 soldiers (commissioned and noncommissioned) havereceived training using the system. An independent analysishas shown that the US Army has achieved significant sav-ings (as much as $30 million) in training systems researchand development costs by reusing this existing system andhave realized greater flexibility in the ability to respond totheater driven changing training requirements6. The design-ers and administrators are Army or contracted personnel out-side ICT, and the interactors are soldiers, using the systemsfor training.

Virtual World Guides Since 2008, NPCEditor has beenused to develop several AI avatars in online virtual worldsincluding Second Life and Active Worlds. These charactersare used for aides in educational settings as well as guidesof the virtual space. These characters have been designedand administrated at ICT, but the interactors were people inthe virtual worlds who came to visit the areas and interactwith the characters. In contrast to the other characters de-scribed in this section, the online virtual world characters donot use speech recognition but the native virtual world chatand IM facilities. Probably the most advanced of these isLT Moleno, a staff duty officer, who patroled the US ArmyWelcome island in Second Life for over a year. He answeredquestions about the island and conducted interactive tours ofthe island facilities. Over 4,000 visitors interacted with LTMoleno. More details on LT Moleno can be found in (Jan etal. 2009).

Gunslinger The Gunslinger project (Hartholt et al. 2009)is a mixed-reality interactive-entertainment experience thatcombines physical props with virtual humans. The partic-ipant physically walks into a saloon room situated some-where in Wild West and interacts with three life-sized vir-tual human characters projected onto screens built into thesaloon walls. The characters listen and talk to the partici-pant and to each other. They observe and react to partici-pant’s location in the room and his actions. For example,they notice when the participant takes out his gun and maycomment on it. NPCEditor handles both the language un-derstating and dialogue management parts of the system. Ituses a state-based dialogue manager which tracks the par-ticipant’s progress through the scenario and handles normalconversation flow. The dialogue manager allows differentresponses to interactor interruption and allows the charactersto exhibit initiative and start their own discussion topics.

6Personal communication. The report is not publicly available.

The character database and the dialogue manager scriptwere created by the Gunslinger group at the institute. Thesystem is setup for a single interactor, however hidden cam-eras at the set allow an audience to observe the interactionremotely.

InterFaces In the fall of 2008 the InterFaces exhibitopened up at the Boston Museum of Science7. The starsof the exhibit are Ada and Grace—two virtual docentswho have approximately 400 answers for questions aboutcomputers, robots, and communications, as well as them-selves and exhibits in the museum’s Cahners ComputerPlace (Swartout et al. 2010)8. The target audience for theexhibit are children from ages 7 to 14. NPCEditor driveslanguage understanding and dialogue management for bothcharacters. The system is operated by the museum volun-teers and one of the volunteers generally serves as the des-ignated interactor.

Virtual Human Toolkit NPCEditor is being used to cre-ate virtual humans in more and more diverse applications. Itis now part of a Virtual Human Toolkit that is a collection ofmodules, tools and libraries that allow developers to createtheir own virtual humans. The toolkit is available withoutcost for academic research purposes9. In September 2008we conducted a 3 day workshop, where approximately 30attendees, mostly graduate students from universities acrossthe country, designed and built 6 different characters for agame of “Clue” over two afternoons. Each character had ap-proximately 30 to 40 responses. This illustrates how quicklya novice character designer can develop a useful virtual hu-man.

6. Future WorkOne of the weakness of the classification-based languageprocessing is the requirement to collect sufficient languagedata. The character designer has to define all possible an-swers, specify sample questions for the answers, and linkthem together. We have experimented with characters thathave as many as 400 answers and 2,000 questions in theirlanguage databases. Our partners in the Army have devel-oped characters with 2,000 answers and more than 20,000questions. The classification approach scales well for largerdatasets. However, creating and expanding the language re-sources while maintaining consistency in the data can be-come very tedious. We plan to incorporate ideas from activelearning into NPCEditor to help with linking of questionsand answers. We are exploring techniques for automaticquestion and answer generation from descriptive text. Weare also interested to investigate solutions for mining rele-vant language data from chat archives or world wide web.

We have shown that the classification approach is robustto the errors in the input text when conditions permitting.However, the ASR performance degrades in noisy environ-

7http://www.mos.org/interfaces/8A video of the characters is available at our web-

site: http://projects.ict.usc.edu/nld/group/videos/jillian-talking-ada-and-grace

9For more information, see our website: http://vhtoolkit.ict.usc.edu/

ments or when dealing with heavily accented speech. Weare looking into alternative representations for the input textto improve the classifier robustness even further. For exam-ple, our recent experiments show that incorporating phoneticinformation into the input for the classifier provides smallbut significant improvement in quality. We are also explor-ing how to integrate multiple ASR hypotheses into the inputrepresentation.

NPCEditor started as a text classification tool for context-free question-answering dialogue where any question canbe asked at any time and the answer depends only on thequestion content. This type of interaction works well forkiosk-like applications in museums or show exhibits. As weare exploring other VH application domains there is a grow-ing need to support more sophisticated dialogue behaviors.There are two way that we can approach this. Firstly, we canintegrate complex dialogue handling into NPCEditor itself.Currently NPCEditor allows a designer to incorporate con-text information by defining multiple states for a single char-acter, specify a classifier for each state, and indicate whenthe inter-character state transitions occur. It integrates theclassifier technology with a dialogue scripting module facil-itating creation of characters with complex behavior. Oneof our goals is to continue the development of NPCEditor,refining the support for complex dialogue strategies. We arelooking into building components to visualize the dialoguestate network, tools for debugging the dialogue transitionsand state information.

Secondly, we can leave the dialogue management to a spe-cialized component. We have built systems where we usethe NPCEditor language classification capability to convertfrom the text input into a semantic representation and passthis information to a specialized dialogue manager (Leuskiand Traum 2008; Gandhe et al. 2008). Currently this con-nection is one-way: there is no feedback from the dialoguemanager for the classifier. We are experimenting with tech-niques for integrating dialogue context information into theclassification process with the hope that it will increase theaccuracy of the classification.

7. ConclusionsIn this paper we presented NPCEditor, a system for buildingand deploying virtual characters capable of engaging a userin spoken dialog on a limited domain. NPCEditor has beenused mainly for question answering characters where an in-teractor asks questions and the character responds. Howeverother types of dialogue have also been successfully imple-mented, for example, the Gunslinger system, in which char-acters take the initiative and question the user. The dialogmay have other forms as long as character responses can befully specified a priori.

NPCEditor contains a state of the art cross-language in-formation retrieval-based classifier that is robust to noisy in-put from speech recognition results. It contains a develop-ment environment that includes a user-friendly GUI to sup-port several classes of user, from developer to interactor andaudience. NPCEditor has been successfully evaluated in thelaboratory and field-tested and proved to be an effective andversatile system in a number of different applications.

AcknowledgmentsWe would like to thank the users of NPCEditor for manyhelpful suggestions that led to specific improvements in us-ability and functionality. We also would like to thank thereviewers for their invaluable suggestions and commentshelping us to improve the paper. The effort described herehas been sponsored by the U.S. Army Research, Develop-ment, and Engineering Command (RDECOM). Statementsand opinions expressed do not necessarily reflect the posi-tion or the policy of the United States Government, and noofficial endorsement should be inferred.

ReferencesAllan, J.; Callan, J.; Croft, W. B.; Ballesteros, L.; Byrd, D.; Swan,R.; and Xu, J. 1998. Inquery does battle with TREC-6. In SixthText REtrieval Conference (TREC-6), 169–206.Artstein, R.; Gandhe, S.; Gerten, J.; Leuski, A.; and Traum, D. R.2009. Semi-formal evaluation of conversational characters. InLanguages: From Formal to Natural, 22–35.Bahl, L. R.; Jelinek, F.; and Mercer, R. L. 1990. A maximum like-lihood approach to continuous speech recognition. In Readings inspeech recognition. San Francisco, CA, USA: Morgan KaufmannPublishers Inc. 308–319.Blei, D. M.; Ng, A. Y.; Jordan, M. I.; and Lafferty, J. 2003. La-tent dirichlet allocation. Journal of Machine Learning Research3:993–1022.Chu-Carroll, J., and Carpenter, B. 1999. Vector-based natu-ral language call routing. Journal of Computational Linguistics25(30):361–388.Gandhe, S.; Gordon, A.; Leuski, A.; Traum, D.; and Oard, D. W.2004. First steps toward linking dialogues: Mediating betweenfree-text questions and pre-recorded video answers. In Proceed-ings of the 24th Army Science Conference.Gandhe, S.; DeVault, D.; Roque, A.; Martinovski, B.; Artstein,R.; Leuski, A.; Gerten, J.; and Traum, D. 2008. From domainspecification to virtual humans: An integrated approach to author-ing tactical questioning characters. In Proceedings of InterspeechConference.Gorin, A.; Riccardi, G.; and Wright, J. 1997. How may i helpyou? Speech Communication 23:113–127.Gratch, J.; Rickel, J.; Andre, E.; Cassell, J.; Petajan, E.; andBadler, N. 2002. Creating interactive virtual humans: Some as-sembly required. IEEE Intelligent Systems 54–63.Gratch, J.; Wang, N.; Okhmatovskaia, A.; Lamothe, F.; Morales,M.; van der Werf, R.; and Morency, L.-P. 2007. Can virtualhumans be more engaging than real ones? In Jacko, J., ed.,Human-Computer Interaction. HCI Intelligent Multimodal Inter-action Environments, volume 4552 of Lecture Notes in ComputerScience, 286–297. Springer Berlin / Heidelberg.Grefenstette, G. 1998. Cross-Language Information Retrieval.Norwell, MA, USA: Kluwer Academic Publishers.Hartholt, A.; Gratch, J.; Weiss, L.; Leuski, A.; Morency, L.-P.;Marsella, S.; Liewer, M.; Thiebaux, M.; Doraiswamy, P.; andTsiartas, A. 2009. At the virtual frontier: Introducing Gunslinger,a multi-character, mixed-reality, story-driven experience. In Rut-tkay et al. (2009), 500–501.Hofmann, T. 1999. Probabilistic latent semantic indexing. InProceedings of the 22nd International ACM SIGIR Conference,50–57.

Jan, D.; Roque, A.; Leuski, A.; Morie, J.; and Traum, D. R. 2009.A virtual tour guide for virtual worlds. In Ruttkay et al. (2009),372–378.Joachims, T. 1998. Text categorization with support vector ma-chines: Learning with many relevant features. In Machine Learn-ing: ECML-98, Lecture Notes in Computer Science. chapter 19,137–142.Johnson, W. L.; Vilhjalmsson, H.; and Marsella, M. 2005. Seriousgames for language learning: How much game, how much AI?In Proceedings of the 12th International Conference on ArtificialIntelligence in Education.Katz, B. 1988. Using english for indexing and retrieving. In Pro-ceedings of the 1st RIAO Conference on User-Oriented Content-Based Text and Image Handling (RIAO ’88).Kenny, P. G.; Parsons, T. D.; and Rizzo, A. A. 2009. Hu-man computer interaction in virtual standardized patient systems.In Proceedings of the 13th International Conference on Human-Computer Interaction. Part IV, 514–523. Berlin, Heidelberg:Springer-Verlag.Krovetz, R. 1993. Viewing morphology as an inference process.In Proceedings of the 16th International ACM SIGIR Conferenceon Research and Development in Information Retrieval, 191–202.Lavrenko, V. 2004. A Generative Theory of Relevance. Ph.D.Dissertation, University of Massachusetts at Amherst.Leuski, A., and Traum, D. 2008. A statistical approach for textprocessing in virtual humans. In Proceedings of the 26th ArmyScience Conference.Leuski, A., and Traum, D. 2010. NPCEditor: A tool for buildingquestion-answering characters. In Proceedings of The SeventhInternational Conference on Language Resources and Evaluation(LREC).Leuski, A.; Patel, R.; Traum, D.; and Kennedy, B. 2006. Buildingeffective question answering characters. In Proceedings of the 7thSIGdial Workshop on Discourse and Dialogue.Lewis, D. D.; Schapire, R. E.; Callan, J. P.; and Papka, R. 1996.Training algorithms for linear text classifiers. In Proceedings ofthe 19th International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, 298–306.Ponte, J. M., and Croft, W. B. 1997. Text segmentation by topic.In Proceedings of the First European Conference on Research andAdvanced Technology for Digital Libraries, 120–129.Reeves, B., and Nass, C. 1996. The Media Equation. Cambridge:Cambridge University Press.Robinson, S.; Traum, D.; Ittycheriah, M.; and Henderer, J. 2008.What would you ask a conversational agent? observations ofhuman-agent dialogues in a museum setting. In Proceedingsof the 5th International Conference on Language Resources andEvaluation.Roque, A., and Traum, D. 2007. A model of compliance andemotion for potentially adversarial dialogue agents. In Proceed-ings of the 8th SIGdial Workshop.Ruttkay, Z.; Kipp, M.; Nijholt, A.; and Vilhjalmsson, H. H., eds.2009. Intelligent Virtual Agents, 9th International Conference,IVA 2009, Amsterdam, The Netherlands, September 14-16, 2009,Proceedings, volume 5773 of Lecture Notes in Computer Science.Springer.Swartout, W. R.; Traum, D. R.; Artstein, R.; Noren, D.; Debevec,P. E.; Bronnenkant, K.; Williams, J.; Leuski, A.; Narayanan, S.;and Piepol, D. 2010. Ada and grace: Toward realistic and engag-ing virtual museum guides. In IVA, 286–300.

Traum, D.; Swartout, W.; Gratch, J.; Marsella, S.; Kenney, P.;Hovy, E.; Narayanan, S.; Fast, E.; Martinovski, B.; Bhagat, R.;Robinson, S.; Marshall, A.; Wang, D.; Gandhe, S.; and Leuski, A.2005. Dealing with doctors: Virtual humans for non-team interac-tion training. In Proceedings of ACL/ISCA 6th SIGdial Workshopon Discourse and Dialogue.Tsochantaridis, I.; Hofmann, T.; Joachims, T.; and Altun, Y.2004. Support vector machine learning for interdependent andstructured output spaces. In Proceedings of the Twenty-first inter-national conference on Machine learning.Voorhees, E. M. 2003. Overview of the trec 2003 question an-swering track. In Proceedings of The Twelfth Text Retrieval Con-ference, 54–69.Wei, X., and Croft, W. B. 2006. Lda-based document models forad-hoc retrieval. In Proceedings of the 29th annual internationalACM SIGIR conference on Research and development in informa-tion retrieval, SIGIR ’06, 178–185. New York, NY, USA: ACM.