goteborg university dialogue systems lab introduction to trindikit dialogue systems 2004 staffan...

Goteborg University Dialogue Systems Lab

Introduction to TrindiKit

Dialogue Systems 2004

Staffan Larsson


What is TrindiKit?

• a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach

• not a dialogue system!


• architecture & concepts• what’s in TrindiKit?• extending TrindiKit• building a system

This lecture


Architecture & concepts


module1 module…

Total Information State (TIS)•Information state proper (IS)•Module Interface Variables•Resource Interface Variables

resource1

control

modulei modulej module… modulen

resource… resourcem

DME


• an abstract data structure (record, DRS, set, stack etc.)

• accessed by modules using conditions and updates

• the Total Information State (TIS) includes– Information State proper (IS)– Module Interface variables– Resource Interface variables

Information State (IS)


Dialogue moves

• abstract description of utterances• e.g. speech acts


• module or group of modules responsible for – updating the IS based on observed moves– selecting moves to be performed

• dialogue moves are associated with IS updates using IS update rules– there are also update rules no directly associated with any

move (e.g. for reasoning and planning)

• update rules: rules for updating the TIS– rule name and class– preconditon list: conditions on TIS– effect list: updates on TIS

• update rules are coordinated by update algorithms

Dialogue Move Engine (DME)


• Modules (dialogue move engine, input, interpretation, generation, output etc.) – access the information state– no direct communication between modules

• only via module interface variables in TIS• modules don’t have to know anything about other modules• increases modularity, reusability, reconfigurability

– may interact with user or external processes

• Resources (device interface, lexicons, domain knowledge etc.)– hooked up to the information state (TIS) – accessed by modules– defined as object of some type (e.g. ”lexicon”)

Modules and resources


What’s in TrindiKit?


What does TrindiKit provide?• High-level formalism and interpreter for

implementing dialogue systems– promotes transparency, reusability, plug-and-play,

etc.– allows implementation and comparison of dialogue

theories – hides low-level software engineering issues

• GUI, WWW-demo • Ready-made modules and resources

– speech– interfaces to databases, devices, etc.– reasoning, planning


• a library of datatype definitions (records, DRSs, sets, stacks etc.)– user extendible

• a language for writing information state update rules

• GUI: methods and tools for visualising the information state

• debugging facilities– typechecking– logs of communication modules-TIS– etc.

TrindiKit contents (1)


• A language for defining update algorithms used by TrindiKit modules to coordinate update rule application

• A language for defining basic control structure, to coordinate modules

• A library of basic ready-made modules for input/output, interpretation, generation etc.;

• A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc.

TrindiKit contents (2)


Special modules and resources included with TrindiKit

• OAA interface resource– enables interaction with existing software

and languages other than Prolog

• Speech recognition and synthesis modules– TrindiKit shells for off-the-shelf recognisers– currently only ViaVoice, but more on the way

• Possible future modules:– planning and reasoning modules– multimodal input and output


Asynchronous TrindiKit

• Internal communication uses either– OAA (Open Agent Architecture) from SRI, or– AE (Agent Environment), a stripped-down

version of OAA, implemented for TrindiKit

• enables asynchronous dialogue management– e.g.: system can listen and interpret, plan

the dialogue, and talk at the same time


How to build a system


TrindiKitinformation state approach

How to use TrindiKit

• We start from TrindiKit– Implements the information state

approach– Takes care of low-level programming:

dataflow, datastructures etc.


TrindiKit

basicdialogue theory

basic system

information state approach

How to build a basic system

• Formulate a basic dialogue theory – Information state– Dialogue moves– Update rules

• Add appropriate modules (speech recognition etc)


TrindiKit


basic system


genre-specific theoryadditions

genre-specific system

How to build a genre-specific system

• Add genre-dependent IS components, moves and rules


TrindiKit


domain & languageresources

basic system

application


genre-specific theoryadditions

genre-specific system

How to build an application

• Add application-specific resources


• Come up with a nice theory of dialogue• Formalise the theory, i.e. decide on

– Type of information state (DRS, record, set of propositions, frame, ...)

– A set of dialogue moves– Information state update rules, including

rules for integrating and selecting moves– DME Module algorithm(s) and basic control

algorithm – any extra datatypes (e.g. for semantics:

proposition, question, etc.)

Building a domain-independent Dialogue Move Engine


Domain independence of the Dialogue Move Engine

• The DME is domain independent, given a certain type of dialogue– information-seeking – instructional– negotiative– ...

• Domain independence of DME is not enforced by TrindiKit, but is good practice– promotes reuse of components– forces abstraction from domain-specific details,

resulting in a more general theory of dialogue


Specifying Infostate type

• the Total Information State contains a number of Information State Variables– IS, the Information State ”proper”– Interface Variables

• used for communication between modules

– Resource Variables• used for hooking up resources to the TIS, thus

making them accessible from to modules

• use prespecified or new datatypes


sample infostate type declaration

infostate_variable_of_type( is, IS ) :- IS = record( [

private : Private, shared : Shared ] ),

Shared = record( [ com : set( proposition ), qud : stack( question ), lm : set( move ) ] ),

Private = record( [ agenda: stack( action ), plan : stackset( action ), bel : set( proposition ),

tmp : Shared ] ) ] ).


resulting infostate type

PRIVATE :

PLAN : stackset( Action )

AGENDA : stack( Action )

SHARED :

BEL : set( Prop )

TMP : (same type as SHARED)

COM : set( Prop )

QUD : stack( Question )

LU: [SPEAKER:dp, MOVES:set( Move )]


Sample interface variable type declarations

interface_variable_of_type( input, string ).

interface_variable_of_type( output, string ).

interface_variable_of_type( latest_speaker, speaker ).

interface_variable_of_type( latest_moves, set(move) ).

interface_variable_of_type( next_moves, set(move) ).


Specifying a set of moves

• amounts to specifying objects of type move (a reserved type)– there may be type constraints on the

arguments of moves

• preconditions and effects of moves– formalised in update rules, not in the move

definition itself– a move may have different effects on the IS

depending e.g. on who performed it


sample move specifications

% Social

of_type( quit, move ).

of_type( greet, move ).

of_type( thank, move ) .

% Q&A

of_type( ask(Q), move ) <- of_type( Q, question ).

of_type( inform(P), move ) <- of_type( P, proposition).

of_type( answer(R), move ) <-

of_type( R, proposition) or

of_type( R, ellipsis ).


Writing rules

• rule = conditions + updates– if the rule is applied to the IS and its

conditions are true, the updates will be applied to the IS

– conditions may bind variables with scope over the rule (prolog variables, with unification and backtracking)


A sample rule

rule( integrateUsrAnswer, [ $/shared/lu/speaker = usr,

assoc( $/shared/lu/moves, answer(R), false ),

fst( $/shared/qud, Q ),

$domain : relevant_answer( Q, R ),

$domain : reduce(Q, R, P)

], [

set_assoc( /shared/lu/moves, answer(R),true),

shared/qud := $$pop( $/shared/qud ),

add( /shared/com, P ) ] ).


A sample rule (old syntax)

• rule( integrateUsrAnswer, [ – val#rec( shared^lu^speaker, usr ), – assoc#rec( shared^lu^moves, answer( R ), false ),– fst#rec( shared^qud, Q ), – domain :: relevant_answer( Q, R ), – domain :: reduce(Q, R, P) – ], [ – set_assoc#rec( shared^lu^moves, answer(R),true), – pop#rec( shared^qud ), – add#rec( shared^com, P ) ] ).


Writing rules

• available conditions and updates depend on the infostate type– the infostate is declared to be of a certain

(usually complex) type

• datatype definitions provide– selectors: Sel(InObj,SelObj)– relations: Rel(Arg1, …, ArgN)– functions: Fun(Arg1, …, ArgN,Result)– operations:

Op(ObjIn,Arg1, …, ArgN,ObjOut)

• New datatypes may be added


Writing rules: locations in TIS

• objects may be specified by giving a path to a location in the infostate; – paths are specified using selectors, which are similar to

functions• $$Sel2($$Sel1) ~ $Sel1/Sel2 • $$fst($/shared/qud) ~ $/shared/qud/fst

– ”$” evaluates a path and gives the object at the location specified

– ”$$” evaluates a function– $$fst($/shared/qud) = $/shared/qud/fst

• example: – is/shared/com is a path, pointing to a location in the TIS– $is/shared/com is the object in that location– the is can be left out, giving $/shared/com


Writing rules: conditions (1)• conditions do not change the information state• if a condition fails, backtracking ensues• condition syntax (incomplete)

– Rel(Arg1, … , ArgN), e.g. • fst($/shared/qud,Q)

– Arg1:Rel(Arg2,…,ArgN), e.g. • $/shared/qud:fst(Q)• $domain:relevant_answer(Q,A)

– Arg1 = Arg2• Q = $$fst($/shared/qud)

– Cond1 and Cond2– Cond1 or Cond2– not Cond1– forall(Cond1, Cond2)– (Arg is object or prolog variable)


Writing rules: conditions (2)

• quantification, binding and backtracking– if an instantiation a of a variable V in a condition C is

found that makes condition C true, V is bound to a– backtracking occurs until a successful instantiation of

all variables in the list of conditions has been found

• example list of conditionsfst($/shared/qud,Q), in($/shared/com,P),$domain:relevant_answer(P,Q)

• Explicit quantificationQ.P. fst($/shared/qud,Q) & in($/shared/com,P) & $domain:relevant_answer(P,Q)


Writing rules: updates

• updates change the information state• if an update fails, an error is reported• variable bindings survive from conditions to

updates• update syntax (incomplete)

– Op(Path,Arg1,…,ArgN)• push(/shared/qud, Q)

– Path : Op(Arg1, … ,ArgN)• /shared/qud : push(Q)

– Store := Fun(Obj,Arg1,…,ArgN)• /private/tmp/qud := $$push($/shared/qud,Q)


Specifying update algorithms

• uses rule classes• constructs include

– Rule– RuleClass– if Cond then S else T– repeat R until C– repeat R– try R– R orelse S– test C– SubAlgorithm


Sample update algorithmgrounding, if $latest_speaker == sys then

try integrate, try database, repeat downdate_agenda, store

else repeat

integrate orelse accommodate orelse find_plan orelse

if (empty ( $/private/agenda ) then manage_plan else downdate_agenda

repeat downdate_agendaif empty($/private/agenda))then repeat manage_plan

repeat refill_agendarepeat store_nim try downdate_qud


Specifying serial control algorithms

• serial constructs include– Module{:Algorithm}– if Cond then S else T– repeat R until C– repeat R– try R– R orelse S– test C– SubAlgorithm


Specifying concurrent control algorithms

• Agent1 | Agent2 | … | AgentN

• where Agenti is • AgentName : {

– import Module1 ,–

… – import Modulep ,– Trigger1 => SerialAlgoritm1 ,– …– Triggerm => SerialAlgoritmm }

• triggers:– condition(C) (C is a subset of the full condition set)– init– new_data(Stream)


Sample control algorithm (1)

repeat ( [ select,

generate,

output,

update,

test( $program_state == run ),

input,

interpret,

update ] )


Sample control algorithm (2)input: {

init => input:display_prompt, new_data(user_input) => input }

| interpretation: { import interpret, condition(is_set(input)) => [ interpret, print_state ] }

| dme: { import update, import select, init => [ select ], condition(not empty(latest_moves)) => [

update, if $latest_speaker == usr then select

] }

| generation: { condition(is_set(next_moves)) => generate }

| output: { condition(is_set(output)) => output } )).


From DME to dialogue system

Build or select from existing components: • Modules, e.g.

– input– interpretation– generation– output

• Still domain independent• the choice of modules determines e.g.

the format of the grammar and lexicon


Domain-specific application

Build or select from existing components:• Resources, e.g.

– domain (device/database) interface– dialog-related domain knowledge, e.g. plan

libraries etc.– grammars, lexicons


Extending TrindiKit


You can add

• Datatypes– Whatever you need

• Modules– e.g. General interfaces to speech

recognizers and synthesizers

• Resources– E.g. General interfaces to (passive) devices

• Important that all things added are reasonably general, so they can be reused in other systsems


Datatype definitions

• relations– relations between objects; true or

false– format: relation(Rel,Args).– Example

• definition: relation(fst,[stack([E|S]),E]).• condition: fst($/shared/qud,Q)


Datatype definitions

• functions– functions from arguments to result– format: function(Fun,Args,Result).– Example

• definition: function(fst,[stack([E|S])],E). • in condition:

– Q = $$fst($/shared/qud)– Q = $/shared/qud/fst

• in effect: – next_move/content := $$fst($/shared/qud)

– every function corresponds to a relation relation(Fun,[Args@[Result]]).


Datatype definitions (2)

• selectors – selects an object (Obj) embedded in

another object (Arg)– selector(Sel,Arg,Obj,ArgWithHole,Hole).

– e.g. selector(fst,stack([E|S]),E,stack([H|S]),H).

– Every selector corresponds to a function function(Sel,[Arg],Object).


Datatype definitions (3)

• operations– operation(Op,InObj,Args,OutObj).

– e.g. operation(push,stack(S),[E],stack([E|S])).

– every operation corresponds to a relation relation(Op,[InObj|Args]@[OutObj]).

– push($/shared/qud,Q,$/shared/qud).


Building modules• DME modules

– Specific to a certain theory of dialogue management

– Best implemented using rules and algorithms

• Other modules– Should be more general, less specific to certain

theory of dialogue management– May be easier to implement directly in prolog or

other language• DME-ADL only covers checking and updating the infostate• These modules may also need to interact with other

programs or devices


Building resources

• Resource– the resource itself; exports a set of predicates

• Resource interface– defines the resource as a datatype T, i.e. in terms of

relations, functions and operations

• Resource interface variable– a TIS variable whose value is an object of the type T

• By changing the value of the variable, resources can be switched dynamically– change laguage– change domain


Sample resource variable type declarations (incl. resource interface)

resource_type( lexiconT ).

resource_variable_of_type( lexicon, lexiconT ).

of_type( lexicon_travel_english, lexiconT ).

of_type( lexicon_autoroute_english, lexiconT ).

of_type( lexicon_travel_svenska, lexiconT ).

of_type( lexicon_cellphone_svenska, lexiconT ).

resource_condition(lexiconT,input_form(Phrase,Move),Lexicon) :- Lexicon : input_form( Phrase, Move ).

resource_condition(lexiconT,output_form(Phrase,Move),Lexicon):-

Lexicon : output_form( Phrase, Move ).


Conclusion


• explicit information state datastructure – makes systems more transparent – enable e.g. context sensitive interpretation,

distributed decision making, asynchronous interaction

• update rules – provide an intuitive way of formalising

theories in a way which can be used by a system

– represent domain-independent dialogue management strategies

TrindiKit features


TrindiKit features cont’d

• resources– represent domain-specific knowledge– can be switched dynamically

• e.g. switching language on-line in GoDiS

• modular architecture promotes reuse– basic system -> genre-specific systems– genre-specific system -> applications


Theoretical advantages of TrindiKit

• theory-independent– allows implementation and comparison of

competing theories– promotes exploration of middle ground

between simplistic and very complex theories of dialogue

• intuitive formalisation and implementation of dialogue theories– the implementation is close to the theory


Practical advantages of TrindiKit

• promotes reuse and reconfigurability on multiple levels

• general solutions to general phenomena enables rapid prototyping of applications

• allows dealing with more complex dialogue phenomena not handled by current commercial systems


Availability

• Version 3.2 avaliable• TrindiKit website

– www.ling.gu.se/projects/trindi/trindikit

• SourceForge project– development versions available– developer community?

• licensed under GPL• more info in

– Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000

– TrindiKit manual (available from website)


Conclusions: TrindiKit & Information State Approach

• a toolkit for dialogue systems R&D• freely available to researchers• close the gap between theory and

practice of dialogue systems• theory-independent• promotes reuse and

reconfigurability on multiple levels


TrindiKit and VoiceXML

• VoiceXML– industry standard– form-based dialogue manager– web/telephony infrastructure– requires scripting dialogues in detail

• Theory-specific?– VoiceXML implements a specific theory of

dialogue– TrindiKit allows implementation of several

different theories of dialogue– More complex dialogue phenomena hard to deal

with in VoiceXML


TrindiKit and VoiceXML, cont’d

• Combine VoiceXML with TrindiKit?– future research area – support connecting TrindiKit to VoiceXML infrastructure– use TrindiKit system as VoiceXML server, dynamically building VoiceXML

scripts– convert VoiceXML specifications to e.g. GoDiS dialogue plans

• Implement VoiceXML in TrindiKit– Information state?

• forms? set of forms? as records?– Dialogue moves? Just one? Or two: block, prompt?– Update rules?

• implements FIA– Is asynchronous TrindiKit needed?

• events• filler actions

– How deal with grammars on different levels?• probably need to load SR grammars dynamically

• Implement TrindiKit system (e.g. GoDiS) in VoiceXML – use JavaScript


Possible future modifications

• regler i klasser väljs indeterministiskt

• test C -> if C then halt / break (gå ur loop)

• ta bort möjlighet att ha flera regler med samma namn

goteborg university dialogue systems lab introduction to trindikit dialogue systems 2004 staffan...

Documents

trindikit dialogue systems

dialogue moves

update algorithms dialogue

information state approach

tis update rules

tis modules dont

control module

engine dme slide