information extraction tools and methods for understanding dialogue in a companion r. catizone, a....
Post on 24-Dec-2015
215 Views
Preview:
TRANSCRIPT
Information Extraction Tools and Methods for Understanding Dialogue in a
Companion
R. Catizone, A. Dingli, H. Pinto and Y.WilksUniversity of Sheffield
LREC 2008
Morocco
Senior Companion
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Photos
Yellow pagesNews
Developing a range of companions to assist the elderly
The Senior Companion is about…
QuickTime™ and aPhoto - JPEG decompressor
are needed to see this picture.
the user
…through his photographs
Multimodal dialogue for building a picture of…
Behind the scenes using the dialogue to
…tag photos
Octavia
family holiday
Venice
2007
summer
Photos• A Photograph application for reminiscing about personal photos. • The user begins with a set of photos which will be input into the
system in advance.• The system engages in conversation with the user about the photos.
• The system will create a life narrative by extracting key facts from
the user about his/her photos and the people, places and events that are represented.
• The information about the photos can later be retrieved and displayed in a future user session.
• Semantic features extracted during the dialogue will be automatically associated with the photos ( may include audio recordings)
Life Narrative through Photos• Life Narrative - Dynamically builds segments of a person’s life
that correspond to relationships with:– People and Places with respect to
• Life events (Birthdays, marriages, etc)• Journeys and Travels• Special memories
News
• News reading
• Live from BBC RSS feeds
• Retrieved through categories– Sports, business, international, etc.
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Yellow pages (future)
• To incorporate helping with everyday interactions such as yellow pages assistant for finding information.
• Example for finding a plumberUser: “Hello Morgan, my toilet is broken, I really need to find a plumber. Can you help me.”System: “Sure I can,the broken toilet is at your home on Hayfield Rd right?User: “Yes”System: ”Let me have a look and see what I can find.”User: “Ok”System: “Yes, I’ve found a local plumber in your area. Would you like me to get him on the
phone using Skype.”User: “Yes please”System: “Ok here you go”
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Qui
ckTi
me™
and
a
TIFF
(Unco
mpr
esse
d) d
ecom
pres
sor
are
need
ed to
see
this p
ictu
re.
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
The System
Interface LayerNLU
Fusion
Input Queue
Output Manager
Napier Interface
AvatarDialogAct
Tagger
GATE
DialogManager
TripleStoreNLG
Senior Companion Architecture
Modules
• Interface + Avatars• Face identification (Open CV)• Fusion• ASR• Natural Language Understanding• Dialogue Manager• Knowledge Representation/Reasoning• TTS
Interface
• Multimodal - speech and touch (using a touchscreen tablet)
• Includes photos, avatar and text box• Currently designed to discuss user’s photos and read
news.• Displays multiple photos at a time
– Photo management – Photo selection
Avatars
• Experimenting with different avatars– Morgan (AAA)– Crazy Talk
• Woman• Ken Dodd• Lion• A computer
Interface (2)
Interface (2)
Please read me the news
OpenCV
• Face identification software– Finds the coordinates of all of the faces in a photo.– Cannot recognize the same face in more than one
photo, but investigating face recognition software: Polar Rose
Photo Applications
• Also use digital photo metadata such as– Date and time– GPS coordinates
• Photos that have annotations (Facebook)
More information to start makes for more interesting dialogue
Samuel Carter
Thomas Clark Ed Bloch
Olivia Ford
Fusion
• Merges speech and pointing input
“This is my daughter Octavia”
ASR
• Dragon Naturally Speaking– requires 10 minute training session– accuracy is high - up to 99%– application integrated with the SC– full integration of the code using the
Nuance SDK planned for next version
Natural Language Understanding
• Sentence splitting• POS tagger• Parser• Annie Named Entity Recognizer (GATE, USFD)
– person names (10,000)– locations (Web trawl)– 60 relation names
• mother, brother, sister, friend, etc.
• Dialogue Act Tagger (ALB).• Populates Ontology instances.• To use an IE approach for semantic interpretation
NLU: The task so far …
1. Identify people and their relationships to the user and each other
2. Identify locations
The process
Gate
Annie
Dialogue Act Tagger
Syntactic Parser
Ontology Processing
Utterance
Syntactic and Semantic representation
Gate
I went to meet Lynne, my old flatmate, together with my sister Francesca.
PRP VBD TO VB NNP PRP$ JJ NN RB IN PRP$ NN NNP
Verb Phrases
Noun Chunks
Annie
I went to meet Lynne, my old flatmate, together with my sister Francesca.
Relationships
Persons
Syntactic Parser
Who is the sister and who is the flatmate?
I went to meet Lynne, my old flatmate, together with my friend Francesca.
((((I))((went)(((to)(meet)(Lynne)(,)(my)(old)(flatmate)(,))( (together))((with)( (my)(sister)(Francesca))))))(.)))
(((I)(went((to meet Lynne , my old flatmate,)( together)(with( my sister Francesca))))).))
(((I)(went((to meet Lynne , my old flatmate,)( together)(with( my sister Francesca))))).))
Syntactic bindings will help us identify that!
Ontology
Reasoning
• Allows us to infer new knowledge
– Sister(My) = Francesca– Flatmate(My) = Lynne
– If we have in the ontology:• Mother(My) = Mary
– We can infer through the properties of the Sister relationship that• Mother(Francesca) = Mary• Daughter(Mary) = Francesca
Knowledge Base
• Relationships Ontology– Almost 40 person classes (mother, father, etc)– Almost 40 relationships (has friend, etc)
Will be adding more …
• Locations Ontology– Most continents, countries, regions, cities– Plus
• several places of interests per region• top 10 things to do in cities
Will be adding more …
* ASSERT ("This is my sister". Yes and no are also considered asserts, where they are responses to yes/no questions, thus abbreviated assertions.) * OFFER ("Shall we look at another picture?") * COMMIT ("Okay I'll do that") * EXPRESSION (All social expressions such as "you're welcome". Also things like "wow!" and "great!") * INFORMATION REQUEST (open question) * CONFIRMATION REQUEST (yes/no question) * REPEAT REQUEST ("Pardon?") * ACTION DIRECTIVE ("Show me another one." All imperatives.) * OPEN OPTION ("We could look at another picture." Stating an option in a way that doesn't demand an answer.) * OPENING ("Hi") * CLOSING ("Goodbye") * ANSWER (An answer is invariably also an assert. Yes/no answers are asserts.) * BACKCHANNEL ("Uhuh") * REPEAT REPHRASE (Expressing understanding by paraphrasing) * COMPLETION (Completing the utterance of the other speaker) * NON-UNDERSTANDING ("I don't understand") * CORRECTION (An assertion that corrects a previous assertion) * ACCEPT (Accepting a proposal)
* REJECT (Rejecting a proposal)
DAMSL Dialogue Act set
Semantic specification
• Each dialogue begins with a user object and a picture object– User/Person objects have
• name, people relations,age
– Picture object• location, occasions, people, dates)
NameRelationsAge
LocationOccasionPeople
Picture object
User object
QuickTime™ and aPhoto - JPEG decompressor
are needed to see this picture.
Person: Zoe
Person: Roisin, snooks
Occasion: Hilary Duff concertLocation: Birmingham
Information Extraction in the SC
• Why do we want to use IE for Semantic representation?
User utterances are unstructured
• Using GATE IE tools to create templates – Relationship between the Named Entities and the
significant Events.• Categorize events into meaningful classes
Information Extraction (2)• Example sentence
‘That is my daughter Zoe on the right’
• Simple Example template : Relation
• Person1 Related-to Person2– Related-to : is-relation
» is-relation: is-daughter, is-mother etc. is-daughter: lexical string: ‘is my daughter’
• Filled IE Template
Relation : [Person1=‘Zoe’], [Person2=‘Roberta’], [Related-to=is-daughter]
SC Ontology
SC Ontology for Inference
“Here are my daughters, Zoe and Octavia in New York City”
Infer using the family relations Ontology that:-Zoe and Octavia are sisters-Roberta is the mother of Octavia and Zoe
Dialogue Manager (1)• Manages discussion of
– User’s photos with respect to • Location• Time it was taken• Occasion• People in the Photo (exploits positional information of people)
– Name– Age– Relation to the user
– Photo Management• “Show me all the photos of my mother”• Photo selection
– News• Reading and stopping• Choice of politics, sport or business
Dialogue Manager (2)• Accepts pre-annotated photos to allow the
system to engage in more interesting conversation more quickly.
• Handles basic photo management tasks:“Show me all the photos of my mother”“Please move on to the next photo”
– Responds when same person is mentioned in more than one photo (using the user’s name)
• Remembers user information from multiple sessions
• Generation : template based
Dialogue manager (3)
• Dialogue Manager adapted from COMIC DM– General purpose control structure that does the dialogue
planning– Stack based system
• Conforms to common behaviour of conversation: Discuss topic 1, move to topic 2, go back to topic 1.
– Augmented transition networks, called Dialogue Action Forms (DAFs) for handling domain sub-tasks.
– References the Dialogue History, Knowledge base and the User Model
Dialogue Action Forms
• GUI editor for creating DAFs
• Composed of nodes and arcs containing tests and actions
• DAFs pre-stacked, but can be overidden by matching indexing terms (semantic classes, significant words)– Essential for mixed-initiative conversation
SC Dialogue Manager Stack
goodbye
greeting
Run Greeting DAFPop greeting DAFPush photo DAF
goodbye
photo
System start
goodbye
location
event
name
date
occasion
goodbye
location
event
date
Run photo DAFRun people DAFPop people DAF
People DAF
occasion
relation
age
people
DAFs (1)
Machine Learning in the Senior Companion
• Using first Senior Companion prototype to generate more data, augmenting WoZ data gathered by NAP and AAA and hand annotated.
• Plan to “tile” (as in Hearst) the data to seek segmentations corresponding to topics and dialogue moves.
• Plan to generalise across a set corresponding to “same” topic or move to generate a draft Dialogue Action.
www.companions-project.org
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Thank You
Evaluation of the Senior Companion
– Accuracy • Is the information that the system discusses accurate
– Important when the user returns for repeat sessions (system needs to remember and recall collected information at the appropriate time).
• Does the system discuss things in an efficient way? (not ask for clarification when the information is already known)
– User satisfaction• Self-assessment: May include some testing of the user’s level
of contentment with the system while running.
top related