1 enhancing interactions with to-do lists using artificial assistants yolanda gil, timothy chklovski...
TRANSCRIPT
1
Enhancing Interactions with To-Do Lists Using Artificial Assistants
Yolanda Gil, Timothy Chklovski
USC/Information Sciences Institute{gil, timc}@isi.edu
March 26, 2007
2
Structured statements about tasks and objects: - setting up a videoconference
camera: switch on computer: turn on, start up, link up microphone: turn on, test, adjust
Problems & remedies: - during a meeting
projector not available locate a portable projector
Learning Common Knowledge from Volunteers to Support Assistance
Learning about tasks
Learning to anticipate and repair
600,000+ statements collected over 12 months
Targets collection by topic and knowledge type
Learning about objects Learner
Proactively broadens coverage
Formulates relevant followup
questions in real time
Validates through other users
Guides knowledge entry
Learning paraphrases
3
To Do Lists
To Do lists are pervasive, and present large opportunity for assistance and learning We’ve been working with TOWEL, the To Do list
manager in CALO Glimpses / hints of users’ goals
To Do lists Have some regularity & structure Contents and surface form may vary widely Similar to the collected statements
4
To Do Lists: Related Work
Ethnographic studies look at usage of To Do lists Eg: V Bellotti, B Dalal, N Good, P Flynn, D Bobrow, N
Ducheneaut. What a To-Do: Studies of Task Management Towards the Design of a Personal Task List Manager. CHI 2004
Analysis of work activities and how tools may support it, cognitive aids
B Harrison. An Activity-Centric Approach To Context-Sensitive Time Management. In CHI 2004: Workshop on the Temporal Aspects of Work.
A Dey, G Abowd. CybreMinder: A Context-Aware System for Supporting Reminders. HUC, 2000
Some commercial tools support prioritization Based on activity type, urgency, time available Eg, Life Balance, http://www.llamagraphics.com/, Voo2Do,
“Remember the milk”, Tada.com (provides public to do lists) Not focused on NL interpretation or on automation, mostly
on human factors and usability
5
Assisting with To Do Lists: the Idea
Key idea: Develop interpretation & mapping of To Do entries to assistant-supported tasks
Exploit large knowledge repositories and preprocessed texts
Paraphrases to help interpret text Use knowledge repositories to interpret and connect user
knowledge to operationalized tasks in CALO
Build on prior work on volunteer collection of paraphrases to assist speech recognition / utterance identification [Chklovski, KCAP ’04 & KCVC ’04]
Build on prior work on extraction from large corpora [Chklovski & Pantel, EMNLP ’05], and volunteer validation
6
BEAM: “Broad-coverage Entity Analysis and Mapping”
To Do interface integrating BEAM and providing task interpretation and monitoring via CALO’s TOWEL
List is automatically updated when the “Plan conference travel” action is completed
To Do entry made by user
BEAM mapping to TOWEL task
User To Do EntriesFind hotel w/ pool for Joe
Reserve conf room for talk
Buy bread on the way home
Get talk abstract from Joe
CALO Task Ontology:Catalog of Automated
Procedures
Execution and Monitoring
InstrumentedDesktop
TaskLearning
Set airport pickup for Joe
Announce room for talk
Reserve accommodations
Host a visitor
Arrange meeting
Opportunities for Interpretation-based Assistance
Map entries to task procedures
Anticipate & suggest missing
Entries, sub-tasks
Assist with how activities are done in the organization
BEAM
Initiate and report execution
Group and organize entries
User To Do EntriesFind hotel w/ pool for Joe
Reserve conf room for talk Map entries to task procedures
Anticipate & suggest missing
Entries, sub-tasks
Assist with how activities are done in the organization
BEAM
Buy bread on the way home
Get talk abstract from Joe
Initiate and report execution
Group and organize entries
CALO Task Ontology:Catalog of Automated
Procedures
Execution and Monitoring
InstrumentedDesktop
TaskLearning
Set airport pickup for Joe
Announce room for talk
Reserve accommodations
Host a visitor
Arrange meeting
OntologiesNGrams
Repository“to schedule a meeting”
SubtasksRepository:
“Reserve X has-subtask
find X”
Organization-specificTask knowledge
“Airport pickup of visitorsis common, but not here”
Action ParaphrasesRepository:
“plan X schedule X”“lease car rent car”
Verb Relations
Repository:“schedule
happens-before reschedule”
From Text ExtractionFrom Web Volunteers From Volunteers in the Organization
From Knowledge Engineers
BEAM Knowledge Sources for NL Interpretation and Assistance
RepairsRepository:
“If projector not working,
try a new bulb”
Map entries to task procedures
Anticipate & suggest missing
Entries, sub-tasks
Assist with how activities are done in the organization
BEAM
Initiate and report execution
Group and organize entries
User To Do EntriesFind hotel w/ pool for Joe
Reserve conf room for talk
Added by BEAM
Added by user
Deletedby BEAMbecause unnecessary
Markedby BEAMas user-only
Executed& monitored
Buy bread on the way home
Get talk abstract from Joe
OntologiesNGrams
Repository“to schedule a meeting”
SubtasksRepository:
“Reserve X has-subtask
find X”
Organization-specificTask knowledge
“Airport pickup of visitorsis common, but not here”
Action ParaphrasesRepository:
“plan X schedule X”“lease car rent car”
Verb Relations
Repository:“schedule
happens-before reschedule”
From Text ExtractionFrom Web Volunteers From Volunteers in the Organization
From Knowledge Engineers
BEAM Knowledge Sources for NL Interpretation and Assistance
RepairsRepository:
“If projector not working,
try a new bulb”
CALO Task Ontology:Catalog of Automated
Procedures
Execution and Monitoring
InstrumentedDesktop
TaskLearning
Set airport pickup for Joe
Announce room for talk
Reserve accommodations
Host a visitor
Arrange meeting
Identify Omitted Actions
Identify Related Sub-tasks
Toleratesyntactic variety
Interpret entries
10
Our Approach
Extended existing semantic parsing techniques to take advantage of broad-coverage knowledge repositories
Based on the standard Semantic Parsing approach [Acero; Zue; Allen et al]
Syntactic chunking + identification of categories of entities + semantic parsing of annotated result
Our problem is simpler in some ways Allows simplifying assumptions about structure of entries,
speech acts present Our problem is harder in other ways: actions may not be
fully specified, new actions may be automated (learned) To assist with user requests, need to identify implied actions &
sub-actions Leverage large knowledge repositories Leverages paraphrase collection for speech system [Chklovski, K-
CAP05, KCVC-05]
11
BEAM in Year 3: What Can and Cannot Be Interpreted
Statements are processed into their semantic components
Some entries cannot be interpreted because content is not recognized
Paraphrasing knowledge allows rewriting of entries so they can be interpreted
12
BEAM’s Stages of Mapping a User’s To Do Entry
1. Syntactic parsing
2. Identification of semantic
components,present & implied
Volunteercontributed
paraphrases (also task-subtask
pairs)
User’s To Do entry
4. Automatic entry rewriting
yes
no
yes
no
Mapping failed
Rewrites available?
Mapped entry
3. Ontological mapping of semantic
components
Mapped?
Teraword textual frequency repository
ontology
knowledge from volunteers and text extraction
13
BEAM’s Stages of Mapping a User’s To Do Entry
1. Syntactic parsing
5. Identification of sub-tasks
2. Identification of semantic
components,present & implied
Volunteercontributed
paraphrases (also task-subtask
pairs)
User’s To Do entry
ontology
knowledge from volunteers and text extraction
4. Automatic entry rewriting
yes
no
yes
no
Mapping failed
Rewrites available?
Mapped entry
3. Ontological mapping of semantic
components
Mapped?
Teraword textual frequency repository
14
Using BEAM with a To Do Manager: BEAM API
15
Recent Developments: Smarter BEAM Identifies Likely Implied Actions
To Do entry does not specify an action
BEAM looks in 1012 word corpus for mentions of “to * a meeting”, etc, identifying actions
These actions are then mapped to the specific target ontology
http://seagull.isi.edu/cgi-bin/todo-mgmt/api2?q=a%20meeting%20with%20Yolanda;gv=1
16
Leveraging the Teraword Ngrams Source
“quarterly meeting on Monday” “to * a meeting” “presents for John” “to * a present” 4876 buy 642 send
3482 get 619 bring … 593 open 1751 give 499 wrap 1352 find http://seagull.isi.edu/cgi-bin/ngram-extract/query-ngrams.pl
But what if there is little or no data? (Eg, “TGW meeting”) We’re exploring backoff strategies
But what if there is noise in the data? Stoplists (eg, “be”, “have”) can provide some relief Validation & feedback could also help
17
Example of Volunteer-based Validation Completed in 2006
Snapshot of validation: 1,113 harvested statements were put through context-
directed validation for “likely to be purchased in an office categories of times”
Most were filtered out; 107 (9.6%) passed
Determined to be likely to be purchased in an office:0.977 'office supplies' > planners 0.922 'office supplies' > equipment0.674 Business & Industrial > Food Service & Retail >
Bar & Beverage Equipment > CoffeeCategories determined to be unlikely to be purchased in an
office:0.017 Home & Garden > Pet Supplies > Cats > Cat Toys0.000 'building supplies' > 'concrete finishing'0.000 cards > 'racing-nascar'
18
Towards Identifying Subtasks:Start with Large Corpus
“a meeting with Peter” QUERY1: “the [Y] for the [X]” QUERY2: (filter) “need the [Y]” QUERY3: “to [Z] the [X] [Y]”
“approve meeting agenda”, “set/change/confirm meeting time”
Similarly: for “flight to SFO” Top suggestions include: “buy/purchase flight ticket”, “make flight arrangements”
19
Towards Identifying Subtasks:Start with Large Corpus
“a meeting with Peter” QUERY1: “the [Y] for the [X]” QUERY2: (filter) “need the [Y]” QUERY3: “to [Z] the [X] [Y]”
“approve meeting agenda”, “set/change/confirm meeting time”
Similarly: for “flight to SFO” Top suggestions include: “buy/purchase flight ticket”, “make flight arrangements”
Under Development – Stay Tuned
20
2006 Test System: Mappings to Task Ontology Categories
Early integration done to support a test question not otherwise addressed by CALO:
Any small contribution to test results considered a success No lead time for targeted collection for relevant task entries
PQ0166: What instances of task type (choose one) {|sc:%Communicate|, |sc:%Decide|, |sc:%Obtain|, |sc:%PlanAndSchedule|} are on user’s to-do list?
Examples handled by BEAM: arrange travel #PlanAndSchedule::Plan office supplies #Obtain::Buy hire a car #Obtain::Rent respond to Mary's request #Communicate::Answer
This is more powerful than straightforward application of, eg, WordNet synonyms
BEAM handles some situations where there is no synonymy or is-a relation between terms, eg. (arrange travel #PlanAndSchedule::Plan)
BEAM handles some situations where action is not even present(office supplies #Obtain::Buy)
21
Evaluation of Paraphrase Component
Despite early integration, BEAM contributed to evaluation
582 To Do statements collected for CALO Y3 evaluation 31.1% were mappable using the paraphrase repository 24.6% without paraphrase repository
Paraphrase repository was collected without focus on these items specifically
Contained 3,114 items, but (we estimate) only ~100 related to the domain covered by the test question
Additional knowledge sources and larger repositories will support further improvement of performance
Now also have data from public online To Dos, “tada list”
22
Conclusions
Developed BEAM, first system to demonstrate To Do list interpretation to enable automation
Integrated with working ToDo list manager, CALO’s Towel
Extended existing semantic parsing techniques to take advantage of preexisting large knowledge sources
Leveraged volunteers-created paraphrase corpus to improve ToDo entry interpretation
Identify likely implied actions Subtask suggestion in the works
23
Ongoing & Planned Work: New Capabilities
Improve interpretation and mapping capabilities:
Evaluate support of To Do entries which have no verb – ability to identify the implied actions
Proactively identify automatable subtasks for To Do lists
Acquire knowledge about relevant subtasks
Use BEAM’s semantic frames to provide information for task arguments, Towel forms (eg, “travel to Boston”)
Validate with/acquire from volunteers knowledge about sub-tasks, mappings
To Do entry made by user
BEAM mapping to TOWEL task