weakly supervised learning of semantic parsers for mapping instructions … · 2017. 10. 17. ·...
TRANSCRIPT
-
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to ActionsYoav Artzi and Luke ZettlemoyerUniversity of Washington
TACL VOL. 1(2013):49−62 / NAACL 2013
1
-
Semantic Parsing
Less Supervision
More Domains
DatabasesLarge knowledge-basesInstructionsReferring expressionsRegular expressions
AnswersDemonstrationsSituated examples
Show me all papers about semantic parsing
�x.paper(x) ^ topic(x, SEMPAR)
Later thi
s
session
2
-
Semantic Parsing
Less SupervisionMore
Domains
Modeling
Situated Parsing
Show me all papers about semantic parsing
�x.paper(x) ^ topic(x, SEMPAR)
3
-
Executing Navigation Instructions
place your back against the wall of the t intersection
turn left
go forward along the pink flowered carpet hall two segments to the intersection with the brick hall
4
-
go to the next sofa
Resolve Referents
5
-
go to the next sofa
Resolve Referents
6
-
Disambiguate Word Sense
go to the chair7
-
Disambiguate Word Sense
go to the chair8
-
Identify Executable Actions
walk forward twice9
-
Identify Executable Actions
walk forward twice
move, move? I can do that!
10
-
Understand Implicit Requests
at the chair, turn right11
-
Understand Implicit Requests
at the chair, turn right
need to move first
12
-
Grounded Learning
move forward?turn?
turn to face the chair13
-
Grounded Learning
move forward?turn?
turn to face the chair14
-
Learning Signal
at the chair, move forward three steps past the sofa
Instruction:
Demonstration:
15
-
Learning Signal
Instruction:
Demonstration:
During learning: validate executions of different interpretations against demonstrations
at the chair, move forward three steps past the sofa
16
-
LearningDemonstrations
ParsingLogical Form
ExecutionJoint
InferenceWeighted
CCG
Situated Signals
Validation
17
-
LearningDemonstrations
Parsing1Logical Form2
Execution3
Lexical Generation4
Joint Inference
Weighted CCG
Situated Signals
Validation
18
-
Combinatory Categorial Grammarsmove to the chair
S AP/NP NP/N N
�a.move(a) �x.�a.to(a, x) �f.◆x.f(x) �x.chair(x)
>NP
◆x.chair(x)
>AP
�a.to(a, ◆x.chair(x))
S\S�f.�a.f(a) ^ to(a, ◆x.chair(x))
>S
�a.move(a) ^ to(a, ◆x.chair(x))
19
-
Combinatory Categorial Grammarsmove to the chair
S AP/NP NP/N N
�a.move(a) �x.�a.to(a, x) �f.◆x.f(x) �x.chair(x)
>NP
◆x.chair(x)
>AP
�a.to(a, ◆x.chair(x))
S\S�f.�a.f(a) ^ to(a, ◆x.chair(x))
>S
�a.move(a) ^ to(a, ◆x.chair(x))
Lexicon Combinators20
-
Combinatory Categorial Grammarsmove to the chair
S AP/NP NP/N N
�a.move(a) �x.�a.to(a, x) �f.◆x.f(x) �x.chair(x)
>NP
◆x.chair(x)
>AP
�a.to(a, ◆x.chair(x))
S\S�f.�a.f(a) ^ to(a, ◆x.chair(x))
>S
�a.move(a) ^ to(a, ◆x.chair(x))
Lexicon Combinators
x
z
y
21
-
Weighted Linear CCGs
• Given a weighted linear model:- CCG lexicon Λ- Feature function - Weights • The best parse is:
• We consider all possible parses y for sentence x given the lexicon Λ
y
⇤= argmax
yw · f(x, y)
f : X ⇥ Y ! Rm
w 2 Rm
22
-
LearningDemonstrations
Parsing✓Logical Form2
Execution3
Lexical Generation4
Joint Inference
Weighted CCG
Situated Signals
Validation
23
-
Constrain sets
Specific entities
Relations between entities
Sets of entities
Modeling for Semantic Parsing
Nouns
PPs and adjectives
Noun phrases
Verbs
24
-
Modeling for Semantic Parsing
Works well for natural language interfaces for DBs
Constrain sets
Specific entities
Relations between entities
Sets of entitiesNouns
PPs and adjectives
Noun phrases
Verbs
25
-
Modeling for Semantic Parsing
Previous work on instructional language adopted procedural representation
[Matuszek et al. 2010; 2012; Chen, Mooney 2011; Chen 2012; Kim, Mooney 2012]
Constrain sets
Specific entities
Relations between entities
Sets of entitiesNouns
PPs and adjectives
Noun phrases
Verbs
26
-
Modeling for Semantic Parsing
How can we use this approach for instructions?
Previous work on instructional language adopted procedural representation
Constrain sets
Specific entities
Relations between entities
Sets of entitiesNouns
PPs and adjectives
Noun phrases
Verbs
27
-
Relations between entitiesDavidsonian Events
Modeling for Semantic Parsing
Name objects
Instructions to execute
Constrain sets
Specific entities
Sets of entitiesNouns
PPs and adjectives
Noun phrases
Verbs
Sets of eventsImperatives
28
-
Spatial Environment Modeling1 2 3 4 5
1
2
3
4
5
• Maps are graphs of connected positions
• Agent can move forward, turn right and turn left
• Agent perceives clusters of positions
• Clusters capture objects
Agent
29
-
1 2 3 4 5
1
2
3
4
5
Spatial Environment Modeling
• Maps are graphs of connected positions
• Agent can move forward, turn right and turn left
• Agent perceives clusters of positions
• Clusters capture objects
30
-
Spatial Environment Modeling1 2 3 4 5
1
2
3
4
5
• Maps are graphs of connected positions
• Agent can move forward, turn right and turn left
• Agent perceives clusters of positions
• Clusters capture objects
31
-
Spatial Language1 2 3 4 5
1
2
3
4
5
• Nouns• Noun phrases• Adjectives• Prepositional phrases• Spatial relations
32
-
Spatial Language1 2 3 4 5
1
2
3
4
5
chair
�x.chair(x)
{ },
Nouns denote sets of objects
33
-
Spatial Language1 2 3 4 5
1
2
3
4
5
Noun phrases name specific entities
34
-
Spatial Language1 2 3 4 5
1
2
3
4
5
the chair
Noun phrases name specific entities
◆x.chair(x)
35
-
Spatial Language1 2 3 4 5
1
2
3
4
5
the chair
Noun phrases name specific entities
◆x.chair(x)
Definite determiner depends on agent state
36
-
Spatial Language1 2 3 4 5
1
2
3
4
5
the chair
Noun phrases name specific entities
◆x.chair(x)
Definite determiner depends on agent state
37
-
Spatial Language1 2 3 4 5
1
2
3
4
5
• Nouns• Noun phrases• Adjectives• Prepositional phrases• Spatial relations
38
-
Modeling Instructions
• Sequences of identical actions are events• Use Neo-Davidsonian event semantics• Represent imperatives as sets of events
39
-
Modeling Instructions1 2 3 4 5
1
2
3
4
5
move
Imperatives are sets of events
�a.move(a)
{ }, ,
40
-
Modeling Instructions1 2 3 4 5
1
2
3
4
5
move
{ },�a.move(a)
,Disambiguate by preferring
shorter sequences
Imperatives are sets of events
41
-
Modeling Instructions1 2 3 4 5
1
2
3
4
5
go to the chair
{ }
Events can be modified by adverbials
�a.move(a)^to(a, ◆x.chair(x))
42
-
Executing Logical Forms1 2 3 4 5
1
2
3
4
5
go to the chair�a.move(a) ^ to(a, ◆x.chair(x))
Consider all variable assignments to find
satisfying ones
43
-
Executing Logical Forms1 2 3 4 5
1
2
3
4
5
go to the chair�a.move(a) ^ to(a, ◆x.chair(x))
44
-
Executing Logical Forms1 2 3 4 5
1
2
3
4
5
go to the chair�a.move(a) ^ to(a, ◆x.chair(x))
45
-
Dynamic Models1 2 3 4 5
1
2
3
4
5
move until you reach the chair
�a.move(a)^post(a, intersect(◆x.chair(x), you))
World models change during execution
46
-
Dynamic Models1 2 3 4 5
1
2
3
4
5
move until you reach the chair
�a.move(a)^post(a, intersect(◆x.chair(x), you))
4
3
World models change during execution
47
-
Dynamic Models1 2 3 4 5
1
2
3
4
5
move until you reach the chair
�a.move(a)^post(a, intersect(◆x.chair(x), you))
4
3Never
inters
ects
World models change during execution
48
-
Dynamic Models1 2 3 4 5
1
2
3
4
5
move until you reach the chair
�a.move(a)^post(a, intersect(◆x.chair(x), you))
Update model to reflect state change
World models change during execution
49
-
Dynamic Models1 2 3 4 5
1
2
3
4
5
move until you reach the chair
�a.move(a)^post(a, intersect(◆x.chair(x), you))
2
3
Update model to reflect state change
Update
World models change during execution
50
-
Implicit Actions1 2 3 4 5
1
2
3
4
5
at the chair, turn left
Consider actions with prefixed implicit actions
�a.turn(a) ^ dir(a, left)^pre(a, intersect(◆x.chair(x), you))
51
-
Implicit Actions1 2 3 4 5
1
2
3
4
5
at the chair, turn left
�a.turn(a) ^ dir(a, left)^pre(a, intersect(◆x.chair(x), you))
Consider actions with prefixed implicit actions
52
-
Implicit Actions1 2 3 4 5
1
2
3
4
5
at the chair, turn left
�a.turn(a) ^ dir(a, left)^pre(a, intersect(◆x.chair(x), you))
Implicit actions
Consider actions with prefixed implicit actions
53
-
Joint Inference
Language Understanding
World State
Actions
54
-
• Execution is integrated into parsing• During parsing, execute logical forms and
observe the result
Demonstrations
Parsing✓Logical Form✓
Execution✓
Lexical Generation4
Joint Inference
Weighted CCG
Situated Signals
ValidationLearning
55
-
LearningDemonstrations
Parsing✓Logical Form✓
Execution✓
Lexical Generation4
Joint Inference
Weighted CCG
Situated Signals
Validation
56
-
Demonstrations
Parsing✓Logical Form✓
Execution✓
Lexical Generation4
Joint Inference
Weighted CCG
Situated Signals
Validation
Learning
• Based on a small set of seed templates
• New coarse-to-fine parsing algorithm to gradually prune the potential lexical entries
• Conservative approach to introduce new entries to the model
57
-
LearningDemonstrations
Parsing✓Logical Form✓
Execution✓
Lexical Generation✓
Joint Inference
Weighted CCG
Situated Signals
Validation
58
-
Validation-Driven Learning
• Online• 2 steps:- Lexical generation-Parameter update• Driven by a weak validation signal
59
-
Validation-Driven Learning
For T iterations, for each training sample:
• Step 1: Lexical generation- Generate a large number of potential lexical entries- Parse with the generated lexicon using the model- Select the best valid parses from the k-best parses- Add their lexical items to the lexicon• Step 2: Update parameters
60
-
Validation-Driven Learning
For T iterations, for each training sample:
• Step 1: Lexical generation• Step 2: Update parameters- Parse using the model- Split all parses into two sets:
max scoring valid and invalid
- Find margin violating pairs between the sets- Do a perceptron-style update using these violations
61
-
Related Work
Supervised seman.c parsing [Kate, Mooney 2006; Wong, Mooney 2007; Muresan 2011 ]
with CCGs [ZeElemoyer, Collins 2005; 2007; Kwiatkowski et al. 2010; 2011]
Weakly supervised seman.c parsing
[Clarke et al. 2010; Goldwasser, Roth 2011; Liang et al. 2011; Kirshnamurthy, Mitchell 2012; Goldwasser
et al. 2011]
Grounded Seman.c Analysis [Liang et al. 2009; Chen et al. 2010; Matuszek et al. 2012]
Execu.ng Instruc.ons with Shallow Representa.on
[Branavan et al. 2009; 2010; Vogel, Jurafsky 2010; Wei et al. 2009; Kollar et al. 2010; Tellex et al. 2011]
Non-‐joint Execu.on of Instruc.ons [Matuszek et al. 2010; 2012; Chen, Mooney 2011; Chen 2012; Kim, Mooney 2012]
62
-
Experimental Setup
• Seed lexicon from an annotated randomly selected 12 instruction sequences
• Features: lexical, type-raising usage and repetitions in logical coordinations
• Consider only executable parses as complete
63
-
ResultsSAIL Corpus - Cross Validation
0
14
28
42
56
70
Single Sentence Sequence
Chen and Mooney 2011Chen 2012Kim and Mooney 2012Final State ValidationTrace ValidationKim and Mooney 2013
64
-
ResultsSAIL Corpus - Cross Validation
0
14
28
42
56
70
Single Sentence Sequence
31.93
65.28
30.9
64.25
Chen and Mooney 2011Chen 2012Kim and Mooney 2012Final State ValidationTrace ValidationKim and Mooney 2013
65
-
ResultsSAIL Corpus - Cross Validation
0
14
28
42
56
70
Single Sentence Sequence
31.93
65.28
30.9
64.25
Chen and Mooney 2011Chen 2012Kim and Mooney 2012Final State ValidationTrace ValidationKim and Mooney 2013
66
-
ResultsOracle Corpus - Held-out Set
0
16
32
48
64
80
Single Sentence Sequence
58.05
78.63
54.63
77.6
Final State ValidationTrace Validation
67
-
Contributions
• Linguistically-driven modeling of instructional language
• Joint inference for interpretation and execution of grounded language
• General weakly-supervised learning approach for semantic parsers
68
-
UW SPF
Open source semantic parsing framework
http://yoavartzi.com/spf
Semantic Parser
Flexible High-Order Logic Representation
Learning Algorithms
69
-
UW SPF
Open source semantic parsing framework
http://yoavartzi.com/spf
Semantic Parser
Flexible High-Order Logic Representation
Learning Algorithms
Navigation code and data available online
http://yoavartzi.com/navi Comin
g up:
ACL tuto
rial
70
-
[fin]
71