david l. chen and raymond j. mooney department of computer science the university of texas at austin...
TRANSCRIPT
![Page 1: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/1.jpg)
David L. Chen and Raymond J. MooneyDepartment of Computer ScienceThe University of Texas at Austin
Learning to Interpret Natural Language Navigation Instructions
from Observations
Twenty-Fifth Conference on Artificial Intelligence (AAAI-11)August 9, 2011
![Page 2: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/2.jpg)
Navigation Task
• Learn to interpret and follow free-form navigation instructions – e.g. Go down this hall and make a right when you see an
elevator to your left • Assume no prior linguistic knowledge• Learn by observing how humans follow instructions• Use virtual worlds and instructor/follower data
from MacMahon et al. (2006)
![Page 3: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/3.jpg)
3
H
C
L
S S
BC
H
E
L
E
Environment
H – Hat Rack
L – Lamp
E – Easel
S – Sofa
B – Barstool
C - Chair
![Page 4: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/4.jpg)
4
Environment
![Page 5: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/5.jpg)
5
Example Task
Task: Navigate from location 3 to location 4
3
H 4
![Page 6: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/6.jpg)
6
Example Task
• “Take your first left. Go all the way down until you hit a dead end.”
• “Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4.”
• “Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4.”
• “Walk forward once. Turn left. Walk forward twice.”
![Page 7: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/7.jpg)
7
Example Task
3
H 4
Observed primitive actions:
Forward, Left, Forward, Forward
Task: Navigate from location 3 to location 4
![Page 8: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/8.jpg)
Related Work
• Simpler worlds, no prior linguistic knowledge– Shimizu and Haas 2009– Matuszek et al. 2010
• More complex environments with prior linguistic knowledge– MacMahon et al. 2006– Vogel and Jurafsky 2010– Kollar et al. 2010
![Page 9: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/9.jpg)
Observation
Instruction
World State
Training
Action Trace
![Page 10: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/10.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Training
Action TraceNavigation Plan Constructor
![Page 11: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/11.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Training
Action TraceNavigation Plan Constructor
Semantic Parser Learner
![Page 12: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/12.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Training
Action TraceNavigation Plan Constructor
Semantic Parser Learner
Plan Refinement
![Page 13: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/13.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Instruction
World State
Training
Testing
Action TraceNavigation Plan Constructor
Semantic Parser Learner
Plan Refinement
![Page 14: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/14.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Instruction
World State
Training
Testing
Action TraceNavigation Plan Constructor
Semantic Parser Learner
Plan Refinement
Semantic Parser
![Page 15: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/15.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Execution Module (MARCO)
Instruction
World State
Training
Testing
Action TraceNavigation Plan Constructor
Semantic Parser Learner
Plan Refinement
Semantic Parser
Action Trace
![Page 16: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/16.jpg)
Learning system for parsing navigation instructions
Observation
Instruction
World State
Execution Module (MARCO)
Instruction
World State
Training
Testing
Action TraceNavigation Plan Constructor
Semantic Parser Learner
Plan Refinement
Semantic Parser
Action Trace
![Page 17: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/17.jpg)
Constructing Navigation Plans
Basic plan: Directly model the observed actions Travel Turn
steps: 1 LEFT
Instruction: Walk to the couch and turn leftAction Trace: Forward, Left
![Page 18: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/18.jpg)
Constructing Navigation Plans
Landmarks plan: Add interleaving verification steps
Basic plan: Directly model the observed actions
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1
at: SOFA LEFT
Travel Turn
steps: 1 LEFT
Instruction: Walk to the couch and turn leftAction Trace: Forward, Left
![Page 19: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/19.jpg)
Plan Refinement
• Remove extraneous details in the plans• First learn the meaning of words and short
phrases• Use the learned lexicon to remove parts of the
plans unrelated to the instructions
![Page 20: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/20.jpg)
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
Lexicon Learning
VerifyTravel Turn Verify
front: BRICK HALL
steps: 5 at: SOFA RIGHT front: CHAIR
Turn and walk to the couch
Walk to the couch and turn left
Walk to the couch and head down the brick hallway
1. Collect all plans g that co-occur with a word or short phrase w
![Page 21: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/21.jpg)
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
Lexicon Learning
VerifyTravel Turn Verify
front: BRICK HALL
steps: 5 at: SOFA RIGHT front: CHAIR
Possible meanings of walk to the couch:
1. Collect all plans g that co-occur with a word or short phrase w
![Page 22: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/22.jpg)
Lexicon Learning
VerifyTravel Turn Verify
front: BRICK HALL
steps: 5 at: SOFA RIGHT front: CHAIR
Possible meanings of walk to the couch:
2. Take intersections of all possible pairs of meanings
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
![Page 23: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/23.jpg)
Lexicon Learning
VerifyTravel Turn Verify
front: BRICK HALL
steps: 5 at: SOFA RIGHT front: CHAIR
Possible meanings of walk to the couch:
2. Take intersections of all possible pairs of meanings
Turn
LEFT
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
![Page 24: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/24.jpg)
Lexicon Learning
VerifyTravel Turn Verify
front: BRICK HALL
steps: 5 at: SOFA RIGHT front: CHAIR
Possible meanings of walk to the couch:
2. Take intersections of all possible pairs of meanings
Turn
LEFT
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
VerifyTravel
at: SOFA
![Page 25: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/25.jpg)
Lexicon Learning
VerifyTravel Turn Verify
front: BRICK HALL
steps: 5 at: SOFA RIGHT front: CHAIR
Possible meanings of walk to the couch:
2. Take intersections of all possible pairs of meanings
Turn
LEFT
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
VerifyTravel
at: SOFA
…
![Page 26: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/26.jpg)
Lexicon Learning
Possible meanings of walk to the couch:
3. Rank the entries by the scoring function
Turn
LEFT
Verify TravelTurn Verify
LEFT steps: 2 at: SOFAfront: SOFA
front: BLUEHALL
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1 at: SOFA LEFT
VerifyTravel
at: SOFA
…
![Page 27: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/27.jpg)
Refining Plans Using A Lexicon
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1
at: SOFA LEFT
Walk to the couch and turn left
![Page 28: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/28.jpg)
Refining Plans Using A Lexicon
Walk to the couch and turn left
Lexicon entry:
turn leftTurn
LEFT
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1
at: SOFA LEFT
![Page 29: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/29.jpg)
Refining Plans Using A Lexicon
Walk to the couch and
Lexicon entry:
walk to the couchVerifyTravel
at: SOFA
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1
at: SOFA LEFT
![Page 30: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/30.jpg)
Refining Plans Using A Lexicon
and
Lexicon exhausted
VerifyTravel Turn Verify
front: BLUEHALL
steps: 1
at: SOFA LEFT
![Page 31: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/31.jpg)
Refining Plans Using A Lexicon
and
Remove all unmarked nodes
VerifyTravel Turn
at: SOFA LEFT
![Page 32: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/32.jpg)
Refining Plans Using A Lexicon
Walk to the couch and turn left
VerifyTravel Turn
at: SOFA LEFT
![Page 33: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/33.jpg)
Experiments
Single Sentences Paragraphs
# Instructions 3236 706Vocabulary Size 629 660
Avg. # sentences 1.0 5.0
Avg. # words 7.8 37.6Avg. # actions 2.1 10.4
• Three different virtual worlds• Hand-segmented original data to single
sentences
![Page 34: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/34.jpg)
Plan Construction
• Test how well the system infers the correct navigation plans
• Gold-standard plans annotated manually• Use partial parse accuracy as metric– Credit for the correct action type (e.g. Turn)– Additional credit for correct arguments (e.g. LEFT)
• Lexicon learned and tested on the same data from two maps out of three
![Page 35: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/35.jpg)
Plan Construction
Precision Recall F1
Basic Plans 81.46 55.88 66.27
Landmarks Plans 45.42 85.46 59.29
Refined Landmarks Plans 78.54 78.10 78.32
![Page 36: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/36.jpg)
End-to-end Execution
• Test how well the system can perform the overall navigation task
• Leave-one-map-out approach• Strict metric: Only successful if the final
position matches exactly
![Page 37: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/37.jpg)
End-to-end Execution
• Lower baseline– A simple generative model based on the
frequency of actions alone• Upper baselines– Training with human annotated gold plans– Complete MARCO system (MacMahon, 2006)– Humans
![Page 38: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/38.jpg)
End-to-end Execution
Single Sentences ParagraphsSimple Generative Model 11.08% 2.15%Basic Plans 56.99% 13.99%Landmarks Plans 21.95% 2.66%Refined Landmarks Plans 54.40% 16.18%Human Annotated Plans 58.29% 26.15%MARCO 77.87% 55.69%Human Followers N/A 69.64%
![Page 39: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/39.jpg)
End-to-end Execution
Single Sentences ParagraphsSimple Generative Model 11.08% 2.15%Basic Plans 56.99% 13.99%Landmarks Plans 21.95% 2.66%Refined Landmarks Plans 54.40% 16.18%Human Annotated Plans 58.29% 26.15%MARCO 77.87% 55.69%Human Followers N/A 69.64%
![Page 40: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/40.jpg)
End-to-end Execution
Single Sentences ParagraphsSimple Generative Model 11.08% 2.15%Basic Plans 56.99% 13.99%Landmarks Plans 21.95% 2.66%Refined Landmarks Plans 54.40% 16.18%Human Annotated Plans 58.29% 26.15%MARCO 77.87% 55.69%Human Followers N/A 69.64%
![Page 41: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/41.jpg)
End-to-end Execution
Single Sentences ParagraphsSimple Generative Model 11.08% 2.15%Basic Plans 56.99% 13.99%Landmarks Plans 21.95% 2.66%Refined Landmarks Plans 54.40% 16.18%Human Annotated Plans 58.29% 26.15%MARCO 77.87% 55.69%Human Followers N/A 69.64%
![Page 42: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/42.jpg)
Example ParseInstruction: “Place your back against the wall of the ‘T’ intersection.
Turn left. Go forward along the pink-flowered carpet hall two segments to the intersection with the brick hall. This intersection contains a hatrack. Turn left. Go forward three segments to an intersection with a bare concrete hall, passing a lamp. This is Position 5.”
Parse: Turn ( ), Verify ( back: WALL ),Turn ( LEFT ),Travel ( ),Verify ( side: BRICK HALLWAY ),Turn ( LEFT ),Travel ( steps: 3 ),Verify ( side: CONCRETE HALLWAY )
![Page 43: David L. Chen and Raymond J. Mooney Department of Computer Science The University of Texas at Austin Learning to Interpret Natural Language Navigation](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f0d5503460f94c21436/html5/thumbnails/43.jpg)
Conclusion
• Presented an end-to-end system that learns to interpret free-form navigation instructions
• Learn by observing how humans follow instructions
• No prior linguistic knowledge• More details and data/code at:http://www.cs.utexas.edu/~ml/clamp/navigation/