speech & nlp (fall 2014): field study of narrative maps & automated extraction of...
Post on 29-Jun-2015
160 Views
Preview:
TRANSCRIPT
Speech & NLP
A Field Study of Narrative Maps &
Automated Extraction of Geo-Spatial Data
from Route Directions
www.vkedco.blogspot.com
Vladimir Kulyukin
Outline
A Field Study of Narrative Maps
Extraction of Geo-Spatial Data from Route
Descriptions: A Knowledge Representation Lab
A Field Study of Narrative Maps
Narrative Map
Narrative map consists of two verbal descriptions:
verbal route descriptions and verbal surveys of a given
environment
The origins of narrative maps vary from blogs,
forums, books, O&M instructors
Question: To what extent can narrative maps be
mined/parsed to extract various geo-spatial
knowledge (e.g., places, paths, SSH levels)?
Background
We investigated the utility of narrative maps in a
longitudinal study of blind shopping in supermarkets
In our system, called ShopTalk, verbal route
directions were generated from a manually
constructed topological map (inspired by the
topological level of the SSH) of the supermarket’s
locomotor space
ShopTalk 1.0
http://www.youtube.com/watch?v=A7UsazVCE7Y
Field Study Results
Ten visually impaired participants were able to
detect environmental cues needed to make sense of
the generated verbal instructions (provided at
beginning of experiment)
The participants used their O&M skills to localize and
orient themselves in the store, without any wearable
or environment-embedded sensors
Field Study Results
A key finding was that verbal route directions were
sufficient for our sample of independent travelers to
navigate this supermarket reliably
The more they used the system, the less they
requested verbal route directions
This finding suggests that the stores (and other
dynamic and complex environments) may not need to
be instrumented with any external sensors, such as
RFID tags, Wi-Fi routers, IR transmitters, etc.
Extraction of Geo-Spatial Data
from
Route Descriptions
Methodology
Take a route description
Go through it instruction by instruction
For each instruction, identify objects, predicates,
functions, relations (i.e., formalize a
conceptualization)
Run a syntactic parser (e.g., Stanford Parser) on
each instruction and come up with computable
techniques to automatically extract
conceptualizations
Sample Route Description
We will do both syntactic and semantic analysis
of a route description
We will use Stanford Parser as for syntactic and
dependency analysis
We will also attempt to write down a
conceptualization of the first 10 instructions from
the route description
Our conceptualization will be loosely based on
Kuipers’ SSH
Baruch College Route Description
1 Exit northbound bus 101, 102, or 103 at the 24th street stop and turn left.
2 Walk to 24th street in 80 feet.
3 This is a signalized, 2-lane crossing with 1-way Eastbound traffic only.
4 Cross 24th, and continue 225 feet to 25th street.
5 Cross 25th, a 2-lane signalized street with 1-way Westbound traffic.
6 Turn left and cross 3rd Ave.
7 This is a signalized, 6-lane crossing with 1-way Northbound traffic only.
8 Continue to the The Newman library, which is mid-block.
9 Follow the right side building edge.
10 This sidewalk is 15 feet wide, with left curbside obstacles such as poles, benches, trash, etc.
11 At 220 feet, a prominent right side landmark is the round metal revolving door of the library.
12 This is easily detected by a cane.
13 To the right of this door is the accessible entry door.
14 Enter here.
source http://www.clickandgomaps.com
Route Instruction 1: Parser Output
(ROOT
(S
(NP
(NP (NN Exit) (JJ northbound) (NN bus))
(SBAR
(S
(NP
(NP (CD 101) (, ,) (CD 102) (, ,)
(CC or)
(CD 103))
(PP (IN at)
(NP (DT the) (JJ 24th) (NN street))))
(VP (VB stop)
(CC and)
(VB turn)))))
(VP (VBD left))
(. .)))
nn(bus-3, Exit-1)
amod(bus-3, northbound-2)
dobj(stop-14, bus-3)
nsubj(left-17, bus-3)
nsubj(stop-14, 101-4)
nsubj(turn-16, 101-4)
num(101-4, 102-6)
conj_or(101-4, 103-9)
nsubj(stop-14, 103-9)
det(street-13, the-11)
amod(street-13, 24th-12)
prep_at(101-4, street-13)
rcmod(bus-3, stop-14)
rcmod(bus-3, turn-16)
conj_and(stop-14, turn-16)
root(ROOT-0, left-17)
Exit northbound bus 101, 102, or 103 at the 24th street stop and turn left.
Corrected Route Instruction 1: Parser Output
(ROOT
(S
(NP (PRP You))
(VP
(VP (VBZ exit)
(NP (JJ northbound) (NN bus))
(NP
(NP (CD 101) (, ,) (CD 102) (, ,)
(CC or)
(CD 103))
(PP (IN at)
(NP (DT the) (JJ 24th) (NN street) (NN
stop)))))
(CC and)
(VP (VBP turn)
(ADVP (RB left))))
(. .)))
nsubj(exit-2, You-1)
nsubj(turn-17, You-1)
root(ROOT-0, exit-2)
amod(bus-4, northbound-3)
iobj(exit-2, bus-4)
dobj(exit-2, 101-5)
num(101-5, 102-7)
dobj(exit-2, 103-10)
conj_or(101-5, 103-10)
det(stop-15, the-12)
amod(stop-15, 24th-13)
nn(stop-15, street-14)
prep_at(101-5, stop-15)
conj_and(exit-2, turn-17)
advmod(turn-17, left-18)
You exit northbound bus 101, 102, or 103 at the 24th street stop and turn
left.
Route Instruction 1: Conceptualization
Exit northbound bus 101, 102, or 103 at the 24th street stop and turn left.
Objects NorthBoundBus101, NorthBoundBus102, NorthBoundBus103, 24thStreet, 24thStreetStop, Agent
Predicates Place(NorthBoundBus101), Place(NorthBoundBus102), Place(NorthBoundBus103),
Place(24thStreetStop), Path(24thStreet), On(24thStreetStop, 24thStreet), AgentLocation(Agent,
NorthBoundBus101) V AgentLocation(Agent, NorthBoundBus102) V AgentLocation(Agent,
NorthBoundBus103)
Actions <AgentLocation(Agent, NorthBoundBus101), Travel, AgentLocation(Agent, 24thStreetStop>;
<AgentLocation(Agent, NorthBoundBus102), Travel, AgentLocation(Agent, 24thStreetStop>;
<AgentLocation(Agent, NorthBoundBus103), Travel, AgentLocation(Agent, 24thStreetStop>;
<AgentLocation(Agent, 24thStreetStop), Turn(Left), AgentLocation(Agent, 24thStreetStop>;
Route Instruction 2: Parser Output
(ROOT
(S
(VP (VBP Walk)
(PP (TO to)
(NP
(NP (JJ 24th) (NN street))
(PP (IN in)
(NP (CD 80) (NNS feet))))))
(. .)))
root(ROOT-0, Walk-1)
amod(street-4, 24th-3)
prep_to(Walk-1, street-4)
num(feet-7, 80-6)
prep_in(street-4, feet-7)
Walk to 24th street in 80 feet.
Route Instruction 2: Conceptualization
Walk to 24th street in 80 feet.
Objects 24thStreet
Predicates AgentLocation(Agent, 24thStreetStop)
Actions <AgentLocation(Agent, 24thStreetStop), Travel(80 feet), AgentLocation(Agent, 24thStreet>
Route Instruction 3: Parser Output
(ROOT
(S
(NP (DT This))
(VP (VBZ is)
(NP (DT a) (JJ signalized) (, ,) (JJ 2-lane) (NN crossing))
(PP (IN with)
(NP (JJ 1-way) (NNP Eastbound) (NN traffic)))
(ADVP (RB only)))
(. .)))
nsubj(crossing-7, This-1)
cop(crossing-7, is-2)
det(crossing-7, a-3)
amod(crossing-7, signalized-4)
amod(crossing-7, 2-lane-6)
root(ROOT-0, crossing-7)
amod(traffic-11, 1-way-9)
nn(traffic-11, Eastbound-10)
prep_with(crossing-7, traffic-11)
advmod(crossing-7, only-12)
This is a signalized, 2-lane crossing with 1-way Eastbound traffic only.
Route Instruction 3: Conceptualization
This is a signalized, 2-lane crossing with 1-way Eastbound traffic only.
Objects Crossing01
Predicates Crossing(Crossing01); SignalizedCrossing(Crossing01); NumberOfLanesCrossing01, 2);
DirectionOfTraffic(Crossing01, East); OneWayCrossing(Crossing01); Path(Crossing01);
CrossingOf(Crossing01, 24thStreet);
Actions No Actions
Route Instruction 4: Parser Output
(ROOT
(S
(NP (NNP Cross))
(VP
(VP (VBZ 24th))
(, ,)
(CC and)
(VP (VBP continue)
(NP (CD 225) (NNS feet))
(PP (TO to)
(NP (JJ 25th) (NN street)))))
(. .)))
nsubj(24th-2, Cross-1)
nsubj(continue-5, Cross-1)
root(ROOT-0, 24th-2)
conj_and(24th-2, continue-5)
num(feet-7, 225-6)
dobj(continue-5, feet-7)
amod(street-10, 25th-9)
prep_to(continue-5, street-10)
Cross 24th, and continue 225 feet to 25th street.
Route Instruction 4: Conceptualization
Cross 24th, and continue 225 feet to 25th street.
Objects 24thStreet, 25thStreet
Predicates Path(24thStreet); Path(25thStreet)
Actions <AgentLocation(Agent, Crossing01), Travel, AgentLocation(Agent, Crossing01)>;
<AgentLocation(Agent, Crossing01), Travel(225 feet), AgentLocation(Agent, 25thStreet)>;
Route Instruction 5: Parser Output
(ROOT
(NP
(NP (NNP Cross) (NNP 25th))
(, ,)
(NP
(NP (DT a) (JJ 2-lane) (JJ signalized) (NN street))
(PP (IN with)
(NP (JJ 1-way) (NNP Westbound) (NN traffic))))
(. .)))
nn(25th-2, Cross-1)
root(ROOT-0, 25th-2)
det(street-7, a-4)
amod(street-7, 2-lane-5)
amod(street-7, signalized-6)
appos(25th-2, street-7)
amod(traffic-11, 1-way-9)
nn(traffic-11, Westbound-10)
prep_with(street-7, traffic-11)
Cross 25th, a 2-lane signalized street with 1-way Westbound traffic.
Route Instruction 5: Conceptualization
Cross 25th, a 2-lane signalized street with 1-way Westbound traffic.
Objects 25thStreet
Predicates Path(25thStreet); SignalizedStreet(25thStreet); DirectionOfTraffic(West); OneWayStreet(25thStreet);
Crossing(Crossing02); CrossingOf(Crossing02, 25thStreet); NumberOfLanes(25thStreet, 2);
Actions <AgentLocation(Agent, Crossing02), Travel, AgentLocation(Agent, Crossing02)>;
Route Instruction 6: Parser Output
(ROOT
(S
(VP
(VP (VB Turn)
(ADVP (RB left)))
(CC and)
(VP (VB cross)
(NP (JJ 3rd) (NNS Ave.))))))
root(ROOT-0, Turn-1)
advmod(Turn-1, left-2)
conj_and(Turn-1, cross-4)
amod(Ave.-6, 3rd-5)
dobj(cross-4, Ave.-6)
Turn left and cross 3rd Ave.
Route Instruction 6: Conceptualization
Turn left and cross 3rd Ave.
Objects 3rdAve, Crossing03
Predicates Path(3rdAve); Crossing(Crossing03); CrossingOf(Crossing03, 3rdAve);
Actions <AgentLocation(Agent, Crossing02), Turn(Left), AgentLocation(Agent, Crossing02)>;
<AgentLocation(Agent, Crossing02), Travel, AgentLocation(Agent, Crossing03)>;
<AgentLocation(Agent, Crossing03>, Travel, AgentLocation(Crossing03)>;
Route Instruction 7: Parser Output
(ROOT
(S
(NP (DT This))
(VP (VBZ is)
(NP (DT a) (JJ signalized) (, ,) (JJ 6-lane) (NN crossing))
(PP (IN with)
(NP (JJ 1-way) (NNP Northbound) (NN traffic)))
(ADVP (RB only)))
(. .)))
nsubj(crossing-7, This-1)
cop(crossing-7, is-2)
det(crossing-7, a-3)
amod(crossing-7, signalized-4)
amod(crossing-7, 6-lane-6)
root(ROOT-0, crossing-7)
amod(traffic-11, 1-way-9)
nn(traffic-11, Northbound-10)
prep_with(crossing-7, traffic-11)
advmod(crossing-7, only-12)
This is a signalized, 6-lane crossing with 1-way Northbound traffic only.
Route Instruction 7: Conceptualization
This is a signalized, 6-lane crossing with 1-way Northbound traffic only.
Objects Crossing03
Predicates Crossing(Crossing03); NumberOfLanes(Crossing03, 6); SignalizedCrossing(Crossing03);
OneWayCrossing(Crossing03); DirectionOfTraffic(North);
Actions No actions
Route Instruction 8: Parser Output
(ROOT
(S
(VP (VB Continue)
(PP (TO to)
(NP
(NP (DT the) (NNP The) (NNP Newman) (NN library))
(, ,)
(SBAR
(WHNP (WDT which))
(S
(VP (VBZ is)
(ADJP (JJ mid-block))))))))
(. .)))
root(ROOT-0, Continue-1)
det(library-6, the-3)
nn(library-6, The-4)
nn(library-6, Newman-5)
prep_to(Continue-1, library-6)
nsubj(mid-block-10, library-6)
cop(mid-block-10, is-9)
rcmod(library-6, mid-block-10)
Continue to the The Newman library, which is mid-block.
Route Instruction 8: Conceptualization
Continue to the The Newman library, which is mid-block.
Objects TheNewmanLibrary
Predicates Place(TheNumanLibrary); Block(Block01); On(Block01, 3rdAve); MiddleOf(Block01,
TheNewmanLibrary);
Actions <AgentLocation(Agent, Crossing03), Travel, AgentLocation(Agent, TheNewmanLibrary)>
Route Instruction 9: Parser Output
(ROOT
(S
(VP (VB Follow)
(NP (DT the) (JJ right) (NN side) (NN building) (NN edge)))
(. .)))
root(ROOT-0, Follow-1)
det(edge-6, the-2)
amod(edge-6, right-3)
nn(edge-6, side-4)
nn(edge-6, building-5)
dobj(Follow-1, edge-6)
Follow the right side building edge.
Route Instruction 9: Conceptualization
Follow the right side building edge.
Objects TheRightSideBuildingEdge
Predicates Edge(TheRightSideBuildingEdge);
Actions <AwareOf(Agent, TheRIghtSideBuildingEdge), Follow, AwareOf(Agent, TheRightSideBuildingEdge)>
Route Instruction 10: Parser Output
(ROOT
(S
(NP (DT This) (NN sidewalk))
(VP (VBZ is)
(ADJP
(NP (CD 15) (NNS feet))
(JJ wide))
(, ,)
(PP (IN with)
(NP
(NP (JJ left) (NN curbside) (NNS obstacles))
(PP (JJ such) (IN as)
(NP (NNS poles) (, ,) (NNS benches) (, ,) (NNS trash) (, ,) (FW
etc.))))))
(. .)))
det(sidewalk-2, This-1)
nsubj(wide-6, sidewalk-2)
cop(wide-6, is-3)
num(feet-5, 15-4)
npadvmod(wide-6, feet-5)
root(ROOT-0, wide-6)
amod(obstacles-11, left-9)
nn(obstacles-11, curbside-10)
prep_with(wide-6, obstacles-11)
nn(trash-18, poles-14)
dep(trash-18, benches-16)
prep_such_as(obstacles-11, trash-18)
dep(trash-18, etc.-20)
Follow the right side building edge. This sidewalk is 15 feet wide, with left curbside obstacles such as poles, benches,
trash, etc.
Route Instruction 10: Conceptualization
This sidewalk is 15 feet wide, with left curbside obstacles such as poles, benches,
trash, etc.
Objects Sidewalk01
Predicates Path(Sidewalk01), WidthOf(Sidewalk01, 15 feet);
Actions <AgentLocation(Agent, LeftCurbOf(SideWalk01), Sense, AwareOf(Agent, X) & Pole(X) &
Obstacle(X)>;
<AgentLocation(Agent, LeftCurbOf(SideWalk01), Sense, AwareOf(Agent, X), & Bench(X) &
Obstacle(X)>;
<AgentLocation(Agent, LeftCurbOf(SideWalk01), Sense, AwareOfAgent, X) & Trash(X) &
Obstacle(X)>;
References
B. Kuipers. (2000). “The Spatial Semantic Hierarchy.” Artificial Intelligence 119, pp. 191-
233.
R. Schank, C. Riesbeck W. A. (1981) Inside Computer Understanding. Lawrence Erlbaum &
Associates.
J. Nicholson, V. Kulyukin, D. Coster. (2009). “ShopTalk: Independent Blind Shopping Through
Verbal Route Directions and Barcode Scans.” The Open Rehabilitation Journal, ISSN: 1874-
9437 Volume 2, 2009, DOI 10.2174/1874943700902010011.
Kulyukin, V. and Tammineni, T. Digital Labeling and Narrative Mapping in Mobile Remote
Audio Signage. Lap Lambert Academic Publishing, May 2014, ISBN-13: 978-3-659-52863-7,
ISBN-10: 3659528633, EAN: 9783659528637.
Kulyukin, V. and Reddy, T. Narrative Map Augmentation with Automated Landmark
Extraction and Path Inference. In K. Miesenberger et al. (Eds.): Proceedings of the 14th
International Conference on Computers Helping People with Special Needs (ICCHP 2014),
Part II, LNCS 8548, pp. 50-53, 2014. Springer International Publishing Switzerland 2014.
top related