Learning Spatial-Semantic Representations fromNatural Language Descriptions and Scene Classifications

Sachithra Hemachandra, Matthew R. Walter, Stefanie Tellex, and Seth Teller

Abstract— We describe a semantic mapping algorithm thatlearns human-centric environment models from by interpretingnatural language utterances. Underlying the approach is acoupled metric, topological, and semantic representation ofthe environment that enables the method to infer and fuseinformation from natural language descriptions with low-levelmetric and appearance data. We extend earlier work with anovel formulation incorporates spatial layout into a topologicalrepresentation of the environment. We also describe a factorgraph formulation of the semantic properties that encodeshuman-centric concepts such as type and colloquial name foreach mapped region. The algorithm infers these propertiesby combining the user’s natural language descriptions withimage- and laser-based scene classification. We also proposea mechanism to more effectively ground natural languagedescriptions of spatially non-local regions using semantic cuesfrom other modalities. We describe how the algorithm employsthis learned semantic information to propose valid topologicalhypotheses, leading to more accurate topological and metricmaps. We demonstrate that integrating language with othersensor data increases the accuracy of the achieved spatial-semantic representation of the environment.

I. INTRODUCTION

A challenge in realizing robots capable of working pro-ductively alongside human partners is the development ofefficient command and control mechanisms. Researchershave recently sought to endow robots with the ability tointeract more effectively with people through natural lan-guage speech [1, 2, 3, 4, 5] and gesture understanding [6].Efficient interaction can be facilitated when robots reasonover models that encode high-level semantic properties of theenvironment. For example, such models could help a micro-aerial vehicle interpret a first responder’s command to “flyup the stairway on the right, go down the hall, and observethe kitchen.”

Semantic mapping [7, 8, 9, 10] methods extend the metricenvironment models traditionally employed in robotics toinclude higher-level concepts, including types and colloquialnames for regions, and the presence and use of objects in theenvironment. Such methods typically operate by augmentinga standard SLAM metric map with a representation of theenvironment’s topology, and a distinct representation of itssemantic properties, the latter of which is populated byinterpreting the robot’s sensor stream, through (e.g.) scene

S. Hemachandra, M.R. Walter, and S. Teller are with the ComputerScience and Artificial Intelligence Laboratory at the Massachusetts Instituteof Technology, Cambridge, MA 02139 USA {sachih, mwalter,teller}@csail.mit.edu

S. Tellex is with the Computer Science Department at Brown University,Providence, RI 02912 USA stefie10@cs.brown.edu

Fig. 1. Maximum likelihood semantic map of the 6th floor of Stata building(pie charts denote the likelihood of different region categories).

classification. In this layered approach, the underlying met-ric map induces and embeds the topological and semanticattributes. However, while refinements to the metric mapimprove the topological and semantic maps, most techniquesdo not allow knowledge inferred at the semantic and topo-logical levels to influence one another or the metric map.Rather, semantic information has been inferred from LIDARand camera data, coupled with pre-trained scene appearancemodels. Some efforts are capable of integrating egocentricdescriptions about the robot’s current location (e.g., “weare in the kitchen”), but cannot handle allocentric spatiallanguage involving spatial relations between and labels forpotentially distant regions in the environment (e.g., “the exitis next to the cafeteria”).

We addressed some of these limitations in previouswork [11] with an algorithm that maintains a joint distribu-tion over a Semantic Graph, a coupled metric, topological,and semantic environment representation learned from userutterances and the robot’s low-level sensor data, during aguided tour. Our framework was able to to learn propertiesof the environment that could not be perceived with typicalsensors (e.g. colloquial names for regions, properties of areasoutside the robot’s sensor range) and use semantic knowledgeto influence the rest of the semantic graph, allowing robotsto efficiently learn environment models from users.

However, since that algorithm assumed that the environ-ment is a collection of regions that take the form of fixed,uniformly-sized collections of sequential nodes, it can resultin a topology that is inconsistent with human concepts ofspace. Consequently, the representation may not model thespatial extent to which the user’s descriptions refer, resultingin incorrect language groundings. The semantic informationin the framework was limited to user-provided colloquialnames and did not provide a means to reason over properties

such as region type that can be inferred from LIDARs,cameras, or other onboard sensors. Additionally, due to theabsence of other semantic information, the framework re-quired that a landmark location be labeled by the user beforethe utterance can be grounded (e.g., processing the phrase“the kitchen is down the hallway”, requires the “hallway” tohave already been labeled).

This paper describes an extension of our earlier approachto learn richer and more meaningful semantic models of theenvironment. Whereas our earlier framework reasoned onlyabout the connectivity of deterministically created regions(at fixed intervals), the current approach reasons over theenvironment’s region segmentation as well as its inter-regionconnectivity. Additionally, we propose a factor graph repre-sentation for the semantic model that reasons not only overeach region’s labels, but also its canonical type. As before,we infer region labels from user-provided descriptions, butwe also incorporate scene classification using the robot’sonboard sensors, notably camera and laser range-finders, toestimate region types. By modeling the relation betweenan area’s type and its colloquial name, the algorithm canreason over both region type and region label, even inthe absence of speech. This enables the method to moreeffectively ground allocentric user utterances (e.g., whengrounding the phrase “the kitchen is down the hallway”, weno longer require the user to explicitly label the “hallway”beforehand). We also describe a mechanism by which thealgorithm derives a semantically meaningful topology ofthe environment based upon the richer factor graph model,where edges are proposed using a spatial-semantic priordistribution. We show that the improved topology model thenallows the method to better handle ambiguities common innatural language descriptions.

II. RELATED WORK

A number of researchers have focused on the problemof constructing semantic representations [7, 10, 8, 9]. Mostapproaches have augmented lower-level metric maps withhigher-level topological and/or semantic information. How-ever, these typically follow a bottom-up approach in whichhigher-level concepts are constructed from lower-level infor-mation, without any information flow back down to lower-level representations. In Walter et al. [11] we addressed thisby introducing a framework that uses semantic informationderived from natural language descriptions uttered by hu-mans to improve the topological and metric representations.Our proposed approach uses additional semantic cues to eval-uate semantic similarity of regions to update the topology.

Several existing approaches [8, 10] have incorporatedhigher-level semantic concepts such as room type and pres-ence of objects with the use of appearance models. Pronobisand Jensfelt [10] describe a multi-modal probabilistic frame-work incorporating semantic information from a wide varietyof modalities including detected objects, place appearance,and human-provided information. However, their approach islimited to handling egocentric descriptions (e.g., “we are inthe living room”). Additionally, they infer topology based on

Fig. 2. Example of a semantic graph: Two regions R1 and R2 andtheir constituent nodes ni’s; distributions over node poses xi; and thecorresponding factor graph.

door detections, a heuristic which works well only in certainkinds of environments; they do not maintain a distributionover likely topologies. In [11], we maintained a hypothesisover the distribution of topologies, but the topology wasconstructed from segments created at fixed spatial intervals,which can be inconsistent with human concepts. In thepresent work, we address this limitation by proposing amethod that maintains multiple hypotheses about regionsegmentations and about connections among regions.

Mapping linguistic elements to their corresponding phys-ical entities have been studied by several researchers [3, 4,2, 12, 13, 14, 15, 16] in the robotics domain. However,these efforts have not focused on constructing semanticrepresentations. In [11] we augmented our semantic graphwith complex language descriptions. However, this could beaccomplished only when the user had explicitly labeled alocation before describing another in reference to it (e.g.,“the gym is down the hallway” requires prior labeling ofthe hallway). The resulting representation contained onlylabels obtained through language descriptions; it did notintegrate semantic cues from other sources (such as ap-pearance models). The present method improves upon thisby integrating information from multiple semantic sourcesand maintaining a distribution over a larger set of semanticproperties, rendering it capable of grounding language evenin the absence of pre-labeled landmark locations.

III. SEMANTIC GRAPH REPRESENTATION

This section presents our approach to maintaining a distri-bution over semantic graphs, an environment representationthat consists jointly of metric, topological, and semanticmaps.

A. Semantic Graphs

We define the semantic graph as a tuple containing topo-logical, metric and semantic representations of the environ-ment. Figure 2 shows an example semantic graph for a trivialenvironment.

The topology Gt is composed of nodes ni that denote therobot’s trajectory through the environment (sampled at 1 m

distances), node connectivity, and node region assignments.We associate with each node a set of observations thatinclude laser scans zi, semantic appearance observationsai based on laser li and camera ii models, and availablelanguage observations λi. We assign nodes to regions Rα ={n1, .., nm} that represent spatially coherent areas in theenvironment (such as rooms or hallways) compatible withhuman concepts. Undirected edges exist between nodesin this graph, denoting traversability between two nodes.Edges (denoting connectivity) between regions are inferredbased on the edges between nodes in the graph. A regionedge exists between two regions if at least one graph edgeconnects a node from one region to a node in the other. Thetopological layer consists of the nodes, edges and the regionassignments for the nodes.

The pose of each node ni is represented by xi in aglobal reference frame. The metric layer is conditioned onthe topology, where edges in the topology also includemetric constraints between the poses of the correspondingnodes. Metric constraints are calculated by scan-matchingthe corresponding laser observations of each region. A posegraph representation is employed to maintain the distributionover the pose of each node, conditioned on these constraints.Occupancy maps can be constructed based on the distributionof these node poses and their corresponding laser observa-tions.

Semantic information is also conditioned on the topologyas shown in the Fig. 2. The semantic layer consists of afactor graph, where variables represent properties in eachregion (i.e., the type Cr and labels Λr), properties thatcan be observed at each node (in each region), and factorsthat denote the joint likelihood of these variables (e.g., thelikelihood of observing a label given a particular room type).Observations of these region properties are made using laser-and image-based scene classifiers and by grounding humandescriptions of the environment.

B. Distribution Over Semantic Graphs

We estimate a joint distribution over the topology Gt(which includes the nodes, their connectivity, and theirregion assignments), the vector of locations Xt, and theset of semantic properties St. Formally, we maintain thisdistribution over semantic graphs {Gt, Xt, St} at time tconditioned upon the history of metric exteroceptive sensordata zt = {z1, z2, . . . , zt}, odometry ut = {u1, u2, . . . , ut},scene appearance observations at = {a1, a2, . . . , at} (wherein our implementation at = {lt, it}), and natural languagedescriptions λt = {λ1, λ2, . . . , λt}:

p(Gt, Xt, St|zt, ut, at, λt). (1)

Each variable λi denotes a (possibly null) utterance, suchas “This is the kitchen,” or “The gym is down the hall.” Wefactor the joint posterior into a distribution over the graphsand a conditional distribution over the node poses and labels:

p(Gt, Xt, St|zt, at, ut, λt) = p(St|Xt, Gt, zt, at, ut, λt)

× p(Xt|Gt, zt, at, ut, λt)× p(Gt|zt, at, ut, λt) (2)

Algorithm 1: Semantic Mapping Algorithm

Input: Pt−1 ={P

(i)t−1

}, and (ut, zt, at, λt), where

P(i)t−1 =

{G

(i)t−1, X

(i)t−1, S

(i)t−1, w

(i)t−1

}Output: Pt =

{P

(i)t

}for i = 1 to n do

1) Employ proposal distribution to propagate thegraph sample based on ut, λt and at.

a) Sample region allocationb) Sample region edgesc) Merge newly connected regions

2) Update the Gaussian distribution over the nodeposes X(i)

t conditioned on topology.3) Update the factor graph representing semantic

properties for the topology based on appearanceobservations (lt and it) and language λt.

4) Compute the new particle weight w(i)t based

upon the previous weight w(i)t−1 and the metric

data zt.end

Normalize weights and resample if needed.

As in Walter et al. [11], we maintain this factored distribu-tion using a Rao-Blackwellized particle filter, mitigating thehyper-exponential hypothesis space of the topology [11].

We represent the joint distribution over the topology, nodelocations, and labels as a set of particles:

Pt = {P (1)t , P

(2)t , . . . , P

(n)t }. (3)

Each particle P (i)t ∈ Pt consists of the set

P(i)t =

{G

(i)t , X

(i)t , S

(i)t , w

(i)t

}, (4)

where G(i)t denotes a sample from the space of graphs;

X(i)t is the analytic distribution over locations; S(i)

t is thedistribution over semantic properties; and w(i)

t is the weightof particle i.

IV. SEMANTIC MAPPING ALGORITHM

Algorithm 1 outlines the process by which the method re-cursively updates the distribution over semantic graphs (2) toreflect the latest robot motion, metric sensor data, laser- andimage-based scene classifications, and the natural languageutterances. The following sections explain each step in detail.

A. The Proposal Distribution

We compute the prior distribution over the semantic graphGt, given the posterior from the last time step Gt−1, bysampling from a proposal distribution. This proposal distri-bution is the predictive prior of the current graph given theprevious graph, sensor data (excluding the current time step),appearance data, odometry, and language:

p(Gt|Gt−1, zt−1, at, ut, λt) (5)

We augment Gt−1 to reflect the robot’s motion by addinga node nt to to the topology and an edge to the previousnode nt−1, resulting in an intermediate graph G−t . Thisrepresents the robot’s current pose and the connectivity toits previous pose. This yields an updated vector of posesX−t and semantic properties S−t . The new node is assignedto the current region.

1) Creation of New Regions: We then probabilisticallybisect the current region Rc using the spectral clusteringmethod outlined in Blanco et al. [17]. We construct thesimilarity matrix using the laser point overlap between eachpair of nodes in the region. Equation 6 defines the likelihoodof bisecting the region, which is based on the normalized cutvalue (Nc) of the graph involving the proposed segments.The likelihood of accepting a proposed segmentation risesas the Nc value decreases, i.e. as the separation of the twosegments improves (minimizing the inter-region similarity):

P (s/Ncut) =1

(1 + αN3c ).

(6)

This results in more spatially distinct areas in the worldhaving a higher likelihood of being separate regions (i.e.,more particles will model these areas as separate regions).If a particle segments the current region, a new region Ri iscreated which does not include the newly added node.

2) Edge Proposals: When a new region Ri is created,the algorithm proposes edges between this region, and otherregions in the topology (excluding the current region Rc).

Fig. 3. Example of region edges being proposed (black lines representsrejected edge proposals; the red line represents an accepted edge)

The algorithm samples inter-region edges from a spatial-semantic proposal distribution that incorporates the semanticsimilarity of regions, as well as the spatial distributionsof its constituent nodes. This reflects the facts that closeregions and semantically similar regions are more likely to beconnected. We measure semantic similarity based upon thelabel distribution associated with each region. The resultinglikelihood has the form:

pa(Gt|G−t , zt−1, ut, at, λt) =∏

j:eij /∈E−

p(Gijt |G−t ) (7a)

∝∏

j:eij /∈E−

px(Gijt |G−t )ps(Gijt |G−t ) (7b)

where we have omitted the history of language observationsλt, metric measurements zt−1, appearance measurementsat, and odometry ut for brevity. Equation (7a) reflectsthe assumption that additional edges expressing constraints

involving the current node eij /∈ E− are conditionallyindependent. While px(Gijt |G−t ) encodes the likelihood ofthe edge based on the spatial properties of the two re-gions, ps(G

ijt |G−t ) describes the edge likelihood based on

the regions’ semantic similarity. Equation (7b) reflects theconditional independence assumption between the spatial andthe semantic likelihood of each edge.

For the spatial distribution prior, we define a variable dij tobe the distance between the mean nodes of the two regions,where the mean node is the node with its pose closest to theregion’s average pose:

px(Gijt |G−t ) =

∫X−

t

p(Gijt |X−t , G−t , ut)p(X−t |G−t ) (8a)

=

∫dij

p(Gijt |dij , G−t )p(dij |G−t ). (8b)

The conditional distribution p(Gijt |dij , Gt−1, zt−1, ut) ex-

presses the likelihood of adding an edge between regionsRt and Rj based upon the location of their mean nodes.We represent the distribution for a particular edge betweenregions Ri and Rj a distance dij = |X̄Ri − X̄Rj |2 apart as:

p(Gijt |dij , G−t , zt−1, ut) ∝ 1

1 + γd2ij

, (9)

where γ specifies distance bias. For the evaluations in thispaper, we use γ = 0.3. We approximate the distance priorp(dij |G−t , zt−1, ut) with a folded Gaussian distribution.

For the semantic prior, regions with similar label Λ dis-tributions will have a higher likelihood of having an edgebetween them. The label distributions for the regions arepart of the semantic properties (of each region) modeled inthe semantic layer:

ps(Gijt |G−t ) =

∑S−t

p(Gijt |S−t , G−t )p(S−t |G−t ) (10a)

=∑

Λ−i ,Λ

−j

p(Gijt |Λ−i ,Λ−j , G

−t )p(Λ−i ,Λ

−j |G

−t ).

(10b)

Equation (11) expresses the likelihood of an edge existingbetween two regions, given the value of the regions’ respec-tive label values:

p(Gijt |Λi,Λj) =

{θΛi

if Λi = Λj

0 if Λi 6= Λj, (11)

where θΛidenotes the label-dependent likelihood that edges

exist between nodes with the same label. In practice, we as-sume a uniform saliency prior for each label. Equation (10b)then measures the cosine similarity between the label distri-butions.

After a region edge is sampled from the spatial-semanticprior, a scan-match procedure attempts to find the bestalignment between the two regions. Upon convergence ofthe scan-match routine, the edge is accepted and is used toupdate the topology.

3) Region Merges: After a new region Ri has beencreated and edges to other regions have been checked andadded, the algorithm determines whether it is possible tomerge with each region to which it is connected. The newly-created region is merged with an existing (connected) regionif the observations associated with the smaller of the tworegions can be adequately explained by the larger region.This results in regions being merged when the robot revisitsalready explored spaces (i.e., already-represented regions).This merge process is designed to ensure that the complexityof the topology increases only when the robot explores newareas, leading to more efficient region edge proposals as wellas more compact language groundings.

B. Updating the Metric Map Based on New Edges

The algorithm then updates the spatial distribution of thenode poses Xt conditioned on the proposed topology,

p(Xt|Gt, zt, ut, λt) = N−1(Xt; Σ−1t , ηt), (12)

where Σ−1t and ηt are the information matrix and infor-

mation vector that parametrize the canonical form of theGaussian. We use the iSAM algorithm [18], which iterativelysolves for the QR factorization of the information matrix.

C. Updating the Semantic Layer

Compared with Walter et al. [11], our updated repre-sentation maintains a distribution about a larger set ofsemantic properties associated with the environment. Thedistribution over the semantic layer is maintained using afactor graph [19] that is conditioned on the topology foreach particle.

Fig. 4. Semantic Layer (plate representation)

As Fig. 4 shows, the semantic layer maintains two vari-ables associated with each region r, namely the regioncategory Cr (e.g., hallway, conference room, etc.) and thelabels associated with the region Λr. For example, a confer-ence room can have multiple labels, such as meeting room,conference room. The factor that joins these two variablesrepresents the likelihood of each room category generatinga particular label.

At each node n within a region, the robot can observeone or more of these semantic properties. In our currentimplementation, these are the region appearance observedfrom laser scanners ln or cameras in, and the region labelsλk (and the associated correspondence variables Φk). We runbelief propagation for each particle at each time step as newvariables and factors are added. We subsequently update thecategory and label distributions for each region.

The following subsections outline how we integrate ob-servations of these node properties to the semantic layer.

1) Integrating Semantic Classification Results: Each nodehas an appearance variable an which is related to its re-gion category. We outline several general appearance classes(“room”, “hallway” and “open area”) that are then observedusing robot sensors. The factor that connects a region cate-gory variable Cr to an appearance variable an encodes thelikelihood of a region category generating an appearanceclass (e.g., how often does a conference room appear as aroom). The category to appearance factor was trained usingannotated data from several other floors of the Stata building.

We make two observations of an using the laser andcamera observations at node n. These are represented inthe factor graph as the laser appearance ln and the imageappearance in. We use two pre-trained appearance models forlaser observations and camera images. The laser appearanceclassification model has been trained using laser featuressimilar to those outlined in Mozos et al. [20], while the imageappearance model has been trained using CRFH [21]. Laserand camera appearance variables ln and in are connected tothe node’s appearance an using factors built from the con-fusion matrix for the two trained models. The classificationresults for the two sensors provide a distribution representingthe likelihood of the observations being generated from eachappearance category. The classifier outputs are integrated tothe factor graph as factors attached to variables ln and in.

2) Integrating Language Observations: The robot canalso receive either ego-centric (e.g., “I am at the kitchen”) orallocentric (e.g., “The kitchen is down the hall”) descriptionsof the environment from the user. We use these observationsto update the likelihood of observing labels in each region.We maintain the label for each region as another variable(Λr) in our factor graph. The region label is related to theregion category, as each category is more likely to generatesome labels than others. For example, while a person mightdescribe a “cafeteria” as a “dinning hall”, it is unlikely thathe will describe it as an “office”. For our current experimentswe have identified a limited subset of labels associatedwith each region category (e.g., the hallway category cangenerate “hall”, “hallway”, “corridor”, or “walkway”) in ourrepresentation. When building these factors (between labelsand room categories), for each category we assign higherlikelihoods for its associated labels and smaller likelihoodsfor the other labels (capturing the likelihood of generatingthese labels given a particular room category).

A node can have none, one, or multiple label observations,depending on the way the person describes a location. Werepresent each label observation with a variable λk and acorrespondence variable Φk, which denotes the likelihoodthat the label was intended to describe the current location.The correspondence variable Φ is a binary-valued variablespecifying whether or not the label describes the region. Ifthe label doesn’t correspond to that region (i.e., if Φk =0), the observation λk is uninformative about the region’slabel, and will have equal likelihood for each label value.However, when the correspondence holds (Φ = 1), the factor

Fig. 5. Maximum likelihood semantic map of a multi-building environment on the MIT campus.

encodes the likely co-occurrences between different labels.For example, if the robot heard the label “conference room”,with a high likelihood of Φ = 1, it will result in other likelylabels (that often co-occur with “conference room”) havinghigh likelihoods (e.g., “meeting room”) as well. Currentlyhigh co-occurrence is added for words that are synonyms(e.g., “hallway” and “corridor”). In this way, we use thecorrespondence variable to handle the ambiguity inherentin grounding natural language descriptions. When a label isgrounded to a region, we create a label observation λk andcorrespondence variable Φk, and connect it to the associatedregion’s label variable Λr using a co-occurrence factor. Weintegrate the correspondence observation Φ by attaching afactor encoding this likelihood. We treat the observed labelas having no uncertainty, as our current model does notincorporate uncertainty arising from speech recognition.

We derive the factor denoting the observation of Φ basedon the type of description given by the user. If the userdescribes the current location (e.g., “I am at the livingroom”), we have higher correspondence with spatially localnodes. For such descriptions, we allocate a high likelihood(0.8 probability of Φ = 1) of correspondence with the currentregion. For allocentric descriptions (e.g., “the kitchen is downthe hallway”, where the user describes the location of thereferent “kitchen” with relation to the landmark “hallway”),we use the G3 framework [4] to calculate the correspondencelikelihood given the potential landmarks. We marginalize thelandmarks to arrive at correspondence likelihood of eachreferent region in a manner similar to our previous approach.

In handling allocentric descriptions the current method im-proves upon our earlier approach in two ways. Firstly, we nolonger require common landmarks to be explicitly describedbefore being able to ground the language correctly. We dothis by leveraging the richer semantic representation madepossible by integrating additional semantic information toarrive at likely landmarks. Secondly, while some expressionscan be ambiguous (e.g., there could be multiple regions downthe hall), the presence of other semantic cues allows thefinal label distribution to be more accurate because incorrectgroundings will have less impact if the region’s appearanceis different from the label observation.

D. Updating the Particle Weights and Resampling

Particle weights are updated and resampling carried out inthe same manner as Walter et al. [11].

V. RESULTS

We evaluate our algorithm through four experiments inwhich a human gives a robotic wheelchair [9] a narratedguided tour of the Stata building (S3, S4, S6) as well asa multi-building indoor tour (MF). The robot was equippedwith a forward-facing LIDAR, a camera, wheel encoders, anda microphone. In these experiments we drove the robot usinga joystick, and provided it with natural language descriptionsat specific salient locations by typing.

We evaluate the resulting semantic maps with regards totheir topological accuracy, compactness, segmentation accu-racy and semantic accuracy. All experiments were run with10 particles. The results show that our framework producesmore compact and more accurate semantic graphs than ourprevious approach. They also demonstrate the improvementin semantic accuracy due to language descriptions. We alsoshow the ability of our framework to ground complex lan-guage even in the absence of previous labels for the referent(e.g. it handles the expression “the lobby is down the hall”even when the hall has not been labeled).

A. Topological accuracy

We compare the topological accuracy, conditioned uponthe resulting segmentation, by comparing the maximumlikelihood map with the ground truth topology. We definea topology as matching ground truth if node pairs that arespatially close (1 m) in a metrically accurate alignment are atmost one region hop away. This avoids penalizing occasionalregions that do not contain valid edges as long as a nearbyregion was accurately connected (and possibly merged withthe nodes from a prior visit). This can happen when an edgewas not sampled or when scan-matching failed to converge.

The percentage of close node pairs that were more thanone region hop away from each other for the third, fourth andsixth floor were 2.8%, 3.7% and 3.8%, respectively. Mostregion-matching errors occurred in areas with significantclutter, causing scan-matching failures. Metric maps derivedfrom the maximum likelihood particles were accurate for allthree floors.

Fig. 6. Maximum likelihood semantic map of the 3rd floor of the Statabuilding (pie charts denote the likelihood of different region categories).

TABLE IREGION ALLOCATION EFFICIENCY (Sc)

Floor Proposed OldFramework Framework

Stata Floor 3 (S3) .67 .29Stata Floor 4 (S4) .77 .37Stata Floor 6 (S6) .69 .52

B. Topological Compactness

We compare the allocation of nodes to regions in thecurrent framework to the previous method. In the previousapproach, the topology update did not merge regions evenwhen the robot revisited a region; it simply created an edgebetween the regions. The redundancy of the regions has sev-eral negative implications. Firstly, it unnecessarily increasesthe hypothesis space of possible region edges, reducingthe likelihood of a sample proposing valid region edges.Secondly, it increases the hypothesis space for groundinglanguage, forcing the framework to consider more regionpairs as possible groundings for user descriptions.

We measure the duplicity of the region allocation as

Sc = Ns/Nt, (13)

where Ns is the number of close node pairs (< 1m) assignedto the same region and Nt is the total number of close nodepairs. If the topology is efficient at allocating regions, thisratio should be high, as only nodes near region boundariesshould belong to different regions. Table I shows these scoresevaluated for each approach, on three floors. The new methodscores significantly higher in all three experiments. Thedifference of this score becomes more pronounced as therobot revisits more regions. Since the sixth floor dataset didnot have too many revisited regions, the scores for the twoapproaches are closer.

C. Segmentation Accuracy

Table II outlines the segmentation accuracy (of the maxi-mum likelihood particle) for two datasets, outlined accordingto region type. We picked the best matches based on theJaccard index (number of intersecting nodes divided by thenumber of union nodes) for each ground truth annotatedregion and the resulting segmented region. Since our segmen-tation method depends on the similarity of laser observations,large cluttered region types such as lab spaces and lounges

(a) Appearance only (b) With language

Fig. 7. Region category distribution (a) for a region with only appearanceinformation and (b) and with both appearance and language “the lounge isbehind us”. (category “lounge”: yellow).

TABLE IIREGION SEGMENTATION AND SEMANTIC ACCURACY

Region Segmentation Semantic AccuracyType Accuracy Without Lang With Lang

S3 MF S3 MF S3 MFConference room 80.0 81.7 8.8 15.1 48.5 58.7Elevator lobby 59.7 72.8 18.8 12.8 64.1 46.4Hallway 49.4 55.7 44.5 58.5 44.4 58.0Lab 52.8 30.1 11.8 27.2 14.2 30.6Lounge 42.9 39.4 28.6 36.6 62.0 40.5Office 62.5 76.1 78.1 45.6 98.6 60.2

tend to be over-segmented. Additionally long hallways tendto be over-segmented by our method, which is reflected inthe lower scores for hallways.

D. Inference of Semantic Properties

Table II also outlines the semantic accuracy (of themaximum likelihood particle) for two datasets. Semanticaccuracy was calculated for each ground truth region byassigning each constituent node with its parent region’scategory distribution and taking the cosine similarity. Weobserve that the semantic accuracy with language is higherfor most region types (hallways were rarely described andas such show minimal improvement). Some regions suchas labs, which were labeled with egocentric descriptions,have low scores because the regions are over segmentedand the language is attributed only to the current region.Figure 7 compares the region category properties with andwithout language. In the absence of language (Fig. 7(a)),the appearance of the region gives equal likelihood for both“elevator lobby” and “lounge”. In Fig. 7(b), the region wasgrounded with the label “lounge” and the framework inferreda higher likelihood of the region category being a lounge.

E. Grounding Allocentric Language Descriptions

We also tested our framework with allocentric languagedescriptions. When handling phrases that include a landmarkand a referent (e.g., “the gym is down the hall”), ourearlier framework required the landmark to have already beenlabeled before describing the referent location. With our newframework, the robot is able to ground language when thelandmark corresponds to locations that may not have beenlabeled, but can be inferred from other semantic cues (e.g.,via appearance classification). We tested this situation usingseveral instances in our dataset.

Figure 8 shows instances in which allocentric languageutterances were grounded into the semantic graph. As thelabel distributions for the surrounding regions demonstrate,

(a) (b)

Fig. 8. Resulting category distribution for complex language phrases (a)“the elevator lobby is down the hall” and (b) “the lounge is down the hall”.

the framework is able to ground the referent with a highdegree of accuracy, even though the landmark was neverexplicitly labeled. However, since there is more uncertaintyabout the landmark region, the information derived fromthe allocentric language has less influence on the semanticproperties on the region (since we marginalize the landmarklikelihood when calculating the grounding likelihood Φ).

VI. CONCLUSION

We described a framework enabling robots to constructspatial-semantic representations of toured environments fromnatural language descriptions and scene classifications. Com-pared to earlier methods, our approach handles complexnatural language descriptions more efficiently, and producessemantic representations that are richer and more compact.

Currently, our method grounds descriptive language underthe assumption that it refers to concepts that exist in therobot’s representation, either due to previously interpreteddescriptions or to appearance classification. We are exploringa mechanism that reasons about the presence of the referredentities and grounds the utterance only when it is confidentabout the validity of the grounding. We also plan to inte-grate additional semantic cues that inform region labels andcategories, such as the presence of common objects foundindoors. In our current implementation, the factors betweenregion category and labels were constructed using a set ofpredefined labels and relationships identified by us (usingsynonyms for region types). In the future we plan to learnthese relationships using real-world datasets.

Our approach maintains multiple hypotheses regarding theregion assignments of the topology by using particles tosample region assignments based on a prior that considersthe spatial similarity of nodes. However, particle weightsare calculated based only on laser data. This can lead tovalid segmentation hypotheses being discarded during re-sampling if the laser observation likelihood is low. We planto incorporate region hypothesis scores based on appearance.In the current approach, the robot passively accepts humanassertions about the environment. We also plan to exploremore interactive scenarios where the robot reasons over itshypothesis space and asks questions of the human to resolveambiguities.

VII. ACKNOWLEDGMENTS

We would like to thank S. Shemet and F. Paerhati fortheir help collecting and annotating datasets as well as intraining the appearance models. This work was supported in

part by the Robotics Consortium of the U.S Army ResearchLaboratory under the Collaborative Technology AllianceProgram, Cooperative Agreement W911NF-10-2-0016.

REFERENCES[1] G. Bugmann, E. Klein, S. Lauria, and T. Kyriacou, “Corpus-based

robotics: A route instruction example,” Proc. Intelligent AutonomousSystems, pp. 96–103, 2004.

[2] J. Dzifcak, M. Scheutz, C. Baral, and P. Schermerhorn, “What to doand how to do it: Translating natural language directives into temporaland dynamic logic representation for goal management and actionexecution,” in Proc. IEEE Int’l Conf. on Robotics and Automation(ICRA), 2009, pp. 4163–4168.

[3] C. Matuszek, D. Fox, and K. Koscher, “Following directions usingstatistical machine translation,” in Proc. ACM/IEEE Int’l. Conf. onHuman-Robot Interaction (HRI), 2010, pp. 251–258.

[4] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee,S. Teller, and N. Roy, “Understanding natural language commandsfor robotic navigation and mobile manipulation,” in Proc. Nat’l Conf.on Artificial Intelligence (AAAI), 2011, pp. 1507–1514.

[5] D. L. Chen and R. J. Mooney, “Learning to interpret natural languagenavigation instructions from observations,” in Proc. Nat’l Conf. onArtificial Intelligence (AAAI), 2011, pp. 859–865.

[6] K. Nickel and R. Stiefelhagen, “Visual recognition of pointing gesturesfor human-robot interaction,” Image and Vision Computing, vol. 25,no. 12, pp. 1875–1884, Dec. 2007.

[7] B. Kuipers, “The spatial semantic hierarchy,” Artificial Intelligence,vol. 119, no. 1, pp. 191–233, 2000.

[8] H. Zender, O. Martı́nez Mozos, P. Jensfelt, G. Kruijff, and W. Bur-gard, “Conceptual spatial representations for indoor mobile robots,”Robotics and Autonomous Systems, vol. 56, no. 6, pp. 493–502, 2008.

[9] S. Hemachandra, T. Kollar, N. Roy, and S. Teller, “Following andinterpreting narrated guided tours,” in Proc. IEEE Int’l Conf. onRobotics and Automation (ICRA), 2011, pp. 2574–2579.

[10] A. Pronobis and P. Jensfelt, “Large-scale semantic mapping andreasoning with heterogeneous modalities,” in Proc. IEEE Int’l Conf.on Robotics and Automation (ICRA), 2012, pp. 3515–3522.

[11] M. R. Walter, S. Hemachandra, B. Homberg, S. Tellex, and S. Teller,“Learning semantic maps from natural language descriptions,” in Proc.Robotics: Science and Systems (RSS), 2013.

[12] M. MacMahon, B. Stankiewicz, and B. Kuipers, “Walk the talk:Connecting language, knowledge, and action in route instructions,”in Proc. Nat’l Conf. on Artificial Intelligence (AAAI), 2006, pp. 1475–1482.

[13] M. Skubic, D. Perzanowski, S. Blisard, A. Schultz, W. Adams,M. Bugajska, and D. Brock, “Spatial language for human-robotdialogs,” IEEE Trans. on Systems, Man, and Cybernetics, Part C:Applications and Reviews, vol. 34, no. 2, pp. 154–167, 2004.

[14] T. Kollar, S. Tellex, D. Roy, and N. Roy, “Toward understandingnatural language directions,” in Proc. ACM/IEEE Int’l. Conf. onHuman-Robot Interaction (HRI), 2010, pp. 259–266.

[15] S. Tellex, P. Thaker, R. Deits, T. Kollar, and N. Roy, “Towardinformation theoretic human-robot dialog,” in Proc. Robotics: Scienceand Systems (RSS), 2012.

[16] C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox,“A joint model of language and perception for grounded attributelearning,” in Proc. Int’l Conf. on Machine Learning (ICML), 2012.

[17] J.-L. Blanco, J. Gonzalez, and J. Fernandez-Madrigal, “Consistentobservation grouping for generating metric-topological maps thatimproves robot localization,” in Proc. IEEE Int’l Conf. on Roboticsand Automation (ICRA), 2006.

[18] M. Kaess, A. Ranganathan, and F. Dellaert, “iSAM: Incrementalsmoothing and mapping,” Trans. on Robotics, vol. 24, no. 6, pp. 1365–1378, 2008.

[19] J. M. Mooij, “libDAI: A free and open source C++ library for discreteapproximate inference in graphical models,” J. Machine LearningResearch, vol. 11, pp. 2169–2173, 2010.

[20] O. Mozos, C. Stachniss, and W. Burgard, “Supervised learning ofplaces from range data using adaboost,” in Proc. IEEE Int’l Conf. onRobotics and Automation (ICRA), 2005.

[21] A. Pronobis, O. M. Mozos, B. Caputo, and P. Jensfelt,“Multi-modal semantic place classification,” Int’l J. of RoboticsResearch, vol. 29, no. 2–3, pp. 298–320, 2010. [Online]. Available:http://www.pronobis.pro/publications/pronobis2010ijrr