Guiding Interaction Behaviors forMulti-modal Grounded Language Learning

Jesse Thomason, Jivko Sinapov, and Raymond J. MooneyDepartment of Computer Science, University of Texas at Austin

Austin, TX 78712, USA{jesse, jsinapov, mooney}@cs.utexas.edu

Abstract

Multi-modal grounded language learningconnects language predicates to physicalproperties of objects in the world. Sensingwith multiple modalities, such as audio,haptics, and visual colors and shapes whileperforming interaction behaviors like lift-ing, dropping, and looking on objects en-ables a robot to ground non-visual predi-cates like “empty” as well as visual pred-icates like “red”. Previous work has es-tablished that grounding in multi-modalspace improves performance on objectretrieval from human descriptions. Inthis work, we gather behavior annotationsfrom humans and demonstrate that theseimprove language grounding performanceby allowing a system to focus on relevantbehaviors for words like “white” or “half-full” that can be understood by lookingor lifting, respectively. We also exploreadding modality annotations (whether tofocus on audio or haptics when performinga behavior), which improves performance,and sharing information between linguis-tically related predicates (if “green” is acolor, “white” is a color), which improvesgrounding recall but at the cost of preci-sion.

1 Introduction

Connecting human language predicates like “red”and “heavy” to machine perception is part of thesymbol grounding problem (Harnad, 1990), ap-proached in machine learning as grounded lan-guage learning. For many years, grounded lan-guage learning has been performed primarily invisual space (Roy and Pentland, 2002; Liu et al.,2014; Malinowski and Fritz, 2014; Mohan et al.,

2013; Sun et al., 2013; Dindo and Zambuto, 2010;Vogel et al., 2010). Recently, researchers haveexplored grounding in audio (Kiela and Clark,2015), haptic (Alomari et al., 2017), and multi-modal (Thomason et al., 2016) spaces. Multi-modal grounding allows a system to connect lan-guage predicates like “rattles”, “empty”, and “red”to their audio, haptic, and color signatures, respec-tively.

Past work has used human-robot interaction togather language predicate labels for objects in theworld (Parde et al., 2015; Thomason et al., 2016).Using only human-robot interaction to gather la-bels, a system needs to learn effectively fromonly a few examples. Gathering audio and hapticperceptual information requires doing more thanlooking at each object. In past work, multiple in-teraction behaviors are used to explore objects andadd this audio and haptic information (Sinapovet al., 2014).

In this work, we gather annotations on what ex-ploratory behaviors humans would perform to de-termine whether language predicates apply to anovel object. A robot could gather such informa-tion by asking human users which action wouldbest allow it to test a particular property, e.g. “Totell whether something is ‘heavy’ should I lookat it or pick it up?” Figure 1 shows some of thebehaviors used by our robot in previous work toperceive objects and their properties. In this pa-per, we show that providing a language ground-ing system with behavior annotation informationimproves classification performance on whetherpredicates apply to objects, despite having sparsepredicate-object labels.

We additionally explore adding modality anno-tations (e.g. is a predicate more auditory or morehaptic), drawing on previous work in psychologythat gathered modality norms for many words (Ly-nott and Connell, 2009). Finally, we explore using

grasp lift lower

drop press push

Figure 1: Behaviors the robot used to explore ob-jects. In addition, the hold behavior (not shown)was performed after the lift behavior by holdingthe object in place for half a second. The lookbehavior (not shown) was also performed for allobjects.

word embeddings to help with infrequently seenpredicates by sharing information with more com-mon ones (e.g. if “thin” is common and “narrow”is rare, we can exploit the fact that they are lin-guistically related to help understand the latter).

2 Dataset and Methodology

Previous work provides sparse annotations of 32household objects (Figure 2) with language pred-icates derived during an interactive “I Spy” gamewith human users (Thomason et al., 2016). Eachpredicate p ∈ P from that work is associatedwith objects as applying or not applying, basedon dialog with human users. For example, pred-icate “red” applies to several objects and not toothers, but for many objects its label is not ex-plicitly known. Objects are represented by fea-tures gathered during several interaction behav-iors (Figure 1) as detailed in past work (Sinapovet al., 2016). In this work, we focus on improv-ing the language grounding performance of multi-modal classifiers that predict whether each predi-cate p ∈ P applies to each object o ∈ O.

In previous work, decisions about a predicateand an object are made for each sensorimotor con-text (a combination of a behavior and sensorymodality) with an SVM using the feature spacefor that context (Thomason et al., 2016). A sum-mary of sensorimotor contexts is given in Table 1.

Figure 2: Objects explored via interaction behav-iors and for which we have sparse predicate anno-tations.

Behaviors Modalitieslook color, fpfhdrop, grasp, hold, liftlower, press, push audio, haptics

Table 1: The contexts (combinations of robot be-havior and perceptual modality) we use for multi-modal language grounding. The color modalityis color histograms, fpfh is fast-point feature his-tograms, audio is fast Fourier transform frequencybins, and haptics is averages over robot arm jointforces (detailed in (Sinapov et al., 2016)).

For example, a classifier is trained from the pos-itive and negative object examples for “red” inlook/color space as well as in the less relevantdrop/audio space. These decisions are then av-eraged together, each weighted by its Cohen’s-κagreement with human labels using leave-one-outcross validation on the training data. In this way,the look/color space for “red” is expected to havehigh κ and a large influence on the decision, whiledrop/audio would have low κ and not influence thedecision much.

The decision d(p, o) ∈ [−1, 1] for predicate pand object o is defined as:

d(p, o) =∑c∈C

κp,cGp,c(o), (1)

for Gp,c a supervised grounding classifier trainedon labeled objects for predicate p in the featurespace of sensorimotor context c that returns in{−1, 1} with κp,c its agreement with human la-bels. If d(p, o) ≤ 0, we say p does not apply to o,else that it does. We use SVMs with linear kernelsas grounding classifiers.

We extend the weighting scheme between sen-sorimotor SVMs to include behavior information.For each predicate derived from the “I Spy” game

Figure 3: The distribution over annotator-chosenbehaviors (left) gathered in this work, as well asthe distribution over modality norms (right) de-rived from previous work (Lynott and Connell,2009), for the predicates “white” and “round”.The fpfh modality is fast-point feature histograms.

in previous work, we gather relevant behaviorsfrom human annotators. Annotators were askedto mark which among the 8 exploratory behav-iors (Table 1) they would engage in to determinewhether a given predicate applied to a novel ob-ject. Annotators could mark as many behaviors asthey wanted for each predicate, but were requiredto choose at least one.

We gathered annotations from 14 people, thendiscarded the annotations from those whose av-erage κ agreement with all other annotators wasless than 0.4 (the poor-fair agreement range). Thisleft us with 8 annotators whose average κ = .475(moderate agreement). We release the full setof gathered annotations on 81 perceptual predi-cates across 8 behaviors as a corpus for commu-nity use.1

Then, for each p ∈ P , we induce a distributionover behaviors b ∈ B based on the ratio of an-notators that marked that behavior relevant, suchthat

∑b∈B A

Bp,b = 1, with AB

p,b equal to the pro-portion of annotators who marked behavior b rel-evant for understanding predicate p. Some predi-cates, like “white”, have single behavior distribu-tions. For other predicates, like “metal”, annota-tors chose more complex combinations of behav-iors. Figure 3 (Left) gives some examples of be-havior distributions from our annotations.

1http://jessethomason.com/publication_supplements/robonlp_thomason_mooney_behavior_annotations.csv

The decision dB(p, o) considering behavior an-notations is calculated as

dB(p, o) =∑c∈C

ABp,cb

κp,cGp,c(o), (2)

where cb is the behavior for sensorimotor contextc.

We also experiment with adding modality an-notations (Table 1). In particular, we derive amodality distribution for each p ∈ P such that∑

m∈M AMp,m = 1 from modality exclusivity

norms gathered by past work for auditory, gusta-tory, haptic, olfactory, and visual modalities (Ly-nott and Connell, 2009). We ignore gustatoryand olfactory modalities, which have no counter-part in our sensorimotor contexts, and create AM

p,m

scores from the auditory, haptic, and visual modal-ity norm means. The visual modality norm issplit evenly between relevance scores AM

p,color andAM

p,fpfh, our visual color and shape modalities.The decision dM (p, o) considering modality an-

notations is calculated as

dM (p, o) =∑c∈C

AMp,cb

κp,cGp,c(o) (3)

When the predicate p does not appear in the norm-ing dataset from past work2, a uniform AM

p,m =1/|M | is used. Figure 3 (Right) gives some exam-ples of modality distributions from these norms.

The data sparsity inherent in language ground-ing from limited human interaction means somepredicates have just a handful of positive andnegative examples, while more common predi-cates may have many. If we have few exam-ples for “narrow” but many for “thin,” we canshare some information between them. For ex-ample, if κthin,grasp/haptic is high, we should trustthe grasp/haptic sensorimotor context for “nar-row” more than “narrow”s κ estimates alone sug-gest.

We explore sharing κ information betweenrelated predicates by calculating their cosinedistance in word embedding space by usingWord2Vec (Mikolov et al., 2013) vectors derivedfrom Google News.3 For every pair of predicatesp, q ∈ P with word embedding vectors vp, vq wecalculate similarity as

w(p, q) =1

2(1 + cos(vp, vq)), (4)

2About half the predicates have norming information.3https://github.com/mmihaltz/

word2vec-GoogleNews-vectors

p r f1mc .282 .355 .311κ .406 .460 .422B+κ .489 .489 .465M+κ .414 .466 .430W+κ .373 .474 .412

Table 2: Precision (p), recall (r), and f1 (f1)of predicate classifiers across weighting schemes.mc gives majority class baseline. Weightingschemes consider only validation confidence (κ,as in previous work), confidence and behavior an-notations (B+κ), confidence and modality anno-tations (M+κ), and confidence and word similar-ity (W+κ). Note that we show the average per-predicate f -measure, not the f -measure of the av-erage per-predicate precision and recall.

which falls in [0, 1], and subsequently take aweighted average of κ values using these similari-ties as weights to get decisions dW (p, o) as

dW (p, o) =∑c∈C

|P |−1 ∑q∈P

κq,cw(p, q)

Gp,c(o)

(5)

3 Experimental Evaluation

We calculated precision, recall, and f -measure be-tween human labels and predicate decisions whenweighting constituent sensorimotor context classi-fiers by the schemes described above: kappa confi-dence only (Eq 1, κ), adding behavior annotations(Eq 2, B+κ), adding modality annotations (Eq 3,M+κ), and sharing kappas across predicates usingword similarity (Eq 5, W+κ).

We calculated these metrics for each predicate4

and averaged scores across all predicates. Weuse leave-one-object-out cross validation to obtainperformance statistics for each weighting scheme.

Table 2 gives the results for predicates that haveat least 3 positive and 3 negative training objectexamples.5

We observe that adding behavior annotations ormodality annotations improves performance over

4Decisions were made for each testing object and markedcorrect or incorrect against human labels that object, if avail-able for the predicate.

5The trends are similar when considering all predicates,but the scores and differences in performance are lower dueto many predicates having only a single positive or negativeexample.

using kappa confidence alone, as was done in pastwork. Sharing kappa confidences across similarpredicates based on their embedding cosine simi-larity improves recall at the cost of precision.

Adding behavior annotations helps more thanadding modality norms, but we gathered behav-ior annotations for all predicates, while modal-ity annotations were only available for a subset(about half). Adding behavior annotations helpedthe f -measure of predicates like “pink”, “green”,and “half-full”, while adding modality annotationshelped with predicates like “round”, “white”, and“empty”.

Sharing confidences through word similarityhelped with some predicates, like “round”, at theexpense of domain-specific meanings of predi-cates like “water”. In the “I Spy” paradigm fromwhich these data were gathered, the authors notedthat “water” correlated with object weight becauseall of their water bottle objects were partially orcompletely full (Thomason et al., 2016). Thus, inthat domain, “water” is synonymous with “heavy”.In a less restricted domain, word similarity mayadd less real world “noise” to the problem.

4 Conclusions and Future Work

In this work, we have demonstrated that behaviorannotations can improve language grounding fora platform with multiple interaction behaviors andmodalities. In the future, we would like to applythis intuition in an embodied dialog agent. If a per-son asks a service robot to “Get the white cup.”,the robot should be able to ask “What should Ido to tell if something is ‘white’?”, a behavior an-notation prompt. A human-robot POMDP dialogpolicy could be learned, as in previous work (Pad-makumar et al., 2017), to know when this kind offollow-up question is warranted.

Additionally, we will explore other methods ofsharing information between predicates from lex-ical information. For example, choosing a maxi-mally similar neighboring word, rather than doinga weighted average across all known words, mayyield better results (e.g. the best neighbor of “nar-row” is “thin”, so don’t bother considering thingslike “green” at all).

Acknowledgments

We thank our anonymous reviewers for theirtime and insights. This work is supported by aNational Science Foundation Graduate Research

Fellowship to the first author, an NSF EAGERgrant (IIS-1548567), and an NSF NRI grant (IIS-1637736). A portion of this work has taken placein the Learning Agents Research Group (LARG)at UT Austin. LARG research is supportedin part by NSF (CNS-1330072, CNS-1305287,IIS-1637736, IIS-1651089), ONR (21C184-01),AFOSR (FA9550-14-1-0087), Raytheon, Toyota,AT&T, and Lockheed Martin.

ReferencesMuhannad Alomari, Paul Duckworth, David C. Hogg,

and Anthony G. Cohn. 2017. Natural language ac-quisition and grounding for embodied robotic sys-tems. In Proceedings of the Thirty-First AAAI Con-ference on Artificial Intelligence. pages 4349–4356.

Haris Dindo and Daniele Zambuto. 2010. A proba-bilistic approach to learning a visually grounded lan-guage model through human-robot interaction. InInternational Conference on Intelligent Robots andSystems. IEEE, Taipei, Taiwan, pages 760–796.

S. Harnad. 1990. The symbol grounding problem.Physica D 42:335–346.

Douwe Kiela and Stephen Clark. 2015. Multi- andcross-modal semantics beyond vision: Groundingin auditory perception. In Proceedings of the 2015Conference on Emperical Methods in Natural Lan-guage Processing. Lisbon, Portugal, pages 2461–2470.

Changson Liu, Lanbo She, Rui Fang, and Joyce Y.Chai. 2014. Probabilistic labeling for efficient ref-erential grounding based on collaborative discourse.In Proceedings of the 52nd Annual Meeting of theAssociation for Computational Linguistics. Balti-more, Maryland, USA, pages 13–18.

Dermot Lynott and Louise Connell. 2009. Modalityexclusivity norms for 423 object properties. Behav-ior Research Methods 41(2):558–564.

Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In Proceed-ings of the 28th Annual Conference on Neural In-formation Processing Systems. Montreal, Canada,pages 13–18.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor-rado, and Jeffrey Dean. 2013. Distributed represen-tations of words and phrases and their composition-ality. In Proceedings of the 26th International Con-ference on Neural Information Processing Systems.Lake Tahoe, Nevada, pages 3111–3119.

Shiwali Mohan, Aaron H. Mininger, and John E. Laird.2013. Towards an indexical model of situatedlanguage comprehension for real-world cognitive

agents. In Proceedings of the 2nd Annual Confer-ence on Advances in Cognitive Systems. Baltimore,Maryland, USA.

Aishwarya Padmakumar, Jesse Thomason, and Ray-mond J. Mooney. 2017. Integrated learning of dia-log strategies and semantic parsing. In Proceedingsof the 15th Conference of the European Chapter ofthe Association for Computational Linguistics. Va-lencia, Spain, pages 547–557.

Natalie Parde, Adam Hair, Michalis Papakostas,Konstantinos Tsiakas, Maria Dagioglou, VangelisKarkaletsis, and Rodney D. Nielsen. 2015. Ground-ing the meaning of words through vision and inter-active gameplay. In Proceedings of the 24th Inter-national Joint Conference on Artificial Intelligence.Buenos Aires, Argentina, pages 1895–1901.

Deb Roy and Alex Pentland. 2002. Learning wordsfrom sights and sounds: a computational model.Cognitive Science 26(1):113–146.

Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, andPeter Stone. 2016. Learning to order objects usinghaptic and proprioceptive exploratory behaviors. InProceedings of the 25th International Joint Confer-ence on Artificial Intelligence.

Jivko Sinapov, Connor Schenck, and AlexanderStoytchev. 2014. Learning relational object cate-gories using behavioral exploration and multimodalperception. In IEEE International Conference onRobotics and Automation.

Yuyin Sun, Liefeng Bo, and Dieter Fox. 2013. At-tribute based object identification. In InternationalConference on Robotics and Automation. IEEE,Karlsruhe, Germany, pages 2096–2103.

Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Pe-ter Stone, and Raymond Mooney. 2016. Learn-ing multi-modal grounded linguistic semantics byplaying “I spy”. In Proceedings of the 25th Inter-national Joint Conference on Artificial Intelligence.pages 3477–3483.

Adam Vogel, Karthik Raghunathan, and Dan Jurafsky.2010. Eye spy: Improving vision through dialog. InAssociation for the Advancement of Artificial Intel-ligence. pages 175–176.