Learning Grounded Relational Symbols from Continuous Data forAbstract Reasoning

Nikolay Jetchev, Tobias Lang, Marc Toussaint

Abstract— Learning from experience how to manipulate anenvironment in a goal-directed manner is one of the centralchallenges in research on autonomous robots. In the case ofobject manipulation, efficient learning and planning shouldexploit the underlying relational structure of manipulationproblems and combine geometric state descriptions with ab-stract symbolic representations. When appropriate symbols arenot predefined they need to be learned from geometric data.In this paper we present an approach for learning symbolicrelational abstractions of geometric features such that thesesymbols enable a robot to learn abstract transition modelsand to use them for goal-directed planning of motor primitivesequences. This is framed as an optimization problem, wherea loss function evaluates how predictive the learned symbolsare for the effects of given motor primitives as well as forreward. The approach is embedded in a full-fledged symbolicrelational model-based reinforcement learning setting, whereboth the symbols as well as the abstract transition and rewardmodels are learned from experience. We quantitatively comparethe approach to simpler baselines in an object manipulationtask and demonstrate it on a real-world robot.

I. INTRODUCTION

How can autonomous robots learn to manipulate theirenvironment in a goal-directed manner? In natural envi-ronments composed of objects, a robot has to reason ona geometric as well as on an abstract level to learn andplan sequences of motor primitives [1]. The geometriclevel concerns mostly sub-symbolic perception and motiongeneration where machine learning and robotics researchwas greatly successful in the last decades. In contrast, toefficiently determine an appropriate sequence of motionsfrom a given set of motor primitives requires learning andreasoning on an abstract level. Symbolic relational represen-tations are a promising approach to abstract reasoning. Theyreflect the structure inherent in object manipulation tasks interms of abstract properties and relations between objects;and they generalize over object identities to new situationswith previously unseen objects. For this reason, symbolicrepresentations have been investigated in machine learningand artificial intelligence from their beginnings.

It is a key open challenge, however, how sub-symbolicand symbolic learning and reasoning approaches can becombined. To apply symbolic approaches in the physicalworld, one needs to answer the fundamental questions:“Where do the symbols come from?” and “How are the givensymbols grounded in the physical world?” This is known asthe symbol grounding problem in psychology and cognitive

The authors are with the Machine Learning and RoboticsLab, FU Berlin, Arnimallee 7, 14195 Berlin, Germany {nikolay.jetchev,tobias.lang,marc.toussaint}@fu-berlin.de

Geometric features

Symbolic abstraction

Learning and planning

Fig. 1. An illustration of our approach: using continuous geometric featuresy from its sensors the robot learns symbols σ and uses them in a symbolicstate description s for planning an action sequence maximizing the reward.Our major contribution is the learning step indicated by the filled red arrowwhich takes the whole robot performance into account.

science [2]. In our view, a core challenge to answer thesequestions and to progress in the research on autonomousagents is to leverage learning methods to bridge the gapbetween continuous state descriptions and symbolic features.Learning should bootstrap from continuous state descriptionssymbolic abstractions which are empirically useful as a basisfor symbolic learning and reasoning.

In this paper, we approach the problem by assuming thatthe robot has a fixed set of motor primitives (correspondingto symbolic actions) and perceives continuous geometricproperties and relations between objects. To be able to chooseand sequence appropriate actions, the robot has to learnabstract state symbols which capture the general patternsunderlying the effects of its motor primitives on thoseobjects. We formalize this symbol learning problem in areinforcement learning (RL) framework where the goal of therobot is to find motor primitive sequences which maximizeits future rewards. The suitability of learned state symbolsgets thereby directly linked to the robot’s ability to performgoal-directed behavior.

We define a symbol as a relational predicate (initiallywithout semantics) paired with a mapping (classifier) fromthe geometric features to a true/false value of the predicate.Learning symbols then translates to learning appropriateclassifiers. We propose a loss function which favors symbolsthat allow to consistently and discriminately predict theeffects of motor primitives and rewards on an abstract level.We integrate this symbol learning procedure in a full-fledgedsymbolic relational model-based RL approach: the symbols(associated classifiers) as well as the abstract transition andreward models are all learned from experience. Experimentsin a robot manipulation scenario, both in simulation and ona real robot, show that this integrated approach allows an

autonomous robot to find a task-relevant structure in its stateobservations which it can exploit for efficient planning ofsequences of motor primitives. Figure 1 illustrates our work.

In the next section we discuss related and previous workon symbol learning. Section III summarizes background onrelational reinforcement learning, on which our approach inparts builds. Section IV then introduces our specific approachto formalizing the symbol learning problem before, in Sec-tion V, we define the loss function for learning the symbol’sclassifiers. In Section VI we demonstrate the approach ona robot manipulation domain in simulation and in the realworld.

II. RELATED WORK

The origin of symbols that humans seem to use to structuretheir life has been a subject of study for long [2]. In alanguage acquisition context, linguists have researched spa-tial predicate extraction from natural language instructionsin a supervised way. [3] learn the meaning of words like“past” and “to” by analyzing motion trajectories and theirdescriptions from a teacher. Approaches to extract rules andnon-relational symbols from a neural network trained forsupervised learning have been investigated in very simplecomputer simulation scenarios [4], [5]. [6] showed that it isimportant that grounded symbols are coupled and proposedpredictive, correlation and contrastive criteria for learning. Ina biological context, [7] showed that apes can learn symbolsfrom examples provided by a teacher who provides signifi-cant guidance. In contrast to our work, all these approachesdo not focus on learning symbols to enable an autonomousrobot to act in a goal-directed way in a world with objects.

From their early beginnings, research in artificial intel-ligence and cognitive science has investigated modelingreasoning and learning of autonomous agents based on sym-bolic relational representations of the world [8], not withoutsignificant objections [9]. Over the last decade, symbolicrelational approaches have received new enthusiasm with theadvent of machine learning techniques in AI and cognitivescience [10], [11]. However, the critique of “where the sym-bols come from” remains. Feldman [12] provides a formalcomputational model of the conditions when a symbolicrepresentation can capture the variability of a continuousenvironment. But how symbols can be learned and used forreasoning has, to our knowledge, not been shown. Generally,learning symbols in the context of actually working andacting systems has seen only little progress. [13] investi-gated how a real robot can learn to manipulate articulatedobjects using a grounded relational representation, but didnot learn the kinematic symbols themselves. [14] discussesthat a robot needs to learn structured symbols accordingto experience which reflect spatiotemporal correlations inits sensory predictions, motor signals, and internal variables(sensorimotor invariances in its actions). [15] present a studyof a real robot which learns non-relational symbols by meansof neural networks for manipulation tasks. To our knowledge,our approach is the first attempt to learn symbols to enablemodel-based relational reinforcement learning.

III. BACKGROUND ON RELATIONAL RLIn this section we review background on relational (esp.

model-based) reinforcement learning [10]—an area of re-search which so far has been applied using predefinedsymbolic representations. Consider an environment with Mobjects. Learning transition models (the probabilistic effectsof motor primitives or actions) directly on a geometricrepresentation is in general hard. Natural environments areprofoundly structured in terms of their compositionalityby objects. A standard approach to describe the state ofmultiple objects is in terms of a set of unary and binarypredicates P . For instance, given M objects, K unarypredicates and L binary predicates, the propositional statedescription s ∈ {0, 1}ν is given by the ν = KM + LM2

truth values for the instantiations pk(om)k=1:K,m=1:M andpl(om, om′)l=1:L,m=1:M,m′=1:M of all predicates with allobjects (in all permutations)—the propositional state spaceis exponential in the number of objects.

Relational representations generalize over object identitiesand thereby reflect the prior assumption that the effects ofactions should not depend on the explicit identity of anobject, but only on its properties and relations to other ob-jects. Relational reinforcement learning developed methodseither for learning relational value functions [10] or learningcompact relational transition models P (s′ | s, a) [16] andusing them in a model-based RL setting for planning orfurther exploration. This generalization greatly reduces theamount of data necessary for an agent to find a good policyand transition model in comparison to propositional learning[17]. In our approach we will learn a transition model(based on learned symbols) in the form of noisy probabilisticrelational rules [16] from concrete experience in the formatD = {(st, at, st+1)}t, where st, st+1 are propositional statedescriptions before and after the application of an actionat. We use the planning method PRADA [17] based on thelearned rules to find action sequences which maximize theexpected reward.

IV. SYMBOL LEARNING PROBLEM FORMULATION

Consider an environment with M objects O ={o1, . . . , oM} in state x. We assume that, given the statex, the robot has access to a geometric state descriptor ycomprising features yi ∈ Rn for each object oi and featuresyij ∈ Rn′

for each pair oi, oj of objects. For instance,yi may represent the position and size of oi as computedfrom the robot’s vision system; while a basic example forpair-wise features is yij = yi − yj . The robot can executemotor primitives a of different types A on target objectso ∈ O; e.g., grab objects and put them on other objects. Theexecution of motor primitives changes the configuration andthus observed geometric features y of one or more objects.

We are interested in relational symbols to describe theeffects of motor primitives. We define a relational symbolσ = (p, f) as a predicate p together with its groundingf . The predicate on its own (without the grounding f ) iswithout semantics. A grounding f is a classifier that decidesfor all possible permutations o of the objects (for instance,in the case of a binary predicate for all pairs (oi, oj),

oi, oj ∈ O) whether the corresponding instantiation p(o) ofthe predicate holds given the geometric features {yi, yij}for o. For instance, we calculate the grounding of a unarypredicate instantiated with a single object oi from its featuresyi by f1 : Rn → {0, 1} and similarly for binary predicatesfor a pair oi, oj with features yij by f2 : Rn′ → {0, 1}.The predicates together with their binary classifiers form theset of relational symbols Σ. The symbolic relational states ∈ {0, 1}ν is a binary vector that represents whether specificpredicate instantiations can be successfully grounded in statex: for each symbol σ ∈ Σ of arity b, there is an entry in s foreach permutation o of length b of the objects in O. The entrycorresponds to the truth value of pσ(o) which is calculatedby the classifier fσ from the geometric features {yi, yij} foro. A set of symbols Σ thereby defines the state abstractionfunction φ(x; Σ) → s which takes a physical world state xand abstracts it to the relational symbolic state s.

The robot receives a continuous reward signal r ∈ Rin each state. It interacts with its environment in discretetime-steps, resulting in a sequence x0, r0, a0, x1, r1, a1, . . .. The task of the robot is to maximize its rewards R =∑Tt=0 γ

trt for discounting 0 < γ < 1. Our approach is tolearn symbols Σ that allow to abstract xt to its symbolicrelational representation st and then to reason in the abstractrepresentation about motor primitives. Hence, the problemcan be summarized as learning classifiers fσ that definea set of symbols σ ∈ Σ such that model-based RL usingthese symbols will be successful in maximizing the expectedfuture reward. Both, the set of symbols as well as thesymbolic transition and reward models, have to be learnedfrom experiences D = {(yt, rt, at, yt+1)}. This problemformulation frames symbol learning as part of an overall rein-forcement learning problem—in contrast to symbol learningin a supervised fashion from a teacher.

V. LEARNING THE SYMBOL GROUNDINGS fσ

We define a loss function L(Σ;D) over data D to learnsymbols σ ∈ Σ, that is, to find classifiers fσ that minimizethis loss. The loss function needs to reflect our conception ofwhat good symbols are. As discussed in the previous section,the goal of symbol learning is to give the robot the ability toplan efficiently to high-reward states in the physical world.Hence, we want symbols to (i) allow us to learn a transitionmodel which one can use to predict the structural effectsof executing motor primitives and discriminate betweenpredecessor and successor states; (ii) allow us to learn areward model to predict how the reward values (definingtasks) are related to such structural changes; and (iii) becompact so as to allow for efficient reasoning. We definethe loss function

L(Σ;D) = Ltransition(Σ;D) + Lreward(Σ;D) + ‖Σ‖2 . (1)

Learning symbols works by minimizing this loss. The nextsubsections will explain in detail its three terms. As asidenote, in linear value function approximation the Bellmanerror can similarly be decomposed in reward and predictionterms [18].

A. Learning a transition model via Ltransition

Recall that st ∈ {0, 1}ν denotes the truth values of thepredicate instantiations in a world state xt for the givensymbols Σ and objects O. A model T can be learned byabstracting the experiences D = {(yt, rt, at, yt+1)} to theirsymbolic representation DΣ = {(st, rt, at, st+1)} based onthe symbols Σ. We define a loss term Ltransition which rewardssymbols that are suitable for transition model learning as

Ltransition(Σ;D) =∑

(st,rt,at,st+1)∈DΣ

‖st − st+1‖1 (2)

−∑

(st,rt,at,st+1)∈DΣ

logP (st+1 | st, at; T )

where P (s′ | s, a; T ) is the transition probability accordingto the potentially stochastic transition model T that isestimated from D as well. This loss function incorporatestwo different terms which we describe in the following.

The first term in Eq. (3) rewards effect discrimination. Thisis defined by the Hamming distance ‖st − st+1‖1 betweenstate pairs st, st+1. In the data D, motor primitives at alwayschange object configurations in the physical world, that isyt,i 6= yt+1,i for at least one oi ∈ O. Learned symbols shallreflect these changes to account for the structure inherent inthe world and the motor primitives.

The second term in Eq. (3) rewards prediction accuracy.This is defined by the data-likelihood of the successor statesin DΣ given the current states and actions according to thelearned symbolic relational transition model T . This rewardssymbols which allow to predict successor states accurately.Good relational symbols are not only those whose truthvalues reflect the changes according to motor primitives, butalso symbols whose truth values identify relevant objects anddefine contexts for the type of changes according to the motorprimitives.

In principle, any suitable relational method could be usedto learn the transition model, such as advanced relationalclassification techniques and probabilistic relational rules. Inour experiments, we use a basic relational nearest neighbor(NN) predictor. We use this efficient technique since weneed to estimate T frequently in every step of our overalloptimization procedure (described in Section VI) for Σ.For a given state-action pair s, a, this NN predictor findsthe nearest neighbor st, at in the data DΣ and predictsthe according successor state st+1. Relational generalizationover the objects in s, a is achieved by using a permutation-invariant kernel that is described in the Appendix.

B. Learning a reward model via Lreward

It is crucial for planning that learned symbols allow todetermine state rewards (which define the robot’s task). Thisis achieved by a relational regressor R : s → R andmotivates the loss term

Lreward(Σ;D) = −∑

(st,rt,at,st+1)∈DΣ

logP (rt | st;R) . (3)

where P (rt | st;R) is the probability of reward rt in staccording to the potentially stochastic reward model R. This

loss reflects how well learned symbols Σ can be used to solvethe relational regression problem of predicting the reward rtfrom the symbolic state representation st.

In our experiments, we again use a relational NN regressorfor R. For a given state s, this regressor looks throughthe data DΣ for the nearest neighbor st and predicts theaccording reward rt. As above, relational generalization isachieved by the permutation-invariant kernel described in theAppendix.

C. Regularizing symbols Σ

The last term in the loss function in Eq. (1), ‖Σ‖2,regularizes the learned symbols Σ. This regularization canbe used to formalize different requirements on symbols.Our idea is that we want grounded symbols to representdistinct geometric structures within physical states. Thisis advantageous for model selection when the number ofsymbols is unknown in advance: we can start learning withmultiple candidate symbols; if a symbol is not needed torepresent abstract state structures, it gets pruned from themodel due to regularization.

To specify the regularization we need to consider aconcrete function class for fσ to ground symbols. We useradial basis function (RBF) classifiers which are defined bya parameter vector w = (wc, ws) consisting of a center anda standard deviation. A symbol σ = (p, w) is then definedby the predicate p and the RBF parameters w. For a binarysymbol, for instance, we calculate the truth value of p(oi, oj)from the activation function

g(yij ;w) = exp(−(yij − wc)> diag(w2s)−1(yij − wc))

(4)

based on the pair-wise geometric feature vector yij of theobjects oi and oj . If g(yij ;w) > 0.5, then p(oi, oj) is true.The regularization now applies to the parameterizations w =(wc, ws). We focus regularization on the widths ws whichcontrol the activation areas of symbols defined by g(y)>0.5.This regularization leads to a trade-off between predictivepower and small activation areas.

VI. EXPERIMENTAL EVALUATION

We investigated whether (a) our approach allows to learnsymbols from continuous geometric data, (b) the resultingsymbols are well grounded in the physical world, and (c)the learned symbols enable planning on the symbolic levelleading the robot to higher rewards. For learning (approxi-mately) optimal symbols Σ∗, that is the parameters w∗ for theRBF classifiers fσ in our case, we use a stochastic searchalgorithm (simulated annealing). To speed up convergencewe used an operator which swaps complete symbols (wc, ws)at once between candidate solutions. About 105 loss functionevaluations were required to reach optima in the differentsetups.

To evaluate our approach on two different tasks, manipu-lating cubes and balls on a table, and playing bowling, andperformed a qualitative real-world experiment. The accom-panying video includes a real-world experiment as well as avideo of our bowling simulation. For each task, we collected

traces ∆ = {⟨yit, a

it, r

it

⟩}n,Ti=1,t=1 of n length-T -sequences of

random actions with random start states, i.e. nT data points.These traces were used by the compared methods for learningappropriate models and symbols for planning.

A. Compared methods

We compared four different methods based on eithercontinuous or symbolic representations. RANDOM performsrandom actions. CONTINUOUS-NN (nearest neighbor) isa baseline that exploits the available data in a relationalway but without using a symbolic representation. Geometricstates yit within these traces are assigned values V (yit) =∑3d=0 γ

drit+d with a discount factor γ = 0.8. (The y actuallydenotes the set of geometric features y = {yi, yij}. In thismethod, yi are treated as states.) We set rt+d := rT fort+ d > T . Given a new geometric state y the robot choosesan action based on nearest neighbor retrieval from thesetraces: the closest states to y in ∆ are retrieved based on therelational permutation-invariant similarity kernel k(y, y′) incontinuous space (defined in the Appendix). Then the actiona∗ of the nearest neighbor y∗ with the largest value V (y∗) ischosen, where the action arguments are assigned accordingto the permutation-invariant kernel. SYMBOLIC-NN is equalto CONTINUOUS-NN except that states in the traces arerepresented abstractly using the learned grounded symbolsΣ and the permutation-invariant similarity kernel k(s, s′)in S is used. The trace data ∆Σ = {

⟨sit, a

it, r

it

⟩}n,Ti=1,t=1

corresponds to paths through the abstract symbolic statespace. SYMBOLIC-PRADA is a relational planner whichuses learned noisy probabilistic relational rules to find appro-priate symbolic actions (see Sec. III). Rules are learned fromthe learned symbolic abstraction ∆Σ of the experiences—thesame data set used for SYMBOLIC-NN. To evaluate actionsequences, PRADA requires a symbolic description of therewards. We approximate the reward function with a simplesymbolic regression using relational decision trees with thelearned symbols.

We devised these methods to allow for performance com-parison w.r.t. different aspects of the representations used.Both CONTINUOUS-NN and SYMBOLIC-NN are relationalin the sense that they abstract over object identities viathe permutation-invariant kernel. Only SYMBOLIC-NN andSYMBOLIC-PRADA use the learned grounded symbols, andonly SYMBOLIC-PRADA represents a full-blown model-based relational RL approach. With this we aim to separateout the effects of a relational representation and the use oflearned symbols.

B. Simulated robot manipulation scenario

In our first scenario, a physically simulated robot manip-ulates balls and cubes on a table (Fig. 3(a)). The robot canexecute two motor primitives which are affected by noise: itcan grab an object or open its hand over some other object;A = {closeHandAround(X), openHandOver(X)}. Aftergrabbing, the robot always retracts to a neutral “home”position. The object observations of the robot are describedby continuous feature vectors yi ∈ R4 comprising the 3-dimensional position of the center and the size of object oi.

0.1

0.2

30 60 90 120 150

#Data for symbol learning

Lo

ss(S

ym

bo

ls;

Da

ta)

randomlearned

(a)

0

1

2

3

30 60 90 120 150

#Data for symbol learning

Re

wa

rd

randomcontinuous NN

symbolic NNsymbolic PRADA

(b) Only cubes: 3 traces

0

1

2

3

30 60 90 120 150

#Data for symbol learning

Re

wa

rd

randomcontinuous NN

symbolic NNsymbolic PRADA

(c) Only cubes: 9 traces

0

0.5

1

1.5

2

2.5

3

80 160

#Data for symbol learning

Re

wa

rd

randomcontinuous NN

symbolic NNsymb PRADA

(d) Cubes and ballsFig. 2. Simulated robot manipulation scenario. (a) Symbol learning loss ofrandomly defined and learned symbols on 400 test states. (b) / (c) Resultsfor scenario with cubes based on model learning with 3 / 9 traces. Thereward for performing no actions is 0. (d) Results for cubes and balls.

The task is to build towers. This is defined by means of areward function: the average discounted continuous heightof cubes over 10 time-steps.

In our first experiment there were 7 cubes of two distinctsizes on the table. We learned one unary predicate u(X)and one binary predicate b(X,Y ). We tried learning moresymbols, but the regularization term of the loss functionwould usually eliminate additional symbols. Our results areshown in Figs. 2(a), 2(b) and 2(c). We learned symbols basedon increasing numbers of data in D, as given on the x-axes of the diagrams. Fig. 2(a) shows how the validatedloss of the learned symbols improves with an increasingnumber of training data. For learning models (traces forthe NN approaches and the probabilistic relational rulesfor SYMBOLIC-PRADA), we used independent data setsof size 30 (n = 3 traces with T = 10) (Fig. 2(b)) or 90(n = 9 traces) (Fig. 2(c)). Each combination of learnedsymbols and models was evaluated in three new startingsituations where all objects lay on the table. Thus, theinitial reward was 0. The hypothetically optimal reward inthe corresponding deterministic world was 2.963. To getstatistics, we performed 90 runs with different data splitinto independent sets for symbol and model learning anddifferent random seeds. RANDOM is independent of symboland model learning; its performance is thus always thesame in Figs. 2(b) and 2(c). CONTINUOUS-NN performsbetter than RANDOM with a sufficiently large number ofdata for NN retrieval of actions. The symbolic approachesshow superior performance even with limited data for modellearning (Fig. 2(b)). Due to the learned symbolic abstraction,SYMBOLIC-NN performs better with more data for NNretrieval (Fig. 2(c)) as long as the learned symbols aresufficiently accurate. In contrast, SYMBOLIC-PRADA is lesssensitive to noisy or less accurately learned symbols andoutperforms all other methods. This is due to the followingreason: in case of less accurately grounded symbols, sym-bolic structures sometimes get overlooked in a state xit. For

instance, the unary symbol u(X) with the canonical meaningX in hand may not be detected in xit although an object isheld in hand in xit. This is problematic for nearest neighbordetection. In contrast, SYMBOLIC-PRADA is not bound toNN retrieval in traces, but learns rules from breaking thetrace data into individual experiences. Individual faulty sym-bolic state abstractions are handled by SYMBOLIC-PRADAsince the underlying probabilistic relational rules cope withnoisy outcomes of actions and identify the typical outcomesof actions as required for effective planning.

In a second experiment, we investigated the four methodsin a scenario extended by balls (in addition to cubes; thetotal number of objects was seven as above). Here, welearned two unary and one binary symbol with our learningapproach. Our results for 80 and 160 data for symbollearning are presented in Fig. 2(d). We used n = 8 tracesof T = 10 actions to learn models. The results confirmthe previous findings: the symbolic methods show superiorperformance. The advantage of SYMBOLIC-PRADA is moresignificant here since the NN approaches would require muchmore traces for similar performance due to the increasedcomplexity of the situations.

Overall the results show the great potential of usinglearned relational symbols for state abstraction in combina-tion with sophisticated relational planning techniques. Wealso examined briefly the setup of using point cloud datafrom a Kinect sensor. For object features we used the meanof the point cloud and the first 7 PCA components, hencey ∈ R10. We were able to learn a symbol corresponding to aspherical shape with our learning method which can be usedto predict when an object cannot be stacked.

Figure 3 illustrates what was learned as a binary predicateb(X,Y ) in two exemplary runs. In the first run, it canbe interpreted as representing that X is directly on topof Y (specifically with a maximum difference in heightcorresponding to the largest object size); in the second run,that X has similar x/y coordinates as Y but may differin height—that X is in the tower stacked above Y . Inalmost all of our runs, the learned unary predicate u(X)could be interpreted as denoting that X is being held in therobot’s hand. In a few cases, u(x) was learned to denotethat an object is reachable. In the experiments with balls, anadditional unary symbol learned to distinguish between ballsand cubes by means of the object size (which was differentfor balls and cubes) since this is important in the decisionwhich objects to stack. Overall, the learned symbols appealto human intuition and reflect the task structure. For instance,a binary symbol with the meaning “next-to” which could berepresented with RBFs was not learned since it does not helpto predict and collect rewards here.

C. Simulated bowling scenario

For our second scenario we implemented a variant ofbowling with the ODE physical simulator, trying to modelas accurately as possible the setup of a ten-pin bowlinggame. There are 10 pins placed inside an equilateral trianglewith a side length of 60cm. Each pin is described by afeature vector yi ∈ R6 composed of the position dimensions

(a)

(b) (c)Fig. 3. Simulated robot manipulation scenario. (a) Simulator. (b) / (c) Twoexemplary groundings of a relational symbol b(o1, o2) learned in differentruns. The red volumes denote the isosurface of the activation function ofg(x) = 0.5 which is calculated from the continuous features y1,2 = y1 −y2, see Eq. (4); the axes are in meters. In (b), b(o1, o2) captures that o1 isdirectly on top of o2 such as b(oblue, oorange). In (c), o1 is above o2 inthe same stack such as b(ogreen, oblue).

and the z-axis vector, see Figure 4(a). The action set wasA = {throwAtCenter(X), throwRightOfCenter(X)}:the first is a ball-throw straight (aligned with the y-axis) atthe center of the target pin X , the second 7.5cm to the rightof the center of X . This small translation has an effect onthe ball trajectory, which is often deflected after collisions.The ODE simulation also has some noise in the collisionsbetween ball and pins.

Our symbol learning method learned 1 unary and 3 binarysymbols to describe the world, using training data D of 80random ball throws in smaller worlds with 5 pins placedrandomly on a subset of the 10 ball positions. We gatheredtraces of random throws with n = 25, 50, 100 and T = 2,i.e. the points are counted after 2 hits. We compared theperformance of our methods in 100 bowling games with 2hits each. Figure 4(b) indicates that the learned relationalsymbols capture essential information of the bowling world:SYMBOLIC-NN is the superior method. Figure 4(c) showsexamples of a binary symbol we learned: it represents thebasic geometric relation of two pins being on the same linew.r.t. the straight ball trajectories. Such a relation makes itlikely that throwing the ball at one pin will also knock out theother. Another learned symbol can be interpreted as “behindand 15cm to the left”, which is related to the possible balldeflections. The unary symbol could usually be interpretedas denoting “upright” pins.

D. Qualitative real-world experiment

We also performed also a qualitative study on a realrobot platform, depicted in Fig. 1, which consists of aSchunk Light Weight arm with 7 DoF, a Schunk DextrousHand with 7 DoF, 6 × 14 tactile arrays on each of the6 finger segments and a Bumblebee stereo camera. Therobot is placed in front of a table with four differentcans which it can recognize based on their color. The

(a)

6.5

7

7.5

8

8.5

9

25 50 100

#traces for model learning

Re

wa

rd

randomcontinuous NN

symbolic NN

(b)

(c)Fig. 4. Simulated bowling scenario. (a) The goal is to knock down 10simulated pins. The black line shows the ball path before deflections. (b)Results. (c) Exemplary learned binary symbol which indicates whether twoobjects have the same x-coordinate, i.e.“in the same column” for a ball hit.

robot has motor primitives to grab and release cans. Oneunary and one binary relational symbol is learned fromthe experience of 30 random actions. Based on the learnedsymbols, the robot learns to plan efficiently sequencesof motor primitives for building towers of cans. A videoof this study can be found at http://dl.dropbox.com/

u/17220615/RSS-Submission/RSS-Extra-Material.html.http://dl.dropbox.com/u/17220615/RSS%20Submission/RSS%20Extra%20Material.html.

VII. CONCLUSIONS AND FUTURE WORKSymbolic representations are a promising approach to

abstract reasoning of robots about sequences of motorprimitives. But where do these symbols come from? Wehave proposed a bootstrapping approach to learn relationalstate symbols from geometric object features assuming theexistence of a given set of motor primitives. We havedemonstrated in an integrated system that our approach opensa door for real-world robots to leverage methods developed inAI for symbolic reinforcement learning and planning. Thesemethods rely crucially on accurately grounded symbols.

Future work can combine our approach with active learn-ing with a teacher where the robot generates situations inwhich it is uncertain about the symbolic grounding. Anothermajor question is how to combine motor primitive learningmethods with our symbol learning approach, aiming towardsa mutual bootstrapping of motor primitives and respectivesymbolic representations.

APPENDIXIn several contexts we use a relational kernel k(s, s′)

which measures the similarity of states. While we coulduse more elaborated existing relational kernels, here weresorted to a simple kernel that achieves generalization byexhaustively testing all permutations with respect to theidentities of objects in the symbolic states. Let Π be the set ofpermutations of all M objects, containing M ! permutationsπ ∈ Π. Let κπ : {0, 1}ν → {0, 1}ν denote the reordering s

according to the order of objects in π and thus of their pred-icate instantiations pk(om), pl(om, om′). We define a kernelk based on Π as k(s, s′) = minπ∈Π exp(‖s − κπ(s′)‖2).This is the distance between s and the most similar permutedvariation of s′. A similar relational kernel can also be appliedon continuous features y, y′ ∈ Rq composed of objectfeatures yi. Let κπ : Rq → Rq denote the reordering ofthe features yi according to the permutation π. We get thekernel k(y, y′) = minπ∈Π exp(−‖y − κπ(y′)‖2) , showingthat relational methods are not limited to symbolic discreterepresentations.

ACKNOWLEDGMENTS

This work was supported by the German Research Foun-dation (DFG), grants TO 409/1-3 and TO 409/7-1.

REFERENCES

[1] L. P. Kaelbling and T. Lozano-Perez, “Hierarchical task and motionplanning in the now,” in ICRA, 2011, pp. 1470–1477.

[2] S. Harnad, “The symbol grounding problem,” Physica D, vol. 42, pp.335–346, 1990.

[3] T. Kollar, S. Tellex, D. Roy, and N. Roy, “Toward understandingnatural language directions,” in HRI, 2010, pp. 259–266.

[4] F. J. Kurfess, “Neural networks and structured knowledge: Ruleextraction and applications,” Applied Intelligence, vol. 12, pp. 7–13,January 2000.

[5] A. Cangelosi, A. Greco, and S. Harnad, Symbol Grounding and theSymbolic Theft Hypothesis. Springer, London, 2002.

[6] D. Billman and J. Knutson, “Unsupervised concept learning and valuesystematicity: A complex whole aids learning the parts,” Journal ofExperimental Psychology: Learning, Memory, and Cognition, vol. 22,no. 2, pp. 458 – 475, 1996.

[7] V. Truppa, E. Piano Mortari, D. Garofoli, S. Privitera, and E. Visal-berghi, “Same/different concept learning by capuchin monkeys inmatching-to-sample tasks,” PLoS ONE, vol. 6, no. 8, p. e23809, 082011.

[8] A. Newell and H. Simon, “Gps: A program that simulates humanthought,” in Computers and Thought, Feigenbaum and Feldman, Eds.McGraw-Hill, New York, 1963.

[9] H. Dreyfus, What Computers Can’t Do: The Limits of ArtificialIntelligence. New York, USA: MIT Press, 1972.

[10] S. Dzeroski, L. de Raedt, and K. Driessens, “Relational reinforcementlearning,” Machine Learning Journal, vol. 43, pp. 7–52, 2001.

[11] J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman, “Howto grow a mind: Statistics, structure, and abstraction,” Science, vol.331, no. 6022, pp. 1279–1285, 2011.

[12] J. Feldman, “Symbolic representations of probabilistic worlds,” Cog-nition, vol. 123, pp. 61–83, 2012.

[13] D. Katz, Y. Pyuro, and O. Brock, “Learning to manipulate articulatedobjects in unstructured environments using a grounded relationalrepresentation,” in RSS, 2008, pp. 254–261.

[14] K. F. MacDorman, “Grounding symbols through sensorimotor inte-gration,” Journal of the Robotics Society of Japan, vol. 17, no. 1, pp.20–24, 1999.

[15] E. Chinellato, A. Morales, E. Cervera, and A. P. D. Pobil, “Sym-bol grounding through robotic manipulation in cognitive systems,”Robotics and Autonomous Systems, vol. 55, no. 12, pp. 851–859, 2007.

[16] H. M. Pasula, L. S. Zettlemoyer, and L. P. Kaelbling, “Learning sym-bolic models of stochastic domains,” Journal of Artificial IntelligenceResearch (JAIR), vol. 29, pp. 309–352, 2007.

[17] T. Lang and M. Toussaint, “Planning with noisy probabilistic relationalrules,” Journal of Artificial Intelligence Research (JAIR), vol. 39, pp.1–49, 2010.

[18] R. Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. L. Littman,“An analysis of linear models, linear value-function approximation,and feature selection for reinforcement learning,” in Proceedings ofthe 25th Int. Conf. on Machine learning, ser. ICML 08, 2008, pp.752–759.