Robot Cognition using Bayesian Symmetry Networks

Anshul Joshi1, Thomas C. Henderson1 and Wenyi Wang2

1School of Computing, University of Utah, Salt Lake City UT2Department of Mathematics, University of Utah, Salt Lake City UT

{joshi, tch}@cs.utah.edu, u0845772@utah.edu

Keywords: Concept Representation, Wreath Product, Perceptual Organization, Bayesian Network

Abstract: (Leyton, 2001) proposes a generative theory of shape, and general cognition, based on group actions on setsas defined by wreath products. This representation relates object symmetries to motor actions which producethose symmetries. Our position expressed here is that this approach provides a strong basis for robot cognitionwhen:

1. sensory data and motor data are tightly coupled during analysis,

2. specific instances and general concepts are structured this way, and

3. uncertainty is characterized using a Bayesian framework.

Our major contributions are (1) algorithms for symmetry detection and to realize wreath product analysis, and(2) a Bayesian characterization of the uncertainty in wreath product concept formation.

1 INTRODUCTION

Our goal is to develop cognitive capabilities for au-tonomous robot agents. (Vernon et al., 2007) statethat cognition ”can be viewed as a process by whichthe system achieves robust, adaptive, anticipatory, au-tonomous behavior, entailing embodied perceptionand action.” For us, this includes the ability to:

1. analyze sensorimotor data in order to determine arational course of action,

2. represent and recognize important concepts de-scribing the world (including its own self),

3. recognize similarities between concepts,

4. extend concepts to new domains, and

5. determine likelihoods of assertions about theworld.

1.1 Related Work

Bayesian networks (BN) have been used previouslyfor inference in cognitive frameworks. Closely re-lated is the work of (Binford, 1987) who proposeda Bayesian network approach for object recognitionbased on generalized cylinders, but our method differsin that we use the more general wreath product rep-resentation (which can handle generalized cylinders)

and include actuation in the description; furthermore,we derive the Bayesian network directly from the hi-erarchical structure of the wreath product; also see(Zhang and Ji, 2011). Others have addressed issuessuch as spatial awareness and a conceptual model forplace classification; e.g., (Vasudevan and Siegwart,2008). However, in their approach the robot is as-sumed to have high-level object extraction capability(SIFT-based), is taught exemplars, and then searchesfor instances of those in the environment. They usea ”bag-of-words” approach, and no explicit geomet-ric information in used. In contrast to that, we pro-pose a group-theoretic approach to conceptualizationand classification (here - of surfaces based on their3D geometry). Our methods perform structure anal-ysis (i.e., detection and organization of symmetries)and could feed information into a system like Vasude-van’s.

(Gold and Scassellati, 2007) have proposed a wayfor a robot to perceive ”self and ”other” through mo-tion analysis, however their method is limited to therobot differentiating between itself and other animateobjects based on probabilistic inference.

(Buisson, 2001) has proposed a Piagetian ap-proach which resembles our in that it is based on thefollowing ideas (direct quote):• Idea 1 An interaction is a rhythmic succession of

actions, each step being assorted with a set of sen-

sory anticipations.

• Idea 2 Each and every interaction of the agentwith its environment is recorded. This recordingdoes not necessarily amount to a large volume ofdata, since only the sensorial and motor elementsactively implicated in the interaction are stored.

• Idea 3 All past stored interactions try permanentlyto get synchronized spatially and temporally withthe current situation, in order to influence the on-going interaction. In a word, the current inter-action is guided by previous ones, and will itselfserve later as a guide for others.

We also assume that action is essential to the devel-opment of concepts about the world, and that theseconcepts are used to guide current action.

Finally, (Tenenbaum and Griffiths, 2001) and(Jia et al., 2013) have looked at generalization andsimilarity in terms of Bayesian inference; the latterhas applied this to visual concept learning. However,Tenenbaum proposes generalization as the more basiccognitive function, wheras we hold that similarityis more primitive being based on innate symmetrytheories (in terms of similarity of sensor and actuatorsignals). These theories constitute the learner’sknowledge about consequence regions. Gener-alization is achieved by manipulating the wreathproduct representation (e.g., by replacing one groupby another), and the learner acquires knowledge byfinding wreath product structure in the sensorimotordata produced in interacting with the environment.We believe that such an active methodology is morerobust and helps explain a number of findings, e.g.,mirror neurons.

1.2 Wreath Product Representation

We have previously argued that an effective basis forrobot cognition requires some form of innate knowl-edge; see (Henderson et al., 2009). In fact, mostcurrent robot implementations rely mostly on innateknowledge programmed in by the system builder(e.g., sensor data analysis algorithms, object recogni-tion, kinematics and dynamics, planning, navigation,etc.), although some aspects of the robot’s knowl-edge may be learned from experience. This ap-proach to establishing robot cognitive faculties doesnot scale well, and as a more effective and efficientparadigm, we have proposed that a collection of sym-metry theories form the basic innate knowledge ofan autonomous agent (Grupen and Henderson, 1988;Henderson et al., 2011; Henderson et al., 2012a; Hen-derson et al., 2012b). These are defined as formal,

logic-based systems, and then the robot proceeds touse this knowledge by analyzing sensorimotor data todiscover models for the theories. For example, theaxioms which define a group require a set of elementsand an operator that acts on the set such that the clo-sure, identity, inverse and associativity properties aresatisfied. Figure 1 shows how this works. Namely,

Figure 1: Models for Theories are Discovered: (1) A theoryis defined; (2) Sets and Operators are Hypothesized fromSensorimotor Data; (3) Axioms are Validated for Hypothe-ses; (4) Theory is Exploited.

sets and operators are found in the sensorimotor dataand checked to see if they satisfy the axioms. If so,then the robot may use the theory to test new asser-tions (i.e., theorems) that follow from the axioms andhave direct meaning for the model, but which maynever have been experienced by the robot.

We propose that in addition to a logical frame-work to allow assertions about the world, robot cog-nition requires a representational mechanism for sen-sorimotor experience. The wreath product providesthis; for a more detailed account of wreath products,see (Chang, 2004; Foote et al., 2000; Leyton, 2001).The wreath product is a group formed as a splittingextension of its normal subgroup (the direct prod-uct of some group) and a permutation group. As anexample, consider the wreath product representationof the outline of a square shape: {e} oZ2 oℜ oZ4,where {e} represents a point, Z2 is the mod 2 groupmeaning select the point or not, ℜ is the continu-ous (one-parameter Lie) translation group, Z4 is thecyclic group of order 4, and o is the wreath product.[Note that: ℜ oZ4 = ℜ×ℜ×ℜ×ℜoZ4 where o isthe semi-direct product.] Leyton (Leyton, 2001) de-scribes this representation in great detail and explainsthe meaning of the expression as follows:

1. {e}: represents a point. (More precisely, the ac-tion of the group consisting of just the identity act-ing on a set consisting of a point.)

2. {e} oZ2: represents the selection of a point or not;i.e., to get a line segment from a line, just selectthe points on the line segment by assigning the 1

Figure 2: Control Flow in a Wreath Product gives an Ex-plicit Definition in Terms of Actuation of how to Generatethe Shape; e, the group identity (rotate by 0 degrees) acts onthe first copy, ℜ1 to obtain the top of the square; r90 acts onℜ1 by rotating it 90 degrees about the center of the square toobtain the right side, ℜ2; r180 acts on ℜ1 by rotating it 180degrees to obtain ℜ3, and finally, r270 acts on ℜ1 rotating it270 degrees to obtain ℜ4.

element to them.

3. {e} oZ2 oℜ: represents a line segment; i.e., oneside of the square.

4. {e} oZ2 oℜ oZ4: represents the fact that each sideof the square is a rotated version of the base seg-ment; i.e., the four elements of Z4 represent rota-tions of 0, 90, 180 and 270 degrees.

Each new group to the right side of the wreath sym-bol defines a control action on the group to the left,and thus provides a description of how to generatethe shape. E.g., to draw a square, start at the speci-fied point (details of the x,y values of the point, etc.are left to annotations in the representation), translatesome length along a straight line, then rotate 90 de-grees and draw the next side, and repeat this two moretimes. Figure 2 shows how control flows from therotation group, Z4, down to copies of the translationgroup, ℜ.

Thus, taken together, innate theories allow the dis-covery of the basic algebraic entities (i.e., groups) ofthe wreath products, and wreath products allow thedescription of concepts of interest (here restricted toshape and structure). In the following sections, we ex-pand on the exploitation of sensorimotor data (actua-tion is crucial) in operationally defining wreath prod-ucts.

Finally, we propose a probabilistic frameworkfor characterizing the uncertainty in wreath products.The wreath product maps directly onto a Bayesiannetwork (BN) as follows:

1. The rightmost group of the wreath product formsthe root node of the BN. Its children are the directproduct group copies to the left of the semi-directproduct operator.

Figure 3: Bayesian Network for the Wreath Product of aSquare Created by Rotation.

2. Recursively find the children of each interior nodeby treating it as the rightmost group of the remain-ing wreath product.

For example, consider the square expressed as {e} oZ2 oℜ oZ4. Figure 3 shows the corresponding BN.

[Note that the continuous translation group ℜ callsfor an uncountable number of cross product groups(one for each real), but we do not implement thisexplicitly.] The following sections describe how theconditional probability tables can be determined fromfrequencies encountered in the world (e.g., the likeli-hood of squares versus other shapes), and experiencewith the sensors and actuators (e.g., the likelihood ofthe detection of the edge of a square given that thesquare is present in the scene).

2 ROBOT CONCEPTFORMATION

We now describe an architecture to allow object rep-resentations to be constructed from groups discoveredin sensorimotor data, and combined to form wreathproducts. Figure 4 gives a high-level view of the in-teraction of innate knowledge and sensorimotor datato produce derived knowledge. A lower level archi-tecture is given in Figure 5 involving long-term mem-ory, short-term memory, behavior selection and sen-sorimotor data. Cognition can be driven by the data(bottom-up) or by a template presented in the behav-ior selection unit (top-down). In the former case, sym-metries are detected in the data which gives rise tothe assertion of groups. Then the structure of thesegroups is determined to form wreath products.

Suppose that data from a single object (a square)

Figure 4: High-Level Cognitive Process.

Figure 5: Lower Level Layout of Cognitive Process.

is to be analyzed. Figure 6 shows a number of groupelements that can be discovered by symmetry detec-tors applied to image data of a square; this includestranslation, reflection and rotation symmetries; wehave described symmetry detectors for 1D, 2D and3D data elsewhere (Henderson et al., 2011; Hender-son et al., 2012a; Henderson et al., 2012b). The sym-

Figure 6: Symmetry Detection in Image of a Square.

Figure 7: (a) One Face View RGB (b) One Face View DepthMap (c) Three Face View RGB (d) Three Face View DepthMap

metry groups would allow the synthesis of ℜ oD4 (thisis the same as ℜ×ℜ×ℜ×ℜoD4) as a representa-tion of the data. On the other hand if the drawing ofthe square were observed, and it was done by startingwith the top most edge and drawing clockwise aroundthe square, then ℜ oZ4 would be generated. If first thetop edge were drawn, then the bottom edge, then theright, then the left, this would give rise to: Z2 oZ2Actuation used to observe the data can also be usedto determine the appropriate wreath product; e.g., ifmotors move the camera so that the center of the im-age follows the contour of the square, then this givesrise to a symmetry in the angles followed (one angleranges from 0 to 2π, while the other follows a periodictrajectory. [Also, note that all these wreath productscould have a pre-fixed {e} oZ2 which represents se-lecting the points on the line segment.]

Next consider the 3D cube. Just like the square,there are several wreath product representations, eachcorresponding to a distinct generative process. Actual3D data (e.g., from a Kinect), will most likely comefrom viewing one face of the cube or three faces.;Figure 7 shows an example Kinect image for thesetwo situations in the case of a cube-shaped footstool.For the 3-face view, each visible face has a corre-sponding hidden parallel face which can be viewedas either generated from the visible face by reflec-tion or rotation. The Z2 group represents either ofthese (in the specific encoding, we choose reflection).Figure 8(a)(Top) shows the cube viewed along a di-agonal axis through two corners of the cube, and theZ3 rotational symmetry can be seen; i.e., a rotation of120 degrees about the axis (K in Figure 8(a)(Bottom))sends each pair of parallel faces into another pair. Thewreath product for this is then ℜ2 oZ2 oZ3 while thetree structure shown in Figure 8(b) gives the controlaction tree.

3 BAYESIAN SYMMETRYNETWORKS

We have already described how the wreath productprovides a nested description of an object which maps

Figure 8: (a) View of Cube along Diagonal Axis with Z3Symmetries (b) the Control Action Tree for ℜ2 oZ2 oZ3

Figure 9: Bayesian Network for Cube.

in a natural way onto a causal graph structure for aBayesian network. As a working example, we con-tinue with the cube as described in Figure 8(b) forwhich the corresponding Bayesian network1 is shownin Figure 9. The graph structure is defined by thewreath product, and the conditional probability tablesare determined by considering the context (indoors),the sensor data noise, and the algorithm error. Thetop-most node Z3 has been assigned a prior of 5%true based on occurence statistics of the environment.The Z2 nodes have a probability of 30% if there isno known Z3 symmetry, otherwise 80%, and the ℜ2

flat face nodes have conditional probability of 70% ifno Z2 symmetry is known, otherwise 95%. The fig-ure shows the likelihoods of each symmetry assertionwith no evidence. Figure 10 shows the changes inprobabilities for the network when it is known that 3faces exist, and that there is a Z3 symmetry for them.Note that the likelihoods for the unseen parallel facesrise to 91% in this case. This type of information maynot be readily available to a robot without this cogni-

1This network was constructed using the AgenaRisksoftware which accompanies (Fenton and Neil, 2013).

Figure 10: Bayesian Network for Cube with ObservationsAsserted.

Figure 11: Kinect Cube Data, and Surface Normals.

tive structure.This works reasonably well in practice; Figure 11

shows Kinect data for a cube shape viewed along adiagonal and surface normals. To determine if the Z3symmetry exists, we determine the average normal foreach face, then check the symmetry of the three nor-mals rotated through 2π radians about their mean vec-tor. Figure 12(a) shows the similarity measure (bestmatch of three original normals with rotated versionsof themselves, while (b) shows the trajectories of thethree normal endpoints under the rotation.

4 CONCLUSIONS AND FUTUREWORK

Our position is that wreath product representations ofobjects provide a very powerful object concept mech-anism, especially when combined with deep connec-tions to sensorimotor data, tied to specific object de-scriptions, and embedded in a probabilistic inferenceframework. Current issues include:

0 100 200 300 4000.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Angle of Rotation about Mean Vector

Sim

ilarit

y of

3 N

orm

al D

irect

ions

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

X AxisY Axis

Z A

xis

Figure 12: (a) Similarity of 3 Surface Normals under Ro-tation about their Mean Vector; (b) Trajectories of NormalEndpoints under Rotation.

• General Concept Representation: Arbitrary ob-jects can be represented by use of tensor splinegroups, as well as shape modification processesas described by Leyton. Implementation will re-quire careful attention. In addition, Leyton arguedthat wreath products could represent any concept;therefore, extensions need to be found for struc-tural, material properties, social, and other typesof relations.

• Combining Sensor and Actuation Data: A de-tailed characterization of how sensorimotor data isembedded in the wreath product representation isrequired. There are limitations in what the actua-tors can achieve compared to what the sensors canperceive, depending on the robot used and its De-grees of Freedom (DoF). In the cube example, wementioned that a robot with a head having 360◦

3-DoF can actuate the concepts as motions. Sim-ilarly, a robot with a dextrous hand will be able tomanipulate the object and be able to map the ac-tuations to the concepts: e.g., in 2D the robot candraw the square, given the generative concept ofa square, while in 3D it can trace its fingers alongthe smooth faces of a cube and infer the objecttype based on the Bayesian network for a cube.On the contrary, a two-wheeled robot with only arange sensor and no movable head or hands canonly move on the ground, and will thus have lim-itations in relating actuations directly to concepts(especially 3D shape concepts).

• Prior and Conditional Probabilities: must bedetermined for the networks. This requires a rig-orous learning process for the statistics of the en-vironment and the sensors. For example, we are

Figure 13: Error Distribution in Plane-Fitting for KinectData.

studying the error in fitting planes to Kinect data,and first results indicate that the noise appearsGaussian (see Figure 13).

• Prime Factorization: In the shapes we address, aBayesian network can be created in multiple waysfor the same shape; e.g., two square representa-tions are ℜ oZ4 and ℜ oZ2×Z2 oZ2. The equiv-alence of resulting shape must be made knownto the agent (even thought the generative mech-anisms are different); we are exploring whethersubgroups of the largest group generating theshape allow this to be identified.

• Object Coherence and Segmentation: Objectsegmentation is a major challenge, and objectclassification processing will be more efficient ifrelated points are segmented early on. We arelooking at the use of local symmetries (e.g., color,texture, material properties, etc.) to achieve this.Moreover, object coherence can be found frommotion of the object; namely, there will be a sym-metry in the motion parameters for all parts ofa rigid object which can be learned from experi-ence.

REFERENCES

Binford, T., Levitt, T. and Mann, W. (1987). Bayesian Infer-ence in Model-Based Machine Vision. In Uncertaintyin Artificial Intelligence 3 Annual Conference on Un-certainty in Artificial Intelligence (UAI-87), pages 73–95, Amsterdam, NL. Elsevier Science.

Buisson, J.-C. (2001). A Computer Model of Interactivismand Piagetian Assimilation Applied to Visual Percep-tion. In Proc. in the 31th. annual symposium of theJean Piaget Society, Berkeley, CA.

Chang, W. (2004). Image Processing with Wreath Prod-ucts. Master’s thesis, Harvey Mudd College, Clare-mont, CA.

Fenton, N. and Neil, M. (2013). Risk Assessment and De-cision Analysis with Bayesian Networks. CRC Press,Boca Raton, FL.

Foote, R., Mirchandani, G., Rockmore, D., Healy, D., andOlson, T. (2000). A Wreath Product Group Approachto Signal and Image Processing: Part I – Multiresolu-tion Analysis IEEE-T Signal Processing, 48(1):102–132.

Gold, K. and Scassellati, B. (2007). A Bayesian Robot thatDistinguishes ”self” from ”other”. In Proceedings ofTwenty-ninth Annual Meeting of the Cognitive ScienceS ociety, pages 384–392, NJ. Lawrence Erbaum.

Grupen, R. and Henderson, T. (1988). Apparent Sym-metries in Range Data. Pattern Recognition Letters,7:107–111.

Henderson, T., Fan, Y., Alford, A., Grant, E., and Cohen, E.(2009). Innate Theories as a Basis for AutonomousMental Development. Technical Report UUCS-09-004, The University of Utah.

Henderson, T., Joshi, A., and Grant, E. (2012a). From Sen-sorimotor Data to Concepts: The Role of Symmetry.Technical Report UUCS-12-012, University of Utah,School of Computing, Salt Lake City, UT.

Henderson, T., Peng, H., Sikorski, C., Deshpande, N., andGrant, E. (2011). Symmetry: A Basis for Sensorimo-tor Reconstruction. Technical Report UUCS-11-011,University of Utah, School of Computing, Salt LakeCity, UT.

Henderson, T. C., Cohen, E., Grant, E., Draelos, M., andDeshpande, N. (2012b). Symmetry as a Basis for Per-ceptual Fusion. In Proceedings of the IEEE Confer-ence on Multisensor Fusion and Integration for Intel-ligent Systems, Hamburg, Germany.

Jia, Y., Abbott, J., Austerweil, J., Griffiths, T., and Darrell,T. (2013). Visual Concept Learning: Combining Ma-chine Vision and Bayesian Generalization on conceptHierarchies. In Neural Information Processing Sys-tems 2013, Lake Tahoe, NV.

Leyton, M. (2001). A Generative Theory of Shape.Springer, Berlin.

Tenenbaum, J. and Griffiths, T. (2001). Generalization,Similarity and Bayesian Inference. Behavioral andBrain Sciences, 24:629–640.

Vasudevan, S. and Siegwart, R. (2008). Bayesian SpaceConceptualization and Place Classification for Se-mantic Maps in Mobile Robots. Robotics and Au-tonomous Systems, 56:522–537.

Vernon, D., Metta, G., and Sandini, G. (2007). A Surveyof Artificial Cognitive Systems: Implications for theAutonomous Development of Mental Capabilities inComputational Agents. IEEE Transactions on Evolu-tionary Computation, 11(2):151–180.

Zhang, L. and Ji, Q. (2011). A Bayesian Network Model forAutomatic and Interactive Image Segmentation. IEEETransactions on Image Processing, 30(9):2582–2593.