Machine Consciousness in CiceRobot, a Museum Guide Robot

Irene Macaluso and Antonio ChellaDipartimento di Ingegneria Informatica, Università di Palermo

Viale delle Scienze, 90128, Palermo, Italy

Abstract

The paper discusses a model of robot perceptual aware-ness based on a comparison process between the effec-tive and the expected robot input sensory data generatedby a 3D robot/environment simulator. The paper con-tributes to the machine consciousness research field bytesting the added value of robot perceptual awareness onan effective robot architecture implemented on an oper-ating autonomous robot RWI B21 offering guided toursat the Archaeological Museum of Agrigento, Italy.

IntroductionAn autonomous robot operating in real and unstructured en-vironments interacts with a dynamic world populated withobjects, people, and in general, other agents: people andagents may change their position and identity during time,while objects may be moved or dropped. In order to workproperly, the robot should be able to pay attention to rele-vant entities in the environment, to choose its own goals andmotivations, and to decide how to reach them. We claim thatthe robot, in order to properly move and act in complex anddynamic environment should have some form of perceptualawareness of the surrounding environment.

Taking into account several results from neuroscience,psychology and philosophy, summarized in the next Sect.,we hypothesize that at the basis of the robot perceptualawareness there is a continuous comparison process betweenthe expectation of the perceived scene obtained by a projec-tion of the 3D reconstruction of the scene, and the effectivescene coming from the sensory input. The paper contributesto the machine consciousness research field (Chella & Man-zotti 2007) by implementing and testing the proposed robotperceptual awareness model on an effective robot architec-ture implemented on an operating autonomous robot RWIB21 offering guided tours at the Archaeological Museum ofAgrigento (Fig. 1).

Theoretical remarksAnalyzing the perceptual awareness from an evolutionarypoint of view, (Humphrey 1992) makes a distinction be-tween sensations and perceptions. Sensations are active re-

Copyright c© 2007, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Figure 1: CiceRobot at the Archaeological Museum of Agri-gento.

sponses generated by the body in reaction to external stim-uli. They refers to the subject, they are about what is hap-pening to me. Perceptions are mental representations relatedto something outside the subject. They are about what ishappening out there. Sensations and perceptions are twoseparate channels; a possible interaction between the twochannels is that the perception channel may be recoded interms of sensations and compared with the effective stimulifrom the outside, in order to catch and avoid perceptual er-rors. This process is similar to the echoing back to sourcestrategy for error detection and correction.

(Gärdenfors 2004b) discusses the role of simulators re-lated to sensations and perceptions. He claims that sensa-tions are immediate sensory impressions, while perceptionsare built by simulators of the external world. A simulatorreceives as input the sensations coming from the externalworld, it fills the gaps and it may also add new informationin order to generate perceptions. The perception of an objectis therefore more rich and expressive than the correspondingsensation. In Gärdenfors terms, perceptions are sensationsthat are reinforced with simulations.

The role of simulators in motor control has been exten-sively analyzed from the neuroscience point of view, see(Wolpert, Doya, & Kawato 2003) for a review. In this line,

90

(Grush 2004) proposes several cognitive architectures basedon simulators (emulators in Grush terms). The basic archi-tecture is made up by a feedback loop connecting the con-troller, the plant to be controlled and a simulator of the plant.The loop is pseudo-closed in the sense that the feedback sig-nal is not directly generated by the plant, but by the simula-tor of the plant, which parallels the plant and it receives asinput the efferent copy of the control signal sent to the plant.

A more advanced architecture proposed by Grush and in-spired to the work of (Gerdes & Happee 1994) takes into ac-count the basic schema of the Kalman filter (Haykin 2001).In this case, the residual correction generated by the com-parison between the effective plant output and the simulatoroutput are sent again to the simulator via the Kalman gain.In turns, the simulator sends its inner variables as feedbackto the controller. According to (Gärdenfors 2004a), the sim-ulator inner variables are more expressive that rough plantoutputs and they may contain also information not directlyperceived by the system, as the occurring forces in the per-ceived scene, or the object-centred parameters, or the vari-ables employed in causal reasoning.

(Grush 1995) also discusses the adoption of neural net-works to learn the operations of the simulators, while (Oz-top, Wolpert, & Kawato 2005) propose more sophisticatedlearning techniques of simulators based on inferences of thetheory of mind of others.

The hypothesis that the content of perceptual awarenessis the output of a comparator system is in line with the Be-havioural Inhibition System (BIS) discussed by (Gray 1995)starting from neuropsychological analysis. From a neuro-logical point of view, (Llinas 2001) hypothesizes that theCNS is a reality-emulating system and the role of sensoryinput is to characterize the parameters of the emulation. Healso discusses (Llinas & Pare 1991) the role of this loop dur-ing dreaming activity.

An early implementation of a robot architecture based onsimulators is due to (Mel 1986). He proposed a simulatedrobot moving in an environment populated with simple 3Dobjects. The robot is controlled by a neural network thatlearns the aspects of the objects and their relationships withthe corresponding motor commands. It becomes able to sim-ulate and to generate expectations about the expected ob-ject views according to the motor commands; i.e., the robotlearns to generate expectations of the external environment.A successive system is MURPHY (Mel 1990) in which aneural network controls a robot arm. The system is able toperform off-line planning of the movements by means of alearned internal simulator of the environment.

Other early implementation of robots operating with in-ternal simulators of the external environment are: MetaToto(Stein 1991), and the internalized plan architecture (Payton1990). In both systems, a robot builds an inner model on theenvironment reactively explored by simulated sensorimotoractions in order to generate action plans.

An effective robot able to build an internal model of theenvironment has been proposed by (Holland & Goodman2003). The system is based on a neural network that con-trols a Khepera minirobot and it is able to build a model ofenvironment and to simulate perceptual activities in a sim-

Figure 2: The robot architecture.

plified environment. Following the same principles, (Hol-land, Knight, & Newcombe 2007) describe the successiverobot CRONOS, a complex anthropomimetic robot whoseoperations are controlled by SIMNOS, a 3D simulator of therobot itself and its environment.

Robot architectureThe robot architecture proposed in this paper is inspired bythe work of Grush previously presented and it is based onan internal 3D simulator of the robot and the environmentworld (Fig. 2). The Robot block is the robot itself and itis equipped with motors and a video camera. It is modelledas a block that receives in input the motor commands Mand it sends in output the robot sensor data S, i.e., the sceneacquired by the robot video camera.

The Controller block controls the actuators of the robotand it sends the motor commands M to the robot. The robotmoves according to M and its output is the 2D pixel matrixS corresponding to the scene image acquired by the robotvideo camera.

At the same time, an efferent copy of the motor commandsis sent to the 3D Robot/Environment simulator. The simu-lator is a 3D reconstruction of the robot environment withthe robot itself. It is an object-centred representation of theworld in the sense of (Marr 1982).

The simulator receives in input the controller motor com-mand M and it simulates the corresponding motion of therobot in the 3D simulated environment by generating a cloudof possible robot positions {xm} according to a suitableprobability distribution that takes into account the motorcommand, the noise, the faults of the controllers, the slip-pery of the wheels, and so on. For each possible positionxm, a 2D image S′m is generated as a projection of the sim-ulated scene acquired by the robot in the hypothesized posi-tion. Therefore, the output of the simulator is the set {S′m}of expected 2D images. Both S and S′m are viewer-centredrepresentations in Marr’s terms.

The acquired and the expected image scenes are thencompared by the comparator block c and the resulting errorε is sent back to the simulator to align the simulated robotwith the real robot (see Sect. ). At the same time, the simu-lator send back all the relevant 3D information P about therobot position and its environment to the controller, in orderto adjust the motor plans.

We claim that the perceptual awareness of the robot is theprocess based on the acquisition of the sensory image S, thegeneration of hypotheses {S′m} and their comparison via the

91

comparator block in order to build the vector P that containsall the relevant 3D information of what the robot perceivesat a given instant. In this sense, P is the full interpretationof robot sensory data by means of the 3D simulator.

Planning by expectationsThe proposed framework for robot perceptual awarenessmay be employed to allow the robot to imagine its ownsequences of actions (Gärdenfors 2007). In this perspec-tive, planning may be performed by taking advantage fromthe representations in the 3D robot/environment simulator.Note that we are not claiming that all kinds of planning mustbe performed within a simulator, but the forms of planningthat are more directly related to perceptual information cantake great advantage from perception in the described frame-work.

As previously stated, P is the perception of a situation ofthe world out there at time t. The simulator, by means of itssimulation engine based on expectations (see below), is ableto generate expectations of P at time t + 1, i.e., it is ableto simulate the robot action related with motor command Mgenerated by the controller and the relationship of the actionwith the external world.

It should be noticed that in the described framework, thepreconditions of an action can be simply verified by geo-metric inspections of P at time t, while in the STRIPS-likeplanners (Fikes & Nilsson 1971) the preconditions are veri-fied by means of logical inferences on symbolic assertions.Also the effects of an action are not described by adding ordeleting symbolic assertions, as in STRIPS, but they can beeasily described by the situation resulting from the expec-tations of the execution of the action itself in the simulator,i.e., by considering the expected perception P at time t + 1.

We take into account two main sources of expectations.On the one side, expectations are generated on the basis ofthe structural information stored in a symbolic KB which ispart of the simulator. We call linguistic such expectations.As soon as a situation is perceived which is the precondi-tion of a certain action, then the symbolic description elicitthe expectation of the effect situation, i.e., it generates theexpected perception P at time t + 1.

On the other side, expectations could also be generatedby a purely Hebbian association between situations. Sup-pose that the robot has learnt that when it sees somebodypointing on the right, it must turn in that direction. The sys-tem learns to associate these situations and to perform therelated action. We call associative this kind of expectations.

In order to explain the planning by expectation mecha-nism, let us suppose that the robot has perceived the currentsituation P0, e.g., it is in a certain position of a room. Letus suppose that the robot knows that its goal g is to be ina certain position of another room with a certain orienta-tion. A set of expected perceptions P1,P2, . . . of situationsis generated by means of the interaction of both the linguisticand the associative modalities described above. Each Pi inthis set can be recognized to be the effect of some action re-lated with a motor command Mj in a set of possible motorcommands M1,M2, . . . where each action (and the corre-

Figure 3: The map of the Sala Giove of the Museum.

sponding motor command) in the set is compatible with theperception P0 of the current situation.

The robot chooses a motor command Mj according tosome criteria; e.g., it may be the action whose expected ef-fect has the minimum Euclidean distance from the goal, or itmay be the action that maximizes the utility value of the ex-pected effect. Once that the action to be performed has beenchosen, the robot can imagine to execute it by simulatingits effects in the 3D simulator then it may update the situa-tion and restart the mechanism of generation of expectationsuntil the plan is complete and ready to be executed.

Linguistic expectations are the main source of delibera-tive robot plans: the imagination of the effect of an action isdriven by the description of the action in the simulator KB.This mechanism is similar to the selection of actions in de-liberative forward planners. Associative expectations are atthe basis of a more reactive form of planning: in this lat-ter case, perceived situations can immediately recall someexpected effect of an action.

Both modalities contribute to the full plan that is imag-ined by the robot when it simulates the plan by means of thesimulator. When the robot terminates the generation of theplan and of its actions, it can generate judgments about itsactions and, if necessary, imagine alternative possibilities.

Robot at workThe presented framework has been implemented in CiceR-obot, an autonomous robot RWI B21 equipped with sonar,laser rangefinder and a video camera mounted on a pan tilt.The robot has been employed as a museum tour guide op-erating at the Archaeological Museum of Agrigento, Italyoffering guided tours in the Sala Giove of the museum (Fig.3). A first session of experimentations, based on a previousversion of the architecture, has been carried out from Jan-uary to June 2005 and the results are described in (Chella,Frixione, & Gaglio 2005) and in (Macaluso et al. 2005).The second session, based on the architecture described inthis paper, started in March and ended in July 2006.

The task of museum guide is considered a significant casestudy (Burgard et al. 1999) because it concerns perception,self perception, planning and human-robot interactions. Thetask is therefore relevant as a test bed for robot perceptual

92

Figure 4: The object centered view from 3Drobot/environment simulator.

Figure 5: The viewer centered image from the robot point ofview.

awareness. It can be divided in many subtasks operating inparallel, and at the same time at the best of the robot capa-bilities.

Fig. 4 shows the object-centred view from the 3Drobot/environment simulator. As previously described, thetask of the block is to generate the expectations of the inter-actions between the robot and the environment at the basisof robot perceptual awareness. It should be noted that therobot also simulates itself and its relationships with its envi-ronment. Fig. 5 shows a 2D image generated from the 3Dsimulator from the robot point of view.

In order to keep the simulator aligned with the externalenvironment, the simulator engine is equipped with a parti-cle filter algorithm (Thrun, Burgard, & Fox 2005). As dis-cussed in the previous Sect., the simulator hypothesizes acloud {xm} of expected possible positions of the robot. Foreach expected position, the corresponding expected imagescene S′m is generated, as in Fig. 6 (left). The comparatorthus generates the error measure ε between each of the ex-pected images and the effective image scene S as in Fig. 6(right). The error ε weights the expected position under con-sideration; in subsequent steps, only the winning expectedpositions that received the higher weights are taken, whilethe other ones are dropped.

Fig. 7 (left) shows the initial distribution of expectedrobot position and Fig. 7 (right) shows the small cluster ofwinning positions. Now the simulator receives a new mo-tor command M related with the chosen action and, startingfrom the winning hypotheses, it generates a new set of hy-pothesized robot positions.

Figure 6: The 2D image output of the robot video camera(left) and the corresponding image generated by the simula-tor (right).

Figure 7: The operation of the particle filter. The initial dis-tribution of expected robot positions (left), and the cluster ofwinning expected positions.

To compute the importance weight ε of each hypothesis,we developed a measurement model that incorporates the3D representation provided by the simulator. The imagesS′m, generated by the simulator in correspondence to eachhypothesized robot position, and the image S, acquired bythe robot camera, are both processed to obtain the VE im-ages, in which vertical edges are outlined. For each VEimage a vector s is computed:

si =∑

j

VEij (1)

Each vector s can be considered as a set of samples drawnfrom an unknown distribution. To estimate such a proba-bility density function, we adopted the Parzen-window for-mula:

pk(x) =1|s|

∑i

1h

δ(x − si

h) (2)

where the window function δ is the normal distribution.Then we adopted the Kullback-Leibler distance:

d(p, ps) = −∫

p(x)lnps(x)p(x)

dx. (3)

as measure of the dissimilarity between the distribution pcorresponding to the image S acquired by the robot cameraand each of the distributions ps corresponding to the imagesS′m generated by the simulator.

93

(a)

(b)

(c)

Figure 8: (a) Image acquired by robot camera. (b) VerticalEdge image. (c) Distribution of the random variable repre-senting vertical edges.

The importance weight ε of the hypothesized robot posi-tion xm is the inverse of the Kullback-Leibler distance be-tween the image Sm generated in simulation and the realimage S acquired by the robot.

Fig. 8 (a) shows the image S obtained by the robot videocamera, the corresponding image VE (b), and the distribu-tion of the random variable representing the vertical edgescomputed according to Eq. 2 (c). Fig. 9 shows two exam-ple images generated by the simulator. Fig. 9 (a),(c) and(e) correspond to the hypothesis with the highest weight, i.e.the least Kullback-Leibler distance with respect to the cam-era generated distribution. Fig. 9(b),(d) and (f) correspondto an hypothesis with a lesser weight.

Fig. 10 shows the operation of the robot during the tourguide. The Fig. shows the map build up by state-of-art laser-based robotic algorithms. By comparing this Fig. with thereal map of the Sala Giove in Fig. 3, it should be noticedthat the museum armchairs are not visible to laser. However,thanks to the perceptual process, the robot is able to integratelaser data and video camera sensory data in a stable innermodel of the environment (Franklin 2005) and to correctlymove and act in the museum environment (Fig. 10).

(a) (b)

(c) (d)

(e) (f)

Figure 9: (a) The simulated image corresponding to the hy-pothesis with the highest weight. (b) A simulated image cor-responding to an hypothesis with a lesser weight. (c)-(d)Vertical Edge maps of the images shown in (a) and (b). (e)-(f) Distributions of the random variables representing verti-cal edges.

Figure 10: The operation of the robot equipped with the ar-chitecture.

94

Discussions and conclusionsThe described robot perceptual process is an active process,since it is based on a reconstruction of the inner percepts inegocentric coordinates, but it is also driven by the externalflow of information. It is the place in which a global consis-tency is checked between the internal model and the visualdata coming the sensors. Any discrepancy asks for a read-justment of its internal model.

The robot perceptual awareness is therefore based on astage in which two flows of information, the internal andthe external, compete for a consistent match. There astrong analogy with the phenomenology in human percep-tion: when one perceives the objects of a scene he actuallyexperiences only the surfaces that are in front of him, but atthe same time he builds an interpretation of the objects intheir whole shape.

We maintain that the proposed system is a good startingpoint to investigate robot phenomenology. As described inthe paper it should be remarked that a robot equipped withperceptual awareness performs complex tasks as museumtours, because of its inner stable perception of itself and ofits environment.

ReferencesBurgard, W.; Cremers, A.; Fox, D.; Hähnel, D.; Lake-meyer, G.; Schulz, D.; Steiner, W.; and Thrun, S. 1999.Experiences with and interactive museum tour-guide robot.Artif. Intell. 114:3–55.Chella, A., and Manzotti, R., eds. 2007. Artificial Con-sciousness. Exeter, UK: Imprint Academic.Chella, A.; Frixione, M.; and Gaglio, S. 2005. Planningby imagination in CiceRobot, a robot for museum tours. InProceedings of the AISB 2005 Symposium on Next Gener-ation approaches to Machine Consciousness: Imagination,Development, Intersubjectivity, and Embodiment, 40 – 49.Fikes, R., and Nilsson, N. 1971. STRIPS: a new approachto the application of theorem proving to problem solving.Artif. Intell. 2(3-4):189–208.Franklin, S. 2005. Evolutionary pressures and a stableworld for animals and robots: A commentary on merkel.Consciousness and Cognition 14:115 – 118.Gärdenfors, P. 2004a. Emulators as sources of hidden cog-nitive variables. Behavioral and Brain Sciences 27(3):403.Gärdenfors, P. 2004b. How Homo Became Sapiens: Onthe Evolution of Thinking. Oxford, UK: Oxford UniversityPress.Gärdenfors, P. 2007. Mind-reading as control theory. Eu-ropean Review 15(2):223 – 240.Gerdes, V., and Happee, R. 1994. The use of an inter-nal representation in fast goal-directed movements: a mod-elling approach. Biological Cybernetics 70(6):513 – 524.Gray, J. 1995. The contents of consciousness: A neu-ropsychological conjecture. Behavioral and Brain Sciences18(4):659 – 722.Grush, R. 1995. Emulation and Cognition. Ph.D. Disser-tation, University of California.

Grush, R. 2004. The emulator theory of representation:motor control, imagery and perception. Behavioral andBrain Sciences 27:377 – 442.Haykin, S., ed. 2001. Kalman Filtering and Neural Net-works. New York: J. Wiley and Sons.Holland, O., and Goodman, R. 2003. Robots with internalmodels - a route to machine consciousness? Journal ofConsciousness Studies 10(4 -5):77 – 109.Holland, O.; Knight, R.; and Newcombe, R. 2007. Arobot-based approach to machine consciousness. In Chella,A., and Manzotti, R., eds., Artificial Consciousness. Exeter,UK: Imprint Academic. 156 – 173.Humphrey, N. 1992. A history of mind. New York, NY:Springer-Verlag.Llinas, R., and Pare, D. 1991. Of dreaming and wakeful-ness. Neuroscience 44(3):521 – 535.Llinas, R. 2001. I of the Vortex: From Neurons to Self.Cambridge, MA: MIT Press.Macaluso, I.; Ardizzone, E.; Chella, A.; Cossentino, M.;Gentile, A.; Gradino, R.; Infantino, I.; Liotta, M.; Rizzo,R.; and Scardino, G. 2005. Experiences with CiceRobot,a museum guide cognitive robot. In Bandini, S., and Man-zoni, S., eds., AI*IA 2005, volume 3673 of Lecture Notesin Artificial Intelligence, 474 – 482. Berlin Heidelberg:Springer-Verlag.Marr, D. 1982. Vision. New York: W.H. Freeman and Co.Mel, B. 1986. A connectionist learning model for 3-Dmental rotation, zoom, and pan. In Proceedings of the EigthAnnual Conference of the Cognitive Science Society, 562 –571.Mel, B. 1990. Connectionist robot motion planning:A neurally-inspired approach to visually-guided reaching.Cambridge, MA: Academic Press.Oztop, E.; Wolpert, D.; and Kawato, M. 2005. Mental stateinference using visual control parameters. Cognitive BrainResearch 22:129 – 151.Payton, D. 1990. Internalized plans: A representationfor action resources. Robotics and Autonomous Systems6(1):89 – 103.Stein, L. 1991. Imagination and situated cognition. Tech-nical Report AIM-1277, MIT AI Lab.Thrun, S.; Burgard, W.; and Fox, D. 2005. ProbabilisticRobotics. Cambridge, MA: MIT Press.Wolpert, D.; Doya, K.; and Kawato, M. 2003. A unify-ing computational framework for motor control and socialinteraction. Phil. Trans. Roy. Soc. B 358:593–602.

95

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False

/CreateJDFFile false /Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice