A Touchscreen-Based ‘Sandtray’ to Facilitate, Mediate andContextualise Human-Robot Social Interaction∗

Paul Baxter, Rachel Wood, Tony BelpaemeCentre for Robotics and Neural Systems

Plymouth University, U.K.{paul.baxter, rachel.wood, tony.belpaeme}@plymouth.ac.uk

ABSTRACTIn the development of companion robots capable of any-depth, long-term interaction, social scenarios enable explo-ration of the robot’s capacity to engage a human interactant.These scenarios are typically constrained to structured task-based interactions, to enable the quantification of results forthe comparison of differing experimental conditions. Thispaper introduces a hardware setup to facilitate and medi-ate human-robot social interaction, simplifying the robotcontrol task while enabling an equalised degree of environ-mental manipulation for the human and robot, but withoutimplicitly imposing an a priori interaction structure.

Categories and Subject DescriptorsH.1.2 [Models and Principles]: User/Machine Systems;J.4 [Computer Applications]: Social and Behavioral Sci-ences—Psychology

General TermsDesign, Experimentation, Human Factors, Theory

KeywordsSocial HRI, Sandtray, Unstructured Interactions

1. INTRODUCTIONIncreasingly, the view of robots as potential social com-

panions is coming to the fore [1]. These companion agentshave been proposed for, and applied to, a number of roles,such as support [5], motivation [4] and play within definedtask contexts. However, the more general aim of robot asgeneric (task independent) social peer remains at a distance.Such a companion agent would at the very least be requiredto operate in the real world over extended time-scales, mak-ing use of multiple modalities (both perceptual and physical)to engage the human interactant [6].

This paper presents the ‘Sandtray’, a newly developedplatform intended to aid in the exploration of issues in so-cial human-robot interaction. Social in this context goes fur-ther than directed interaction (e.g. control) or cooperation

∗This work is funded by the EU FP7 ALIZ-E project (grant248116), www.aliz-e.org/.

Copyright is held by the author/owner(s).HRI’12, March 5–8, 2012, Boston, Massachusetts, USA.ACM 978-1-4503-1063-5/12/03.

(e.g. to fulfil a specific objective), by additionally implyinga degree of task-independent engagement [6]. This platformallows emphasis to be placed on the social capacities of therobot rather than real-world implementation issues, whilststill facilitating the manipulation of the interaction context.We explore the role that such a setup can play in the facil-itation of social human-robot interaction, by providing aninteraction context without imposing an a priori structure.

2. THE SANDTRAY SETUPInspiration for this platform is drawn from the ‘Sandbox’

technique in child therapy where sand play is used to fos-ter collaborative story-telling interactions between child andtherapist [2]. The sandbox equipment itself is of incidentalimportance, acting purely as a medium to support the ther-apeutic interaction.

The interaction platform described here centres on use ofa touchscreen as opposed to a sandbox but similarly the in-tention is to bootstrap social engagement through use of acollaborative interaction platform. Thus in this setup thefocus is on social interaction rather than the efficacy of in-teractants’ task-oriented behaviour. The hardware consistsof a 26-inch capacitive touchscreen and associated controlserver, upon which a series of virtual objects (icons) canbe manipulated by dragging (on the part of the human in-teractant), or simulated dragging (on the part of the robotinteractant). The touchscreen thus serves as a medium forcollaboration (figure 1), rather than as a means of robotcontrol, for which they are typically used.

Figure 1: A Child-Robot interaction mediated bythe touchscreen-based Sandtray. In this example,there is a collaborative task to sort the icons intotwo categories (toys and food), by dragging theminto two pre-defined locations on the screen.

Session: LBR Highlights March 5–8, 2012, Boston, Massachusetts, USA

105

The control of this setup maintains a separation betweenthe handling of touchscreen and game functionality, and thecontrol of the robot behaviour, where the touchscreen canbe regarded as a ‘virtual modality’ for the robot (figure 2).In doing so, the state of the interaction medium (Sandtray)can be manipulated by the robot, without having to firstprocess visual information (to acquire the state of the Sand-tray) or coordinate fine motor control (to manipulate itsstate). Furthermore, this manipulation is similar in effectto that achievable by the human interactant (assuming thatthey have a certain level of competence using a touchscreen),resulting in an ‘equalisation’ of sensory and motor compe-tencies regarding the task-centred aspect of the interaction.This reduces the differences in affordances available to thetwo interactants, and thus mitigates the potential effect onthe social interaction.

In addition, there are a number of further advantages af-forded by such a setup involving virtual rather than realobject manipulation:

• Issues of robot manual dexterity and task-related per-ception do not have to be considered, reducing the incidentsof breaks in the flow of joint activity due to these difficulttasks: the sensing and acting competencies of the robot caninstead be focused on the human interactant.

• The context of the interaction can be manipulated, by,for example, changing the icon set displayed on the touch-screen.

• The properties of the interaction medium can be alteredon-the-fly (e.g. visual or auditory cues, modification of dragbehaviour, etc.), enabling the investigation of the effect ofdisturbances on the social interaction.

Figure 2: The control system for the Sandtray: per-ception of the touchscreen and actions regarding thetouchscreen by the robot can be handled as ‘virtual’sensory and motor modalities. In this scheme, Au-tonomous or Wizard-of-Oz (WoZ) robot control isreadily incorporated.

3. SUPPORTING SOCIAL INTERACTIONAs described above, the Sandtray can be used to provide

a context for the interaction by forming the medium uponwhich a task can be conducted. There are a number of tasksthat may be provided, and indeed a number of social rolesfor the robot with respect to the human interactant. So,for example, collaborative tasks may be set, or turn-basedgames (indeed, these would form informative manipulationsto the initial proposals here). However, as indicated above,our primary concern is the examination of social interactionbetween humans and robot (specifically in our applicationcontext of child-robot interaction [1]), and how it can beachieved, rather than an assessment of what robot capabil-ities result in improved joint task execution.

Consider the following: if a robot and person were simplysitting face to face in a location devoid of interaction context,a social exchange would be unlikely to spontaneously arise.In this case, either the robot or the person could choose asubject and initiate interaction, typically verbal - this invari-ably necessitates the robot taking the lead in a turn-basedconversation (given the technical difficulty of handling un-constrained speech) thereby implicitly imposing a structureon the interaction.

The importance for naturalistic social interactions in themaintenance of engagement is clear. One aspect to thisis that the imposition of an interaction structure from asource external to the interactants inhibits naturalistic in-teraction, leading to a reduction of engagement [3]. It istherefore desirable to avoid this. There is a potential con-flict with standard experimental methodology, which tendsto promote highly structured interactions, and may there-fore result in a reduction in the possible level of naturalisticinteraction. While aiming for implicitly unstructured inter-actions, the Sandtray setup partially mitigates any resultantdifficulty by providing a means to readily quantify the be-haviour of the human via the touchscreen (e.g. by providingdrag speed and direction information, etc).

The role of the Sandtray is thus to provide a collabora-tive context, but without necessarily also providing an overtstructure to the interaction, from which a social interactionis more likely to form. The point here is not that the Sand-tray is a sufficient condition for a social interaction, but thatthe presence of a context facilitates this process. Addition-ally, the emphasis on verbal interaction is reduced throughthe presence of a physical interaction medium (albeit vir-tual). The behaviour of the robot remains the key factorin determining whether or not a social interaction wouldform, whether this is the outcome of an autonomous controlsystem, or a Wizard-of-Oz manual control. Indeed, it maybe beneficial to ensure that the task is not engineered asthe primary source of engagement. Engagement between theinteractants would be required for the initiation and main-tenance of genuinely social interaction: a necessity for com-panion robots.

4. REFERENCES[1] P. Baxter, T. Belpaeme, L. Canamero, and et al.

Long-term human-robot interaction with young users.In Proceedings of the Workshop Robots with Children:Practices for Human-Robot Symbiosis at HRI’11,Lausanne, Switzerland, 2011.

[2] K. Bradway. Sandplay with children. Journal ofSandplay therapy, 8(2), 1999.

[3] R. R. Espinoza, M. Nalin, R. Wood, and et al.Child-robot interaction in the wild: Advice to theaspiring experimenter. In Proceedings of ICMI,Alicante, Spain, 2011.

[4] J. B. Janssen, C. C. van der Wal, M. A. Neerincx, andR. Looije. Motivating children to learn arithmetic withan adaptive robot game. In Proceedings of ICSR,Amsterdam, The Netherlands, 2011.

[5] C. Kidd. Designing for long-term human-robotinteraction and application to weight loss. PhD thesis,MIT, Massachusetts, USA, 2008.

[6] A. Tapus, M. J. Mataric, and B. Scassellati. The grandchallenges in socially assistive robotics. IEEE Roboticsand Automation Magazine, 14(1):35–42, 2007.

Session: LBR Highlights March 5–8, 2012, Boston, Massachusetts, USA

106