Learning from the Wizard: Programming Social Interactionthrough Teleoperated Demonstrations

ABSTRACTThis paper considers the question of whether robots canbe effectively programmed for autonomous social interac-tion through learning from demonstration. We advocatefor learning from demonstrations recorded via Wizard-of-Ozteleoperation. First, we give an overview of this paradigmof learning from the wizard (LfW). Then, we present a novelLfW system for educational play between young children anda robot. We describe initial results from a randomized exper-iment comparing a teleoperated robot and an autonomousrobot whose behavior was derived by LfW. Across numer-ous metrics, the teleoperated robot and the autonomousrobot programmed by LfW elicit similar behavior from theirhuman interaction partners. Additionally, when childrenwere asked whether the robot was human-controlled or au-tonomous, approximately half in each condition thought itwas human-controlled. These results demonstrate that arobot programmed via LfW can successfully engage in highlysocial, interactive tasks with young children.

Categories and Subject DescriptorsI.2.9 [Artificial Intelligence]: Robotics

General TermsExperimentation,Algorithms

Keywordssocially assistive robots, learning from demonstration, Wiz-ard of Oz, autonomous social interaction

1. INTRODUCTIONIn human-computer interaction research, a Wizard-of-Oz

scenario involves a human interacting with some machineinterface—such as a socially expressive robot—that is se-cretly controlled by a human teleoperator. We define learn-ing from the wizard (LfW) as a subtype of learning fromdemonstration in which the training demonstrations are de-rived from records of Wizard-of-Oz interaction and are usedto learn policies for interacting autonomously with humans.Previous work on LfW exists but until now has lacked a

Appears in: Proceedings of the 15th International Confer-ence on Autonomous Agents and Multiagent Systems (AA-MAS 2016), John Thangarajah, Karl Tuyls, Stacy Marsella,Catholijn Jonker (eds.), May 9–13, 2016, Singapore.Copyright c© 2016, International Foundation for Autonomous Agents andMultiagent Systems (www.ifaamas.org). All rights reserved.

unifying name and has not been extended to the field ofHuman-Robot Interaction (HRI); we situate the contribu-tions of this paper amongst past work in Section 2.

This paper focuses on validating LfW by answering whetherlearning from the wizard can program effective behavior forsocial interaction and whether that behavior is judged sim-ilarly to Wizard-of-Oz behavior by human interaction part-ners. To this end, we conducted an evaluation of robot LfWin an educational domain with young children.

2. RELATED WORKMuch of the research involving learning socially interac-

tive behavior from demonstration has been conducted withsoftware agents, particularly with dialog agents. Learningfrom the wizard has been used to develop dialog systemsthat act as tour guides [?], provide directions to drivers [?],and decide whether to provide information on a screen orverbally in an in-car music player [?]. This pioneering workcontains a wide range of methods and evaluations. Noneof the work, however, reports quantitative results with thelearned agent acting with full autonomy. For example, insome cases, evaluations focused on prediction accuracy [?,?] on Wizard-of-Oz data, which is informative but not asubstitute for testing the learned system in its own interac-tion, where it may lead to new situations that are seldomencountered during Wizard-of-Oz control.

3. LfW FOR AN EDUCATIONAL DOMAINTo explore the potential of LfW for social tasks, we devel-

oped a robotic learning companion for young children. Thissection describes the educational domain and integrated sys-tem.

Figure 1 shows the interaction setting. Children aged 4–8sat across from a squash-and-stretch robot that resembles asmall plush toy and was designed for social interaction withyoung children. Between the robot and child was a tabletwith an educational app (also shown in Figure 1) designedto teach color-mixing and reading concepts to young chil-dren. Children could tap on leaves to add color to the bird-bath, where it mixes with previously added colors, and towash the bath clean, and they could touch the words to hearthem read aloud. Adding multiple pigments to the birdbathcan create secondary colors and brown; a video of app inter-action can be viewed at https://www.youtube.com/user/

LearningFromWizard.

3.1 General system

Seat+for+child+

Seat+for+observing+caregiver+

MS+Kinect+(face'tracking)+

Webcam+on+robot+

Microphone+and+webcam+on+child+

Educa=onal+app+

Figure 1: Clockwise from left: Interaction setting; experimenter interface; color-mixing app, from the Tinkrbook e-book.

The system sensors include 2 webcams (one pointed atthe child, another at the robot), a Microsoft Kinect, a mi-crophone, and the tablet itself. The tablet reports the startand end of child touches by their locations and times. It alsoreports time-stamped app events, such as triggering pigmentto fall or a word of text to be said aloud.

The final system, used during the experiment, has 30robot actions. 15 are simple emotional utterances pairedwith expressive motions such as an excited “Yeah!” pairedwith an quick up-and-down motion, an impatient“Ooh ooh!”,or a whimper. 5 actions cause the robot to lean towards thetablet and trigger app events. These app events are: adding1 of 3 primary colors, washing the birdbath clean, or tappingthe tub. 9 actions are prompts to the child. 8 of these askfor a possible color in an excited and anticipatory tone (e.g.,“Brown?!”, “Red?!”, or “White?!”), and the 9th is requesting“Now you go.” A recording of the robot performing eachaction can be viewed at https://www.youtube.com/user/

LearningFromWizard.During each interaction, the various system components

communicated via ROS. Time-stamped logs in the form ofrosbags were recorded that contained video from the twowebcams, app events (human- or robot-triggered), estimatesof the child’s face location in 3D space (derived from theKinect), microphone audio, and robot actions.

3.2 Autonomy by learning from the wizardIn this section, we give a brief overview of how we struc-

tured teleoperation, training, and learned autonomous be-havior. The robot’s actions are exclusively determined eitherby a teleoperator during Wizard-of-Oz interaction or by alearned policy during autonomous interaction, never by amixture of the two.

To create the autonomous behavior model, a set of fea-tures (e.g., the current color of the birdbath, the child’s androbot’s last actions, time since last child/robot action, etc.)are computed from logs of teleoperated data during learning.

Our goal is to learn a model that answers: given a fea-ture vector f drawn from sensory history at time t, for eachaction a, what is the probability π(f , a) that a would havebeen triggered by a teleoperator within the previous 100ms?For this 30-class learning problem, we created a hierarchi-cal model using binary logistic regression (from Weka) as abase learner, with L2 regularization applied with a scalingparameter of 10 to account for the large number of features.

During autonomous interaction, these same features arecomputed at the start of each 100 ms time step, and a singleaction is chosen and executed by sampling from the model’soutput distribution over actions, π(f , ·) .

4. STUDY DESIGNData from 37 participants’ interactions with a teleoper-

ated robot over a 4-day period formed the training data.A separate group of 85 participants took part in the finalstage, a randomized experiment with three conditions, dif-fering in what the child played with: tablet-only, a controlcondition without the robot; WoZ-experiment, with theteleoperated robot; and autonomy, with the robot actingaccording to its learned autonomous model.

5. RESULTS AND DISCUSSIONThe main research question of this paper is whether LfW

can be used to program successful socially interactive be-havior. One way to answer that question is to assess howwell the learned behavior emulates the successful social be-havior of the teleoperator. For this assessment, we focus onkey behavioral metrics of the child’s interaction with eithera teleoperated robot or an autonomous robot acting accord-ing to the learned policy during a randomized experimentbetween learned autonomy and Wizard-of-Oz control.

5.1 Learned behaviorIn the authors’ subjective judgement, the robot largely

succeeded in learning to emulate the demonstrated interac-tion heuristics. For instance, the autonomous robot oftencelebrates when the child creates a robot-requested color.Videos of learned and Wizard-of-Oz behavior can be viewedat https://www.youtube.com/user/LearningFromWizard. Wedescribe objective, quantitative results in the following sec-tion.

5.2 Participants guess: teleoperated orautonomous

As a double-blind question, another experimenter asked,“When you were playing with [the robot], [the experimenter]was sitting at the computer behind you. Do you think hewas remote-controlling [the robot], telling him exactly whatto do, or do you think he was just starting and stoppingthe robot?” I.e., they were asked whether the robot was au-tonomous or teleoperated by a human. Through this ques-tion, the experiment serves as a variant of the Turing test [?],albeit social, embodied, and constrained to 30 possible ac-tions. Counting answers of “undecided” or “both” as half avote for each option, 52.8% of respondents in the auton-omy condition thought the robot was teleoperated. 47.2%in the WoZ-experiment condition thought it was teleoper-ated. In other words, the autonomous robot was judged ashuman-teleoperated marginally more often than the robotthat was actually human-teleoperated. A 2x2 Fisher’s exacttest results in p = 1.

Both behavioral metrics and the children’s guesses aboutwhether the robot was autonomous indicate that LfW didprogram social behavior that largely resembled Wizard-of-Oz teleoperation by the demonstrator and was successfulcompared to child-tablet interaction without a robot.

6. CONCLUSIONFrom the perspective of how to program autonomous so-

cially interactive robot behavior, learning from the wizardconstitutes a nascent approach that could be instrumentalin the much-needed transition of social robots from Wizard-of-Oz control towards autonomy. Evaluating LfW withina playful, educational domain, this paper provides the firstvalidation of the technique to program socially interactiverobot behavior.