Combined Kinesthetic and Simulated Interface for Teaching RobotMotion Models

Elizabeth Cha1, Klas Kronander1 and Aude Billard1

Abstract— The success of a Learning from Demonstrationsystem depends on the quality of the demonstrated data.Kinesthetic demonstrations are often assumed to be the bestmethod of providing demonstrations for manipulation tasks,however, there is little research to support this. In this work, weexplore the use of a simulated environment as an alternative toand in combination with kinesthetic demonstrations when usingan autonomous dynamical system to encode motion. We presentthe results of a user study comparing three demonstrationsinterfaces for a manipulation task on a KUKA LWR robot.

I. INTRODUCTION

For robots to be effectively used in home or serviceoriented roles, they must be able to adapt quickly andlearn new tasks. Traditional methods for programming robotsrequire knowledgeable, technically competent users [1]–[3].Thus, there is a need for methods which simplify robot pro-gramming such that users with little experience or technicalknowledge can enable robots to perform new tasks [1]–[5].

Learning from Demonstration (LfD) systems teach a robotnew skills using examples or expert demonstrations of ateacher performing a task [5], [6]. As the robot’s performanceis linked to the quality of demonstration, it puts much of theburden on the teacher. LfD systems have already been usedto teach robots a range of tasks [5], [7]. Methods for givingdemonstrations widely vary but are often customized to suitthe robot, task, and available resources [5], [6].

Recently, kinesthetic teaching has emerged as a popularmethod for teaching manipulation tasks [8], [9]. However,despite its popularity, it is unclear whether kinesthetic teach-ing is significantly better than providing demonstrations in asimulated environment. Each demonstration mode has knownadvantages as well as drawbacks. However, there is anabsence of work that directly compares these demonstrationmodes and their performance. Moreover, this hints at anotherinteresting research question: what happens when using acombination of these demonstration methods.

In this work, we explore the use of a combined interfacewhich allows for both kinesthetic and virtual demonstrations.In our proposed system, users demonstrate a task using arobot and a 2D virtual scene simulating the robot’s real-lifeworkspace. An abstract representation of the robot’s currentknowledge is displayed to the user in the form of streamlineswhich visualize the path that the robot would take whenstarting a trajectory from different parts of the workspaceand can, hence, aid the user in deciding whether additionaldemonstrations are needed and if so, where.

To encode the robot motion, we use an autonomousDynamical System (DS) based on locally modulated linear

1Elizabeth Cha, Klas Kronader, and Aude Billardare with the School of Engineering, Ecole Polytech-nique Federale de Lausanne, 1015 Lausanne, Switzerland,klas.kronander@epfl.ch,aude.billard@epfl.ch,

(a) (b)

Fig. 1: A user gives demonstrations to teach robot motion using the twointerfaces of our system: (a) a 7 DoF robot arm and (b) a simulatedenvironment embedded in a graphical user interface.

dynamics. This motion representation has several advantagessuch as a smooth generalization of the demonstrated motionas well as time-invariance which imply an inherent robust-ness to temporal and spatial perturbations as discussed indetail in [10] and [11]. In contrast to these works which usea parametric estimate of the DS based on Gaussian MixtureModels (GMM) and Gaussian Mixture Regression (GMR),we use a DS model based on locally reshaping an existingDS to locally align the vector field with velocity observed indemonstrations. This is achieved by estimating the rotationangle and speed scaling using Gaussian Process Regression(GPR) [?]. In this work, a simple form of incrementallearning is achieved by expanding the data set used forregression as new data arrives.

We performed a study aimed at comparing three demon-stration methods: kinesthetic, simulated and combined kines-thetic and simulated. Subjects were asked to teach a robot aset of 2D pick-and-place tasks using the different teaching in-terfaces. Our results indicate that when using an autonomousdynamical system, users are most successful when giv-ing demonstrations in the virtual environment. Furthermore,based on the results we propose several design guidelines forincremental motion learning algorithms aimed at lay users.

II. RELATED WORK

Recently, there has been much interest in making Learningfrom Demonstration methods usable for people with littletechnical experience [5], [6]. Current systems typically sup-port demonstrations given in either the real world [8], [12]or in a virtual environment [1], [2], [13].

In the real world, a teacher can give demonstrations usingtheir own body or the robot. When utilizing their own body,performing a task is natural and intuitive. However, mappingdemonstrations from the user to the robot creates a cor-respondence problem. Collecting these demonstrations also

requires a considerable amount of equipment to accuratelytrack and estimate the user’s pose [4].

Kinesthetic teaching overcomes several of these issues:the teacher physically manipulates the robot to provide ademonstration recorded by the robot’s sensors. Hence, thereis no need for additional equipment and no correspondenceproblem [5], [8]. Manipulating the robot also affords usersfirst-hand knowledge of the robot’s kinematics and con-straints and restricts demonstrations to the robot’s usableworkspace, improving the quality of the demonstrations [5].

However, despite the popularity of kinesthetic teaching,many users struggle to provide good demonstrations dueto their inexperience manipulating complex, high degree-of-freedom robots [8]. Providing many demonstrations can alsobe physically tiring, making it difficult to teach the robotcomplex tasks.

Users can also give demonstrations by teleoperating therobot [5], [14]. As with kinesthetic teaching, there is no cor-respondence problem and demonstrations are limited to therobot’s workspace. While teleoperation removes the physicalburden of manipulating the robot, it can be difficult tofind a suitable interface for simulatenously controlling manydegrees-of-freedom and systems often lack feedback [15].

Regardless of control mode, providing demonstrationsusing the robot is time-consuming. Demonstrations given ina virtual or simulated environment can be quick, allowingmany demonstrations to be given with relatively little timeor physical effort [1], [7], [16]. The virtual environment,however, fails to convey the robot’s kinematic constraints,often faces the correspondence problem, and is difficult tocontrol [5]. Some of these problems can be lessened withbetter input devices and additional features to aid the userwhen providing demonstrations : feedback can easily begiven to the teacher to help refine future demonstrations [4],[9], [15], [17].

Feedback is typically given either after a demonstrationfinishes or continuously during the demonstration. Duringthe demonstration, feedback is used to guide or adjust theremaining demonstration. Haptic feedback is often givenwhen users are physically manipulating or teleoperating therobot to make it easier to control [9]. Most systems, how-ever, give feedback by playing the completed demonstrationback to the teacher, so they can evaluate its success [8].This, however, fails to show the full extent of the robot’sknowledge and many playbacks are time consuming. Un-fortunately, concisely representing a robot’s knowledge andstate to everyday users is challenging. Moreover, as mostLfD systems utilize different learning algorithms, it is unclearwhat demonstration method and feedback are most helpful.

III. MOTION LEARNINGIn this work, we are concerned with acquiring a kine-

matic task plan from demonstration. We consider a 2dworkspace, in which the demonstrator can provide examplesof trajectories starting from any point in the plane. Thesedemonstrations are then used to update a task model in theform of a first order Dynamical System (DS). The robot thenuses this DS to generate trajectories on its own.

Let x ∈ R2 represent a 2-dimensional position of robotend-effector in a plane. Consider the following DS:

x = f(x) = νAx

‖Ax‖, A =

[−10 00 −10

](1a)

(a) (b)

Fig. 2: (a) Original dynamics. (b) Examples of reshaped dynamics resultingfrom demonstrated trajectories (red). The predictive influence of the GP isindicated by the shaded area.

ν = min(ν, ‖Ax‖) (1b)

This is a linear isotropic system with truncated speed givenby the parameter ν. Any trajectory generated by this systemends up in the origin. Convergence to different target loca-tions are easily achieved by change of variables. We willrefer to Eq. (1) as the original dynamics.

The system in Eq. (1) serves only as a baseline forensuring that all trajectories reach the target. In order toreshape the system so that it produces trajectories similarto those demonstrated, a local modulation is introducedresulting in a new system termed reshaped dynamics.

x = g(x) =M(x)f(x) (2)

where M(x) ∈ R2×2 is a continuous matrix valued functionthat modulates the original dynamics by rotation and speed-scaling:

M(x) = (1 + κ(x))R(x) (3)

Here, κ(x) is a continuous state-dependent scalar functionstrictly larger than −1 and R(x) ∈ R2×2 is a state-dependentrotation matrix. Note that since M(x) has full rank, thereshaped dynamics will have the same equilibrium pointsas the original dynamics. Similar full-rank modulations ofDS have previously been used to steer trajectories away fromobstacles [18]. If M is chosen such that it is locally active1, itcan be shown that the reshaped dynamics is a stable DS withthe same single equilibrium point as the original dynamics.

In order to learn reshaped dynamics, we parameterize themodulation function M(x) = M(θ(x)) with θ(x) ∈ RP

being a vector of P = 2 parameters: rotation angle φ andspeed scaling κ. For a trajectory data set {xm, xm}Mm=1, acorresponding set {xm, θm}Mm=1 is generated by computingthe rotation angle and speed scaling between f(xm) and xmfor each m = 1 . . .M . After this step, GPR is used to learnthe parameter vector as a function of the robot position. Thisresults in a system in which all trajectories are guaranteedto converge to the target, and aligns with the demonstratedtrajectories in the areas where demonstrations are provided.Examples of reshaped systems are shown in Fig.2. Thegeneralization around the demonstrations is determined bychoice of covariance function used for GPR.In this work,we used a squared exponential covariance function with acharacteristic length scale of 0.2.

IV. DEMONSTRATION METHODS

We explore two different methods for providing demon-strations: kinesthetic demonstrations and virtual demonstra-

1M(x) is said to be locally active if M(x) 6= I only locally in the taskspace.

(a) (b)

Fig. 3: (a) The end effector of the robot is manipulated in the markedplane (green axes) to provide a kinesthetic demonstration., (b) The graphicalinterface shows a tabletop view of the scene with virtual demonstrations(red), kinesthetic demonstrations (purple), and streamlines of the DS.

tions. In this section, we detail the interface and method forgiving each type of demonstration.

A. Kinesthetic DemonstrationsWe use a 7 DoF KUKA LWR robot arm placed in gravity

compensation mode allowing users to physically manipulatethe robot to provide a continuous trajectory (position andvelocity) demonstration. The arm is also controlled to main-tain a fixed orientation for the end-effector. Although thedemonstrator can move the end-effector in Cartesian space,only planar motion is recorded (green axes in Fig.3a right).

The user initiates a new demonstration by saying “start.”Once they receive confirmation, the user moves the arm tocreate a trajectory. When they are finished, the user says“stop” to end the demonstration.

B. Virtual DemonstrationsThe virtual demonstration interface consists of a simulated

2D environment embedded into a graphical user interface.The simulated environment shows a static overhead imageof the real-life scene next to the real robot. Overlaid on thescene are streamlines showing the 2D dynamical system.

A standard computer mouse is used to give demonstrationsin the virtual environment. To start a demonstration, theuser positions the cursor inside the simulated environmentat the starting point of the trajectory and holds down theright mouse button. After drawing a 2D trajectory in theenvironment, the user releases the right mouse button to endthe demonstration. Virtual demonstrations include both theposition and velocity of the trajectory.

C. Graphical InterfaceBoth demonstration methods rely on the graphical inter-

face containing the virtual demonstration environment. Whena demonstration is given, the trajectory is drawn in the sim-ulated environment in purple for kinesthetic demonstrationsand red for virtual demonstrations, see Fig.3b.

The simulated environment also shows the 2D dynamicalsystem. Although the background image of the environmentis static, the streamlines of the dynamical system are updatedeach time a new demonstration is given. The real robot’s endeffector position is shown by an orange dot on the screen.

The graphical interface also contains several buttons forfeatures of the system: 1) to start and stop taking a kinestheticdemonstration, 2) to play trajectories testing the effect of a

demonstration on the dynamical system (on either the robotor in simulation) and 3) to undo the last demonstration.

Undo was an additional feature that allowed user toremove the most recent demonstration. The feature wasintended for when an error occurs while providing a demon-stration or if a user is unsatisfied with the results of theirdemonstration. Users were only able to remove the mostrecent demonstration, but they could use the feature multipletimes to remove the last set of demonstrations.

V. EXPERIMENTWe carried out an experiment looking at 3 research ques-

tions: 1) Do users prefer a demonstration method, 2) Is onedemonstration method more successful with the algorithmused by our system and 3) What are the benefits to acombined system using both demonstration methods.

A. ConditionsWe look at three different conditions:1) Kinesthetic (K): Users provide demonstrations only

by manipulating the robot. They are unable to seethe graphical interface including their previous demon-strations and the DS. The user verbally signals theresearcher to start and stops the demonstration, undoprior demonstrations, and test trajectories on the robot.

2) Simulated (S): Users provide demonstrations only inthe simulated environment using a computer mouse.They are able to see their previous demonstrations andthe updated DS, undo previous trajectories, and testtrajectories in simulation or on the robot.

3) Combined (C): Users are allowed to provide demon-strations using the robot or the simulated environment.All features of the interface are available to use.

We use a within-subjects design where each participant isexposed to all 3 conditions. The ordering of the conditionswas counterbalanced to negate any ordering effects.

B. SubjectsWe recruited 14 participants (4 females and 10 males)

from the Lausanne area. Participants ranged in age from 23to 55 (M = 30, SD = 7.97) years old. Participants werecompensated 15 CHF for successful completion of the study.On average, participants rated their familiarity with robots as2.9 (SD = 1.8) and their familiarity with dynamical systemsas 2.0 (SD = 1.8) on a 7-point Likert scale.

C. Experimental Protocol1) Introduction: At the start of the study, subjects were

told they were going to teach the robot new tasks. They werethen given a short overview of the experiment which consistsof: a training, 3 trials, and a post experiment interview.

2) Training: Participants were then given a training ses-sion to learn to use the system. During the training, the robotwas set up in front of a table with a path marked by greenpaper. Users were told to teach the robot to move only withinthe green area. No matter where along the path the robotstarted, they were told it should reach the end of the path(the attractor), without exiting the green (Fig.4).

During the training, participants used both the robot andsimulated environment to give demonstrations and tried outall other features of the interface. If the researcher noticedthe subject was struggling with the system, they would

START

GOAL

Fig. 4: In the training, users are told to always keep the robot in the greenarea while moving from the start to the goal.

offer participants some advice. Once the subject was able toadequately complete the task and had used all features anddemonstration methods, the training session was complete.

3) Trials: Participants completed one trial for each con-dition for a total of three trials: kinesthetic, simulated andcombined. Each trial contained a scene consisting of 3 cups,3 obstacles and a goal. Participants were instructed to teachthe robot to move each of the cups into the goal, a basket, toclear the table. Unlike the training, subjects were not givena set path the cups had to take. Instead, they were told therobot could take any path as long as it did not bump intoany of the 3 obstacles.

(a) (b) (c)

Fig. 5: Each trial consisted of a scene with 3 cups, 3 obstacles (cerealboxes), and a goal, the basket. Participants attempted to move each of thecups to the basket without hitting an obstacle.

Depending on the trial condition, participants were limitedin the type of demonstrations they could provide and whatinterface features they were allowed to use. In order to verifythey had completed the task, subjects had to test the systemby playing trajectories on the physical robot. If all cups wereput in the basket without touching any obstacles or fallingoff the table, the trial was considered a success. Participantschose when and how many times to play the test trajectories.The trial ended either when all 3 objects were successfullymoved to the goal or when ten minutes elapsed. After eachtrial ended, participants were given a short survey while thenext trial was set up.

4) Interview: After the experiment, we interviewed eachparticipant. The first half of the interview asked at partici-pants background: general demographic information such astheir age and employment and their level of prior experiencewith robots and other components of the system. The secondhalf looked at participants’ preferences regarding the sys-tem. We asked which demonstration interface they preferredoverall, in the long and short term, and why. We also askedparticipants to talk about the advantages and disadvantagesof each of the demonstration conditions.

D. Measures1) Objective MeasuresDemonstration: For each demonstration given, we have anumber of measures:

• position and velocity of each datapoint in the demonstration• demonstration method: kinesthetic or simulated• time: the duration of each demonstration• length: number of datapoints in the demonstration• undo: whether the demonstration was kept in the trial

Success: Each trial is scored out of 3, based on the numberof cups that the robot successfully moves to the goal withouthitting any obstacles or falling off the table.

2) Subjective MeasuresPost-Trial Questionnaire: After each trial, users answered8 questions taken from the NASA Task Load Index (NASA-TLX) [19] and the System Usability Scale (SUS) [20] on theappropriate 7-point Likert scale.

1) How mentally demanding was the task?2) How physically demanding was the task?3) How successful were you in accomplishing the task?4) How insecure, discouraged, irritated, stressed and annoyed

were you?5) I thought the system was easy to use.6) I think that I would need the support of a technical person

to be able to use this system.7) I would imagine that most people would learn to use this

system quickly.8) I felt very confident using this system.

Interview: Each participant was interviewed about their pref-erences of demonstration method including the advantagesand disadvantages of each.

VI. RESULTSEach of the 14 participants performed 3 trials (3 tasks per

trial, 42 total trials) spread equally across the 3 conditions.

A. Objective MeasuresSuccess: Participants were most successful in the simulatedcondition (F (1, 42) = 5.06, p = 0.01): 100% of userssuccessfully completed the trial in the simulated condition,whereas, only 57.1% and 85.7% of users were successful inthe kinesthetic and combined conditions.

Failures occured if users: 1) stopped in the middle ofthe demonstration, 2) changed directions during a demon-stration, or 3) gave demonstrations in opposite directions.These demonstrations are not handled well as the DS, whenpresented with contradictory information, can result in unex-pected trajectories from the users perspective. These failuresusually occured when using the robot to give demonstrations.When users could not see the DS, they were unable to correctthis behavior as it often required a demonstration to be givenin a different area than where the robot made a mistake.

Fig. 6: The middle demonstration is an example of a demonstration that isnot handled well by the dynamical system.

Demonstrations: Users gave the most demonstrations inthe simulated condition (Fig.7): as users could see the DS,they would give several demonstrations before playing thetest trajectories. The number of demonstrations given in thekinesthetic condition was increased by users who failed thetrial as they continued giving new demonstrations until 10minutes had elapsed.

0

20

40

60

80

100

120

140

Condition

Simulated

Kinesthetic

Kinesthetic Simulated Combined 0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

User

Simulated

KinestheticUnsuccessful

Nu

mb

er o

f D

emo

nst

rati

on

s

Fig. 7: The number of kinesthetic and simulated demonstrations given foreach condition and a breakdown of the demonstration method used in thecombined condition for each user.

Overall, users took significantly longer to give a kines-thetic demonstration than a demonstration in the simulatedenvironment (F (1, 309) = 6.30, p = 0.002) , Fig.8. This isunsurprising, as it is expected that manipulating a physicalrobot takes more time than moving a computer mouse. Thisis also the case when looking at the average number ofdatapoints in a demonstration, as this value depends on time(F (1, 309) = 44.54, p < 0.001).

13.72

7.45

0

2

4

6

8

10

12

14

Kinesthetic Simulated

Av

erag

e D

emo

nst

rati

on

Du

rati

on

(se

c)

(a)

85.55

39.49

0

10

20

30

40

50

60

70

80

90

Kinesthetic SimulatedAv

erag

e D

emo

nst

rati

on

Len

gth

(d

atap

oin

ts)

(b)

Fig. 8: The average (a) duration (seconds) and (b) length (number ofdatapoints) for each demonstration method.

Participants had the opportunity to use the undo feature toremove prior demonstrations from the system. Participantskept approximately the same percentage of demonstrationsacross both demonstration methods. We observed that whenusing either demonstration method, participants rarely choseto remove a demonstration that had a poor result on thedynamical system. Instead, they preferred to give moredemonstrations to fix the system. In condition 1, participantswere unable to see the interface, including their previousdemonstrations, so undo was only used when a miscom-munication occurred between the researcher and participantresulting in an erroneous demonstration.

B. Subjective MeasuresPost-Trial Ratings: After each trial, participants were given8 questions from the NASA Task Load Index (NASA-TLX)

Demonstration Type Total Kept Percentage KeptKinesthetic 197 172 87.31%Simulated 239 209 87.45%Total 436 381 87.39%

and the System Usability Scale (SUS) measured on a 7-pointLikert scale.

We found mariginally significant results for only our ques-tions on success and ease of use. From our interviews withparticipants, we found 2 reasons for this. Since participantsonly had a small amount of time to learn to use the system,they were given easy tasks and consequently, tended to ratethe systems highly. Participants also commented they beinginvolved in a study with a robot was “really fun” and “cool.”This novelty effect also seemed to raise their overall ratings.

We expected the kinesthetic condition to be rated the mostphysically demanding and the virtual demonstration as themost mentally demanding. However, participants rated thekinesthetic condition as the most mentally demanding. Theircomments indicated this was due to the difficulty in manip-ulating the robot. Participants rated the combined conditionas most physically demanding which may be because usingboth demonstration methods requires users to both physicallymanipulate the robot and walk between the two systems.

Overall, participants rated the simulated condition as themost successful and easy to use. This corresponds withthe success scores as every participant succeeded in thesimulated condition. However despite, performing on average28.6% worse in the kinesthetic trial, participants surprisinglyrated their success and ease of use of the combined andkinesthetic conditions as equal. Their ratings for frustrationalso corresponded to success and ease of use with thesimulated condition rated as the least frustrating.

We expected that most participants would be inexperi-enced using a robot arm and would thus, be the most appre-hensive with the kinesthetic interface. However, paticipantsrated their confidence the lowest, and the technical knowl-edge needed to use the combined system the highest. But,participants did feel they could learn to use the combinedsystem the fastest, indicating that while initially daunting,users quickly acclimated to the system.

Comments from the participants indicated that while itfelt natural to give a demonstration on the robot, it would bedifficult to understand how their demonstration affected thesystem without seeing the simulated environment. This indi-cates that users associated the greatest challenge with usingthe dynamical system rather than using the demonstrationinterfaces.Interview: After participants completed the experiment, weinterviewed them about their preferences and thoughts aboutthe system.

Overall, participants preferred using the robot to givedemonstrations in the short-term. They explained this wasdue to the robot being new and fun to use. On the other hand,participants preferred to use either the simulated environmentor the combined system if they had to use the system forsome time.

Participants felt that using the robot gave them a betteridea of the geometry of the objects than using the simulatedenvironment. They commented that when using the simulatedenvironment they were unsure whether the robot would beable to fit through narrow spaces as simulated demonstrations

1

2

3

4

5

6

7

Mental Demand Physical Demand Success Frustration Ease of Use Techincal

Knowledge

Learn Quickly Confidence

Av

erag

e L

iker

t R

atin

g

Kinesthetic

Simulated

Combined

F(1,42)= 2.5, p=0.10 F(1,42)= 2.4, p=0.10

Fig. 9: Average likert ratings for the post-trial questionnaire.

were only a line.On the other hand, participants felt that it was easier to

give accurate and precise demonstrations in the simulatedenvironment. They also said it was less physically tiring thanmoving the robot.

All but one participant said they found seeing the simu-lated environment and the streamlines to be helpful. Despitenot knowing what a dynamical system was, all participantsfelt that they understood the effects of their demonstrationson the streamlines. Several commented that seeing the en-vironment was especially helpful when learning to use thesystem or with difficult tasks.

VII. CONCLUSION

In this work, we explored the use of virtual demonstrationas an alternative to and in combination with kinestheticdemonstrations in a LfD system using a dynamical systemfor robust motion representation. Our results indicate thatwhen using a DS, lay users provide better demonstrations ina virtual environment. We believe that this is largely due tousers’ lack of experience with DS, aid given by visualizingthe robot’s knowledge as streamlines, and the demonstrationmethod’s familiar interface (a computer mouse).

Although virtual demonstations in the 2D case performedwell, their success in higher dimensions is unclear as a virtualdemonstration interface becomes more complicated for theuser to interpret. A hybrid demonstration interface that useskinesthetic teaching to initialize the system and to find anappropriate plane of refinement would be an interestingextension. The virtual demonstration could subsequently beused for refining the motion in the plane resulting fromanalysis of the initial kinesthetic demonstrations.

In this work, subjects utilized different teaching inter-faces that provided data to a particular motion learning andrepresentation framework described in Section III. Severalinteresting findings from the data can serve as guidelinesfor future algorithm design. Subjects were reluctant to usingthe undo function and instead preferred to give additionaldemonstrations that they expected would eventually makethe system overcome an older, poor demonstration. Thisbehavior favors a learning algorithm with a forgetting factor,which was not implemented in the DS. Several subjectsalso chose to demonstrate several different paths from thesame starting point, and expected that the robot should beable to choose one of the paths. This often resulted in anintermediate between the demonstrations being followed bythe robot, since the GP tends to average two different valuesat the same input location.

ACKNOWLEDGMENTSThis work was supported by the Swiss National Centre of

Compentence in Research (NCCR) in Robotics.

REFERENCES

[1] J. Aleotti, S. Caselli, and M. Reggiani, “Toward Programming ofAssembly Tasks by Demonstration in Virtual Environments,” in IEEERO-MAN. IEEE, 2003, pp. 309–314.

[2] D. Brageul, S. Vukanovic, and B. A. MacDonald, “An IntuitiveInterface for a Cognitive Programming by Demonstration System,”in IEEE ICRA. IEEE, 2008, pp. 3570–3575.

[3] X. Wu and J. Kofman, “Human-Inspired Robot Task Learning fromHuman Teaching,” in IEEE ICRA. IEEE, 2008, pp. 3334–3339.

[4] J. Aleotti, S. Caselli, and M. Reggiani, “Leveraging on a virtualenvironment for robot programming by demonstration,” Robotics andAutonomous Systems, vol. 47, no. 2, pp. 153–161, 2004.

[5] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A Surveyof Robot Learning from Demonstration,” Robotics and AutonomousSystems, vol. 57, no. 5, pp. 469–483, 2009.

[6] A. Billard, S. Calinon, R. Dillmann, and S. Schaal, “Survey: RobotProgramming by Demonstration.” MIT Press, 2008.

[7] J. Aleotti and S. Caselli, “Grasp recognition in virtual reality for robotpregrasp planning by demonstration,” in IEEE ICRA. IEEE, 2006,pp. 2801–2806.

[8] B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz, “Trajectoriesand keyframes for kinesthetic teaching: A human-robot interactionperspective,” in ACM/IEEE HRI. ACM, 2012, pp. 391–398.

[9] S. Calinon, P. Evrard, E. Gribovskaya, A. Billard, and A. Kheddar,“Learning collaborative manipulation tasks by demonstration using ahaptic interface,” in ICAR. IEEE, 2009, pp. 1–6.

[10] E. Gribovskaya, S. M. Khansari-Zadeh, and A. Billard, “LearningNon-linear Multivariate Dynamics of Motion in Robotic Manipula-tors,” IJRR, pp. 1–37, 2010.

[11] S. M. Khansari-Zadeh and A. Billard, “Learning Stable NonlinearDynamical Systems With Gaussian Mixture Models,” IEEE T-RO,vol. 27, no. 5, pp. 943–957, 2011.

[12] S. Calinon and A. G. Billard, “What is the Teachers Role in RobotProgramming by Demonstration? Toward Benchmarks for ImprovedLearning,” Interaction Studies, vol. 8, no. 3, pp. 441–464, 2007.

[13] Y. Kunii and H. Hashimoto, “Tele-teaching by human demonstrationin virtual environment for robotic network system,” in IEEE ICRA,vol. 1. IEEE, 1997, pp. 405–410.

[14] J. D. Sweeney and R. Grupen, “A model of shared grasp affordancesfrom demonstration,” in IEEE-RAS Humanoids. IEEE, 2007, pp.27–35.

[15] J. Y. Chen, E. C. Haas, and M. J. Barnes, “Human performance issuesand user interface design for teleoperated robots,” IEEE Transactionson Systems, Man, and Cybernetics, vol. 37, no. 6, pp. 1231–1245,2007.

[16] J. Aleotti, S. Caselli, and M. Reggiani, “Evaluation of virtual fixturesfor a robot programming by demonstration interface,” IEEE Transac-tions on Systems, Man, and Cybernetics, vol. 35, no. 4, pp. 536–545,2005.

[17] A. L. Thomaz and C. Breazeal, “Teachable robots: Understandinghuman teaching behavior to build more effective robot learners,”Artificial Intelligence, vol. 172, no. 6, pp. 716–737, 2008.

[18] S. M. Khansari-Zadeh and A. Billard, “A dynamical system approachto realtime obstacle avoidance,” Autonomous Robots, vol. 32, no. 4,pp. 433–454, 2012.

[19] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task loadindex): Results of empirical and theoretical research,” Advances inpsychology, vol. 52, pp. 139–183, 1988.

[20] J. Brooke, “Sus-a quick and dirty usability scale,” Usability evaluationin industry, vol. 189, p. 194, 1996.