Representing Skill Demonstrations for Adaptation and Transfer

Tesca Fitzgerald, Ashok K. Goel, Andrea ThomazSchool of Interactive Computing

Georgia Institute of Technology, Atlanta, GA 30332, USA{tesca.fitzgerald, goel, athomaz}@cc.gatech.edu

Abstract

We address two domains of skill transfer problems encoun-tered by an autonomous robot: within-domain adaptation andcross-domain transfer. Our aim is to provide skill representa-tions which enable transfer in each problem classification. Assuch, we explore two approaches to skill representation whichaddress each problem classification separately. The first rep-resentation, based on mimicking, encodes the full demonstra-tion and is well suited for within-domain adaptation. Thesecond representation is based on imitation and serves to en-code a set of key points along the trajectory, which representthe goal points most relevant to the successful completionof the skill. This representation enables both within-domainand cross-domain transfer. A planner is then applied to theseconstraints, generating a domain-specific trajectory which ad-dresses the transfer task.

IntroductionSkill transfer is necessary for autonomous robots to learnquickly and adapt to new situations. Without the abilityto transfer previously learned skills to a newly encounteredproblem, a robot would require additional time to learn thenew skill from scratch. Leveraging previously learned skillsto bootstrap the learning of new skills would significantly re-duce the amount of time necessary to learn a new skill, andwould require less effort of the skill teacher.

Additionally, skill transfer enables a robot to handle un-familiar problems with flexibility. When encountering anunfamiliar problem, the robot can attempt to transfer a previ-ously learned skill to address the new problem. We refer to a“target problem” as a new problem for which a skill demon-stration has not been provided. It is difficult, however, todefine what qualifies as a “transfer” problem. We refer totwo classifications of skill transfer problems (Thagard 2005;Goel 1997):

• Within-domain adaptation occurs when the robot trans-fers a known skill to a target problem which requires thesame goal and involves similar objects as the learned skill.The known skill’s action model must be modified to ad-dress the target problem. An example of a within-domainadaptation problem is a book-shelving task in which the

Copyright c© 2014, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

target problem involves a book with different dimensionsthan that of the training object. The known skill’s actionmodel can be used to address the new problem with onlyminor parameter adjustments.

• Cross-domain transfer involves transferring an actionmodel to a target problem in which the same goal mustbe attained, but a different set of objects entailing a differ-ent set of features are used. For example, stacking a set ofplates and stacking cups are two tasks which require thesame goal, but are completed by referencing two differentsets of features. Transferring the plate-stacking skill to thetarget problem may require more than a simple parameteradjustment, and may even require a different trajectory toachieve the new goal.

We focus on interactive skill transfer, where the robot istaught new skills via interactive demonstrations. We de-fine two skill representations that dictate what aspects of askill demonstration are stored and made available for trans-fer processes. Our purpose in this work is to consider whatrepresentations may enable a robot learner to go beyond sim-ple mimicking (skill adaptation) and get closer to true imita-tion (transfer).

BackgroundLearning from DemonstrationLearning from Demonstration (LfD) is an approach to skilllearning in which a skill is taught via interactive demonstra-tions provided by a human teacher (Argall et al. 2009). Asthe teacher does not directly program the robot, this modeof skill learning is particularly suited for end-users with-out robotics experience. Skill demonstrations can be pro-vided via several forms of interaction including kinestheticteaching and teleoperation (Argall et al. 2009). We focus onkinesthetic teaching, in which the teacher physically movesthe robot’s arm to demonstrate the skill (Akgun et al. 2012a).

There are several ways of providing a kinesthetic demon-stration. One method is to provide a single, uninterrupteddemonstration, during which the robot records its joint stateat regular intervals throughout the entire trajectory. In an-other method, the robot collects joint states only in sparseintervals specified by the teacher (Akgun et al. 2012b). In-stead of recording the entire motion, the robot records only

28

Knowledge, Skill, and Behavior Transfer in Autonomous Robots: Papers from the 2014 AAAI Fall Symposium

keyframes: robot poses highlighted by the teacher during theskill demonstration.

Social LearningWe approach skill adaptation and transfer using skillrepresentations inspired by the social learning paradigm.Tomasello describes several ways in which a learnercan respond to and represent interactive skill demonstra-tions (Tomasello 2001):

• Stimulus enhancement occurs when the learner is encour-aged to interact with an object after observing the teacherdrawing attention to the same object. This enables thelearner to focus on objects that the teacher determines tobe important or useful.

• Emulation learning involves a learner interacting with anobject after viewing the teacher demonstrate the object af-fordances. The learner copies the teacher’s actions to ex-plore new ways of interacting with the object.

• Mimicking occurs when the learner pays attention to theactions performed by the teacher, without regard for thegoal of the teacher’s actions.

• Imitative learning involves the learner paying attention tothe goals or intention behind the teacher’s actions.

We focus on building skill representations inspired by imi-tative learning. One interpretation of the difference betweenmimicking and imitation is the target of the learner’s atten-tion (Cakmak et al. 2010). By this interpretation, learningfrom imitation occurs when the learner represents an obser-vation as the intention behind that action (Tomasello 2001).Rather than directly copying the demonstrated actions ormethod used to achieve the goal, the learner may choose touse an alternative method that achieves the same goal. In arobotics context, this may involve a robot learner using a dif-ferent trajectory than what was demonstrated, provided thatthe demonstrated goal is still achieved by transferring theobserved constraints (Fitzgerald and Goel 2014). A skilllearned by imitation is likely to be represented computation-ally as a goal or set of sub-goals that are integral to the skill.It would be useful to represent the demonstrated skill as agoal-subgoal tree to learn by imitation.

In contrast, learning from mimicking involves replicatingthe teacher’s actions with emphasis on the method of com-pleting the skill, rather than only learning the goal of theskill (Tomasello 2001). A robot that mimics, rather than im-itates, would choose to use a trajectory that is derived fromthe demonstrated trajectory. Much of the existing work inskill learning by demonstration would qualify as mimick-ing in that the primary purpose of the learned model is tofaithfully reproduce the motion that was demonstrated. Wefocus on defining skill representations which enable a robotlearner to perform both simple mimicking (skill adaptation)and true imitation (transfer).

ApproachWe approach the tasks of within-domain skill adaptation andcross-domain skill transfer by using representations guided

(a) Initial State (b) Goal State

Figure 1: Overhead View of Pick-and-Place Skill

by mimicking and imitation. Action models learned by imi-tation and mimicking may enable different forms of transfer.We discuss two approaches to representing skills learned viakinesthetic demonstrations such that they can then be trans-ferred.

Trajectory-Based RepresentationWe approach mimicking computationally as a skill repre-sentation that encodes both the demonstrated trajectory andthe achieved goal. Many approaches exist for represent-ing recorded trajectories, such as Gaussian Mixture Models(GMMs) and Dynamic Motion Primitives (DMPs) (Schaal2006). We chose to investigate the transferability of skillsrepresented as DMPs, sub-skills that can be adjusted para-metrically. They are used to provide flexibility in complexskills; rather than learn the entire skill as a single unit, theskill can be broken down into sub-skills, or motion primi-tives, that can be learned and parameterized separately, orchained to form a more complex skill (Pastor et al. 2009;Niekum et al. 2012).

We applied DMPs to learning a task consisting of slid-ing an object (the yellow block shown in Figure 1) and thenplacing it on a target (near the red block and on the greensquare). This task was represented by two sub-skills, slid-ing and picking/placing, which were each represented as aDMP. The DMP policy holds that u = π(x, α, t), where thecontrol vector u is dependent on the continuous state vectorx, time t, and the parameter vector α (Schaal 2006), whichwe define as consisting of the following data:

• The robot’s initial end-effector position (as cartesian co-ordinates) and rotation (as quaternions), represented by< x, y, z, qx, qy, qz, qw >

• The position of the block, represented as the tuple <xi, yi > containing the offset, in cartesian coordinates, ofthe center of the block’s left edge, provided by the imagecapture data.

• The intended goal location of the block, represented asthe tuple < xf , yf >, containing the offset, in cartesiancoordinates, of the right edge of the block, provided bythe image capture data.

Preliminary Results By splitting the task into two sub-skills, each represented by a DMP, the task could be trans-ferred to modified examples of the same skill, such as whenthe objects were slightly offset from the original demonstra-tion. The object offset can be determined using the robot’sdepth sensor or RGB camera data. This method is best suitedfor skill adaptation problems in which the two situations, the

29

skill to be transferred and the target, are similar in both thetypes of the objects used and the method of achieving thegoal. However, this skill representation may not allow forcross-domain transfer, such as transfer to scenarios in whichthe demonstrated trajectory cannot be used.

This approach enables mimicry, as both the goal andmethod in which the goal was achieved (the trajectoryrecorded during the demonstration) are encoded in the skillrepresentation. This illustrates how mimicry-based ap-proaches to skill learning enable within-domain transfer.

Constraint-Based RepresentationWe propose a second skill representation based on imita-tion learning, in which only the goal of the demonstrationis encoded, rather than the trajectory used to demonstratethe skill. Our approach builds from the interactive learningmethods described previously.

This representation encodes a skill as a set of goals, whicheach consist of a set of constraints. Two types of constraintsare essential to a skill representation: the geometric con-straints, such as the location of the robot’s end-effector atthe start and end of the skill demonstration, and the dynamicconstraints, such as the end-effector velocity at key points ofthe skill demonstration. Rather than encode the full trajec-tory, this method encodes only what is essential to the skillby building a set of constraints. These constraints are thenused to plan a trajectory such that the skill is successfullyimitated.

In contrast to the trajectory-based method, which requireda single trajectory demonstration, this method requires botha keyframe demonstration and a trajectory demonstration.The keyframe demonstration serves to specify the geometricconstraints of the skill, such as the initial and final configu-ration of the robot’s arm in 7-DOF joint space. The begin-ning and end states of the skill will always be representedas keyframes, with additional keyframes potentially indi-cated at other points throughout the skill demonstration. Thekeyframes are then represented as a set G = {g1, ..., gn},where n > 1 and each element gi represents a geometricconstraint of the skill. Each geometric constraint is encodedas the 7-element tuple containing the joint positions of therobot’s arm. The speed of the robot’s end-effector is not en-coded in these keyframes.

Once the geometric constraints are provided via akeyframe demonstration, the next step is to provide the dy-namic constraints. A trajectory demonstration of the skillis given at full speed, stopping only when the goal of theskill has been reached. The number of robot pose record-ings, n, is specified as n = tf/t, where tf is the duration ofthe entire trajectory demonstration and t is the time intervalbetween robot pose recordings. The trajectory can be rep-resented by the set D = {d1, ..., dn} where di is the robotpose at time interval i.

The set of geometric and dynamic constraints are thenaligned, creating a set of constraints such that each geo-metric constraint is associated with a velocity constraint.A nearest-neighbor approach is used to pair each keyframegi ∈ G to the pose dj ∈ D with the closest end-effectorposition, where dj occurs within a certain time period after

(a) Demonstrated Initial Posi-tion

(b) Demonstrated Apex

(c) Planned Initial Position (d) Planned Apex

Figure 2: Low-Velocity Pendulum Trajectory

the trajectory pose associated with keyframe gi−1. This timeperiod extends until tw, which is defined as the following:

tw = tf ·i+ 1

n(1)

The change in end-effector positions between dj−1 and djalong the x, y, and z axis are used to determine the veloc-ities at keyframe gi. We assume that the trajectory demon-stration starts near the first keyframe and passes near the endkeyframe.

Once the geometric and dynamic constraints have beenaligned, the skill is represented by the set C = {c1, ..., cn},where n is the number of keyframe poses in G, and cirepresents a constraint as a 10-element tuple containingthe geometric constraint (7-DOF arm’s joint state) and dy-namic constraint (velocities along the x, y, and z axis). Anacceleration-limited planner, described in (Kunz and Stil-man 2014b), is then applied to this set of constraints to createa trajectory that, when executed, satisfies the demonstratedconstraints (Kunz and Stilman 2014a; 2014b).

Preliminary Results We applied this approach towardlearning a pendulum skill illustrating the effect of demon-strated velocity on the robot’s skill execution. Two ver-sions of the same skill were demonstrated: one in whichthe pendulum is tapped at a low speed, and one in whichthe pendulum is swung at a higher speed. The velocity atwhich the pendulum is tapped is thus reflected in the heightof the pendulum’s swing. The low-velocity demonstrationand resulting trajectory created by the planner are shownin Figure 2. The higher-velocity demonstration and result-ing planned trajectory are shown in Figure 3. As expected,

30

(a) Demonstrated Initial Posi-tion

(b) Demonstrated Apex

(c) Planned Initial Position (d) Planned Apex

Figure 3: High-Velocity Pendulum Trajectory

the pendulum swing differed depending on the demonstratedspeed, demonstrating one example of within-domain adap-tation.

The primary benefit of this method is that rather than en-code the full skill, only the constraints necessary to com-plete the skill are recorded. This provides an abstractionof the skill, as it leaves the specifics of execution up to theplanning stage. Thus, we suggest that this approach is alsosuitable for within-domain transfer as well. For example,closing a laptop and closing a book have similar goals, butmay require different methods (trajectories) to achieve thatgoal. By encoding the demonstrated skill as its goal (reach-ing the top of the laptop at a particular downward velocity),the skill representation is transferrable to the book-closingtarget problem as well.

Conclusions and Future WorkWe have proposed two classifications of transfer problems:cross-domain transfer and within-domain adaptation. Wehave described two approaches to representing skills suchthat adaptation and transfer are enabled. Trajectory-basedskill representations may enable within-domain adaptationby mimicking the teacher’s method of achieving the goal,while goal-based skill representations may be applicable toboth cross-domain transfer and within-domain adaptation byimitating the intention of the teacher’s actions, rather thanthe teacher’s method of reaching the goal.

Future work will involve a study of each skill represen-tation’s adaptability and transferability. Examples of poten-tial adaptation problems include learning the pendulum skillat different speeds, and learning a pick-and-place skill inwhich objects of varying sizes are used. Examples of po-

tential transfer problems include a closing skill in which therepresentation must be transferred from a box-closing prob-lem to a book-closing problem, and a pushing skill in whichthe representation must be transferred from pushing a cart topushing a drawer.

AcknowledgementsWe thank Tobias Kunz for applying the acceleration-limitedplanner to skill constraints to generate the trajectories shownin Figures 2 & 3. This work was supported by ONR Grant#N000141410120 and NSF GRF Grant #DGE-1148903.

ReferencesAkgun, B.; Cakmak, M.; Jiang, K.; and Thomaz, A. L.2012a. Keyframe-based learning from demonstration. In-ternational Journal of Social Robotics 4(4):343–355.Akgun, B.; Cakmak, M.; Wook Yoo, J.; and Thomaz, L. A.2012b. Trajectories and keyframes for kinesthetic teaching:A human-robot interaction perspective. In ACM/IEEE Intl.Conference on Human-robot interaction (HRI), 391–398.Argall, B. D.; Chernova, S.; Veloso, M.; and Browning,B. 2009. A survey of robot learning from demonstration.Robotics and Autonomous Systems 57(5):469–483.Cakmak, M.; DePalma, N.; Arriaga, R. I.; and Thomaz,A. L. 2010. Exploiting social partners in robot learning.Autonomous Robots 29(3-4):309–329.Fitzgerald, T., and Goel, A. K. 2014. A case-based ap-proach to imitation learning in robotic agents. In Intl. Conf.on Case-Based Reasoning Workshop on Case-Based Agents.Goel, A. K. 1997. Design, analogy, and creativity. IEEEexpert 12(3):62–70.Kunz, T., and Stilman, M. 2014a. Kinodynamic RRTs withfixed time step and best-input extension are not probabilisti-cally complete. In Int. Workshop on the Algorithmic Foun-dations of Robotics.Kunz, T., and Stilman, M. 2014b. Probabilistically completekinodynamic planning for robot manipulators with accelera-tion limits. In IEEE/RSJ Int. Conf. on Intelligent Robots andSystems.Niekum, S.; Osentoski, S.; Konidaris, G.; and Barto, A. G.2012. Learning and generalization of complex tasks fromunstructured demonstrations. In In IEEE/RSJ Int. Conf. onIntelligent Robots and Systems, 5239–5246. IEEE.Pastor, P.; Hoffmann, H.; Asfour, T.; and Schaal, S. 2009.Learning and generalization of motor skills by learning fromdemonstration. In Robotics and Automation, 2009. ICRA’09.IEEE International Conference on, 763–768. IEEE.Schaal, S. 2006. Dynamic movement primitives-a frame-work for motor control in humans and humanoid robotics. InAdaptive Motion of Animals and Machines. Springer. 261–280.Thagard, P. 2005. Mind: Introduction to cognitive science.MIT press.Tomasello, M. 2001. The Cultural Origins of Human Cog-nition. Harvard University Press.

31