A Data-Driven Method for Determining NaturalHuman-Robot Motion Mappings in Teleoperation

Rebecca M. Pierce and Katherine J. Kuchenbecker

Abstract— Though many aspects of teleoperation systemshave been fine-tuned through research, the mapping betweenthe operator’s movement and the robot’s movement is al-most always pre-defined as a simple scaling and offset. Webelieve that implementing nontraditional data-driven motionmappings has the potential to further improve the usabilityof teleoperation platforms, making it easier for a human toremotely complete challenging tasks. This paper presents a newparadigm for determining data-driven human-robot motionmappings for teleoperation: the human operator mimics thetarget robot as it autonomously moves its arm through a varietyof trajectories. We analyze the resulting motion data to find thehuman’s chosen mapping combined with the systematic errorsthe human made when relying on proprioception to executethese arm movements. We report results from a study in whichnine human subjects repeatedly mimicked the planar armmotions of a simulated PR2. We propose three nontraditionalmotion mappings (similarity, affine, and variable similarity),and we fit each of the proposed models to these data sets,verifying within and across trials and subjects. As hypothesized,the newly proposed mappings are found to outperform thetraditional motion mapping model.

I. INTRODUCTION

In the wake of catastrophes such as the Great Hanshinearthquake and the Oklahoma City bombing in the mid-1990s, many researchers turned their efforts to creatingmachines that can safely explore disaster sites to find andassist survivors under the guidance of a remotely locatedoperator. These new rescue robots were tested in action forthe first time in the aftermath of the terrorist attacks onSeptember 11, 2001, assisting with search and rescue atGround Zero in New York City. Casper and Murphy reportedthese experiences and recommended a variety of improve-ments for rescue robotics technology, including significantchanges to the human-machine interfaces [1]. These recom-mendations inspired us to take a fresh look at teleroboticcontrol interfaces, hunting for ways to make them morenatural for humans to use, especially in stressful situations.

First, Casper and Murphy state that rescue robots mustbe transportable and controllable by one person to minimize

Research was sponsored by the Army Research Laboratory and wasaccomplished under Cooperative Agreement Number W911NF-10-2-0016.This work also benefitted from equipment supported by the NationalScience Foundation under Grant No. 0855192. The student investigatorwas supported by research fellowships from the Department of MechanicalEngineering and Applied Mechanics of the University of Pennsylvania andthe National Science Foundation Graduate Research Fellowship Programunder Grant No. DGE-0822.

The authors are with the Haptics Group of the GRASP Laboratory,Department of Mechanical Engineering and Applied Mechanics, Universityof Pennsylvania.{rpierce,kuchenbe}@seas.upenn.edu

the number of people at risk during a mission. Furthermore,the danger of carrying large objects across hazardousenvironments requires all equipment to be transportable viawearable containers. Second, rescue workers usually do notsleep during the first forty-eight hours on the scene, andthey sleep for no more than a few hours per day thereafter.Therefore, Casper and Murphy suggest that the controlinterfaces for rescue workers need to be made as intuitiveas possible to account for the lower cognitive capacitiesthat arise under extreme sleep deprivation. These guidelinesstrongly support investigation of wearable control interfacesthat measure natural human motion through sensors such asinertial measurement units, as an alternative to the typicaloperator control boxes that rely on switches, buttons, andknobs. In addition, there is not yet one robot that can performall the tasks needed in a search and rescue mission, so itwould be beneficial for a single human-machine interfaceto be able to control several different robotic platforms.

Many attributes of a teleoperation interface affect itsusability. One core aspect that has rarely been studied is theway in which the robot’s motion command is calculated fromthe operator’s arm movement. Though typically done witha simple scaling and offset, we believe that creating data-driven motion mappings from the operator to the target robothas the potential for improving the ease of use of wearableteleoperation control devices. Accordingly, we sought tocreate a new platform-independent paradigm for determininga natural mapping from the motion of an operator’s arm tothe motion of a robotic manipulator.

After discussing relevant background material, we test thehypothesis that nontraditional human-robot motion mappingscan be derived by having a subject mimic a robot movingits arm through a trajectory. Systematic differences betweenthe human’s and robot’s trajectories can be modeled andidentified from the data. The calculated data-driven mappingsare the result of a coupling between the motion mappingchosen by the human and any systematic spatial distortionsthey made when moving their arm. We use within-trial,across-trial, and across-subject validation on data from nineindividuals to demonstrate and test our approach.

II. BACKGROUND

Teleoperation predates nearly all achievements in modernrobotics, starting in the early 1950s when Goertz and col-leagues created mechanical and electromechanical systems toallow operators to safely handle radioactive materials [2],[3].In these early teleoperators, the master and the slave werekinematically identical, and the natural arm motions of the

The Fourth IEEE RAS/EMBS International Conferenceon Biomedical Robotics and BiomechatronicsRoma, Italy. June 24-27, 2012

978-1-4577-1198-5/12/$26.00 ©2012 IEEE 169

human were reproduced by the remote slave via joint-levelmechanical or electrical connections. As the field of roboticsbegan to expand, applications broadened, and teleoperationmoved to using computational connections between the twodevices. To work with highly complex robotic platforms,operator interfaces and telerobotic control architectures havegrown increasingly sophisticated. However, one of the sim-plest teleoperation architectures, direct position mapping,is still widely used: the slave’s desired position is almostalways calculated by applying a scaling and an offset to themeasured position of the master device [4].

While this method of calculating desired position hasproven successful, it is important to remember that it rep-resents just one of a wide variety of possible motion map-pings between the human and the robot. Romano et al.compared the standard position mapping scheme to a ratecontroller, which maps the master’s position to the slave’svelocity, and to a mouse ballistics-inspired hybrid controller,which nonlinearly maps the master’s velocity to the slave’svelocity [5]. A user study showed that subjects were ableto complete a targeting task using teleoperated steerableneedles most accurately using the hybrid control law. Thiswork provides evidence that humans may find nontraditionalmotion mappings to be more intuitive than the standardapproach, depending on the needs of the task.

Many other researchers have created operator-adapted con-trollers to improve the fidelity, transparency, and robustnessof teleoperation systems [6]. These researchers often lookto mathematical models of human motion to create higherquality control schemes. One of the most commonly imple-mented models is Flash and Hogan’s minimum jerk criterion,which describes voluntary human arm motion as followingthe smoothest possible path [7]. For example, Maeda et al. [8]and Corteville et al. [9] successfully used the minimum jerkcriterion to predict human motion for improved cooperativeobject transportation and manipulation. These methods couldeasily be adapted to teleoperative applications.

The minimum jerk criterion, however, does not describehow humans perceive space around their bodies when mak-ing voluntary arm motions. Gordon et al. showed that reach-ing movements are planned in a hand-centered coordinateframe with direction and magnitude as two independent pa-rameters [10]. Ghez et al. later showed that healthy subjectsforced to rely solely on proprioception make systematicdirectional errors in reaching movements if their hand islaterally displaced from their body centerline [11]. Thesedirectional errors were further characterized in [12] and [13]and are reprinted in Fig. 1. With the right arm, motions start-ing at the right are rotated clockwise, and motions starting atthe left counter clockwise. This spatially variable directionalerror causes straight-forward reaching motions to tend to veerradially out from the body by up to 20◦ at the edges ofthe arm’s workspace; these errors are generally believed tostem from miscalibrations in proprioception. We note that anoperator relies heavily upon proprioception to teleoperate aremote platform using natural arm motions, since the visualchannel is focused on the target platform and the environ-

Fig. 1. Humans make systematic rotational errors in reaching movementswhen relying on proprioception. Reprinted with permission from Ghez etal. [13]. Copyright MIT Press, 2000.

ment seen through a 2D camera view. Taking inspirationfrom Ghez et al., we hope to improve the naturalness ofteleoperation by inverting the type of spatial distortions seenin Fig. 1 when calculating the desired position for the slaverobot.

III. EXPERIMENTAL SETUP

We hypothesize that we can discover the distortions thatexist in an operator’s spatial perception by having the in-dividual mimic a robot moving its arm through a trajectory.Systematic differences in the trajectories should let us see themotion mapping the human chooses, which is presumablynatural and comfortable. We can test this hypothesis byrecording synchronized robot and human movements andfinding the best mapping from the human’s movement tothat of the robot. Willow Garage’s Robot Operating System(ROS) [14] is a natural choice for use in this project, since amajor goal of our work is to make extensible algorithmsto semi-automatically deduce motion mappings from anoperator to a variety of robotic platforms. The algorithmsdeveloped in this paper are independent of the method usedto capture the human’s arm movement; optical tracking,magnetic tracking, inertial measurement units, and sensorssuch as the Microsoft Kinect would all work.

We note that in graphics, retargeting recorded humanmotion to animate virtual characters has become a standardmethod for creating realistic movements [15]. Techniquesfrom computer animation have also been adapted to animatehumanoid robots, e.g., [16], [17], [18]. However, the goalof this body of prior research has been to create human-like robot motion to enhance human-robot interactions, whileour work will use retargeting techniques to allow humans tointuitively teleoperate robotic platforms in real time.

A. ROS and the PR2

ROS’s modular, multi-lingual, and open-source packagesfacilitate the development of algorithms for use on several

170

Fig. 2. A video of the PR2 making planar motions with its right arm wasprojected on a large screen at the front of the motion capture space. Thesubject mimicked the robot motion in real time using comfortable motionsof her right arm.

different robotic platforms. Willow Garage’s humanoid, thePR2, is one of the best supported robots in ROS and isavailable in the University of Pennsylvania’s GRASP lab.We recorded the PR2 moving its arm through commandedtrajectories using Gazebo, a three-dimensional multi-robotsimulation environment supported by ROS [19]. Fig. 2 showsthe recorded view of the simulated robot presented to thesubject. Equivalent alternatives would have been to recordthe actual robot moving or to physically locate it with theoperator during testing.

This work focuses on identifying transformations betweenhuman motion and robot motion for trajectories confined toa horizontal plane, since this is the plane in which Ghez etal. [13] found systematic distortions. Planar robot arm motionwas produced using ROS’s real-time joint controller. Theshoulder and elbow joints were controlled to follow pre-settrajectories over time, and the other joints were commandedto stay at fixed angles. Fig. 3 shows the trajectories ofthe PR2’s hand for the eight motions used. The robot’shand moved with approximately constant speed in motions 1through 5 and at varied speed in motions 6 and 7. The PR2was recorded making each motion from six to ten times overapproximately 90 seconds. The view point in the movie wasoverhead looking down, as in Fig. 1.

B. Motion Capture

We recorded human movement using the Vicon motioncapture system in the Penn SIG Center. The subject wears afull-body suit covered with 53 passive retroreflective fiducialmarkers. These markers are individually placed adjacent toall major joints in the body, to provide a stationary referencepoint when the corresponding joint is moved. Because [12]and [13] describe systematic errors in hand positions andvelocities, we decided to focus our analysis on comparing theposition of the human’s hand to the position of the robot’shand. The position of the subject’s hand was taken to be thelocation of the marker on the wrist by the base of the thumb.

C. Experimental Procedure

All study procedures were approved by the University ofPennsylvania IRB under protocol #815023. Nine subjects

participated in the study (seven male and two female). Allsubjects were right-handed and between the ages of 20 and31. Each subject gave informed consent before participating.

As shown in Fig. 2, the videos of the PR2 making armplanar motions were projected onto a large screen at the frontof the motion capture space. The subject was instructed tomimic the PR2’s motion as closely as possible, using onlycomfortable motions of the right arm. The instruction ofcomfortable motions was important to the objective of thiswork – we want to derive human-robot motion mappingsthat could be used for hours on end with minimal physicaland mental fatigue. The subjects were given no specificinstruction on how to accomplish this task, since it wasimportant that each picked the mapping that was most naturalto him or her. The subject was first shown the constant-speed practice motion from Fig. 3, then each of the numberedmotions twice. The seven motions were presented in randomorder for each subject, and the two viewings of each motionoccurred sequentially.

This procedure yielded motion recordings of the subjectattempting to mimic the robot repeatedly performing eachtrajectory in Fig. 3. The subject’s motion data was recordedat a constant rate of 120 Hz, while the PR2’s motion wasrecorded with time stamps at an irregular rate of approxi-mately 1000 Hz. The robot data stream was down-sampledto 120 Hz via linear interpolation. The two data streamswere then aligned in time by finding the segment of humandata that yielded the smallest average Cartesian distanceto the corresponding robot data stream under a similaritytransformation. Once aligned in time, both data streamsneeded to be described in similar coordinate systems. Wechoose a right-handed reference frame with its origin at thecenter of rotation of the right shoulder. The X-axis of thisreference frame is directed from the left shoulder to the rightshoulder, the Y -axis points out from the chest, and the Z-axispoints up. A visual examination of the captured data showedthat there were often very large discrepancies between thehuman and robot motions at the beginning of each data set,during the time when the human was learning the periodicmotion of the robot, as would be expected. Additionally,subjects often stopped mimicking the robot just before theend of each video. Therefore, the first twenty seconds andthe last two seconds of each data set were excluded fromanalysis.

At the end of the study, each subject completed a shortquestionnaire to explain the strategies and methods they usedto accurately reproduce the robot’s motions. Subjects alsocompleted a NASA Task Load Index (NASA-TLX) [20] torate the difficulty of the task. Subjects also indicated

IV. FOUR MOTION MAPPING MODELS

Once synchronized in time and space, the motion dataobtained during the study was analyzed to look for trendsin the differences between each subject’s motion and that ofthe robot. These differences can be attributed to the motionmapping chosen by the subject, as well as unintentionalspatial distortions made by the subject while mimicking the

171

Motion 1 Motion 2 Motion 3

500 0 5000

500

1000

X Position [mm]

Y Po

sitio

n [m

m]

Motion 4

500 0 500X Position [mm]

Motion 5

500 0 500X Position [mm]

Motion 6

500 0 500X Position [mm]

Motion 70

500

1000Y

Posi

tion

[mm

]

Practice

Fig. 3. Trajectories of the eight motions created by the PR2’s right hand.

robot’s motion. Four models of increasing complexity werefit to map the motion made by the subject to the motion ofthe robot, as described below.

A. Traditional: Predefined Uniform Scaling

To handle kinematically dissimilar robots, the position ofthe master device is typically scaled and offset to better allowthe user to perform a task in the slave’s environment [4]. Inthe Cartesian plane this mapping can be represented as[

x′ y′]=

[x y

] [S 00 S

]+[γ1 γ2

](1)

where [x′ y′] is the robot’s position and [x y] is the human’s.We set the scale factor, S, to be the ratio of the robot’s armlength to the human’s arm length, while the offset, [γ1 γ2]is the vector from the mean of the scaled human position tothe mean of the robot position.

B. Similarity Transformation

In transformation geometry, a similarity is an operationfor which the distance between two points is proportionalto the distance between the two transformed points [21]. Asimilarity transformation consists of a scaling, a rotation,a reflection, and/or a translation. In the Cartesian plane, asimilarity can be expressed as follows, where c is a scalefactor, T is a 2× 2 orthogonal matrix, and θ is the rotationangle. [

x′ y′]=

[x y

]c T +

[γ1 γ2

](2)

T =

[cos(θ) sin(θ)− sin(θ) cos(θ)

](3)

To find the similarity transformation that best fits the human’smotion to the robot’s, we used the least-squares methoddeveloped by Schonemann [22].

C. Affine Transformation

An affine transformation is a colineation that preservesparallelness between two lines; it can consist of a strain, ashear, a rotation, a reflection, and a translation. In the Carte-sian plane, the transformed point [x′ y′] can be expressed asa linear combination of x and y.[

x′ y′]=

[x y

]T +

[γ1 γ2

](4)

T =

[a cb d

](5)

The best fit affine transformation from the human trajectoryto the robot trajectory can be solved for in the same manneras the best fit similarity transformation, relaxing the con-straint that T be an orthogonal matrix.

D. Variable Similarity

The final and highest dimensional model analyzed in thispaper is a position-based warping of space around the human.This model was inspired by the data presented by Ghez etal. in [13]. As depicted in Fig. 1, these prior results showthat directional errors strongly depend on position of thesubject’s hand. To look for similar trends, we partitioned thehuman and robot data into half-second segments and foundthe best fit similarity transformation for each matched pair.The rotation, scale, X offset and Y offset (θ, c, γ1, and γ2from (2) and (3)) were plotted against the X and Y positionof the first data point in the time segment. To maintain areasonable model dimensionality, we found the best-fit planefor each parameter of the similarity fitting, resulting in amodel with twelve independent parameters:[

x′ y′]=

[x y

]c(x, y)T (x, y) + γ(x, y) (6)

T (x, y) =

[cos(θ(x, y)) sin(θ(x, y))− sin(θ(x, y)) cos(θ(x, y))

](7)

γ(x, y) =[γ1(x, y) γ2(x, y)

](8)

c(x, y) = acx+ bcy + cc (9)

θ(x, y) = aθx+ bθy + cθ (10)

γ1(x, y) = aγ1x+ bγ1y + cγ1 (11)

γ2(x, y) = aγ2x+ bγ2y + cγ2 (12)

While studying pilot data, we also tested a motion mappingscheme that transforms the direction and magnitude of theuser’s velocity as a function of the position of the user’s hand.This mapping in the velocity domain creates robot motionsthat depend on the path of the subject. Thus, certain humanmotions could cause the mapping in the velocity domain tocommand desired positions beyond the robot’s workspace.Additionally, only rotation angle and scaling could be fitin the velocity domain, while the warping presented in thispaper can also fit the X offset and the Y offset. In thevariable similarity transformation, the X offset and Y offsetdescribe both global translation and differential scaling.

E. Summary of Mappings

All of the proposed data-driven motion mappings willbe used to find the best fit transformation from the humanmotion data to the robot data. Applying the fitted mappingto human motion will calculate the desired robot trajectoryfor each data set. The four motion mappings investigated inthis paper are summarized in Table I. This table includesa visualization of how the space around the human will bemorphed to command a desired robot pose. The traditionalposition mapping uniformly scales and offsets human motionto map it to a desired robot position. The similarity mappinguniformly scales, offsets, and rotates the human motion,

172

while the affine transformation adds differential scalingand shear. Finally, the variable similarity motion mappingsmoothly warps, scales, and offsets the human position tocalculate desired robot motion.

Model Fitted Parameters Visual Effect

Traditional S

−600 −400 −200 0 200 400 6000

100

200

300

400

500

600

700

800

900

1000

X Position [mm]

Y P

ositio

n [

mm

]γ1γ2

Similarity c

−600 −400 −200 0 200 400 6000

100

200

300

400

500

600

700

800

900

1000

X Position [mm]

Y P

ositio

n [

mm

]θγ1γ2

Affine a

−600 −400 −200 0 200 400 6000

100

200

300

400

500

600

700

800

900

1000

X Position [mm]

Y P

ositio

n [

mm

]bcdγ1γ2

Variable Similarity ac, aγ1

−600 −400 −200 0 200 4000

100

200

300

400

500

600

700

800

900

1000

X Position [mm]

Y P

ositio

n [m

m]bc, bγ1

cc, cγ1aθ, aγ2bθ, bγ2cθ, cγ2

TABLE ITHE FOUR MOTION MAPPINGS CONSIDERED IN THIS WORK.

V. RESULTS

Preliminary analysis of the data showed that all of themotion mapping schemes were capable of transforminghuman motion to the general shape of the robot motion,but they were unable to fit the global position. For each ofthe three data-driven models, more than 20% of the error ininitial within-trial tests was explained by the offset centersof the transformed human and robot motion. This effectgreatly worsened when human motion was transformed usinga model trained on different data sets, rising to more than35% for all four motion mappings. This discrepancy is due tothe fact that one’s proprioceptive estimation of arm positiondrifts significantly over time [23]. However, even with a largedrift in proprioception, the direction and extent of motionremain relatively constant [24]. This effect is clearly evidentwhen humans blindly draw repeated shapes: subjects renderseveral nearly identical shapes with offset centers [25], [26].In the post-study survey, subjects were asked to estimatethe percentage of time that they focused the center of theirvision on their arms. The mean response to this questionwas 9.2% with a standard deviation of 13%, indicating thatsubjects relied heavily upon proprioception to complete thetask. Thus, it was not surprising that a similar drift was foundin our data set. For this reason, we chose to evaluate allfittings by translating the center of the transformed humanmotion to the center of the robot motion.

0

10

20

30

40

50

60

70

80

90

Erro

rs [m

m]

SimilarityTraditional Variable SimiliarityAffine

test

1te

st 2

test

3te

st 4

test

1te

st 2

test

3te

st 4

test

1te

st 2

test

3te

st 4

test

1te

st 2

test

3te

st 4

Fig. 4. Average errors for tests 1 through 4 on the four tested motionmappings.

A. Validation of Models

The quality of each motion mapping model was deter-mined using four tests that differed in their choice of trainingand validation data. In each test, the parameters of the threedata-driven motion mappings were determined by fittinghuman motion from a training set to the corresponding robotmotion. The identified mappings were then used to transformthe validation set’s human motion to a predicted robot motionfor comparison to the actual robot motion. The scale factorin the traditional fittings was always taken as the ratio ofthe length of the PR2’s arm to the length of the validationsubject’s arm. In the first test, the models were trained andtested on the same data; each trial represented one of thefourteen recordings for one subject. Second, the training datawas set to be the first half of each recording, and the secondhalf of the recording was used as the validation set. Third,we performed a standard leave-one-out cross validation test:for every subject, each of the fourteen data sets was used asthe validation data for models trained on the combined dataof the remaining thirteen sets. Fourth, a standard leave-one-out cross validation test was performed across all subjects;the combined data of each of the nine subjects was usedas the validation set for motion mappings trained on thecombined data for the other eight subjects. The median of theCartesian distance from the transformed human position tothe robot’s actual position was used as the metric to evaluatethe goodness of each fit.

The errors yielded by the four validation tests for eachof the four motion mapping schemes are shown in Fig 4. Atwo-way analysis of variance (ANOVA) was implementedon the average values of the motion mapping errors foreach test, using the fixed factor of mapping type and therandom factor of subject number. These ANOVAs enable usto determine whether the motion mapping models yieldedsignificantly different errors in each test, taking α = 0.05.If model errors were found to differ significantly, a Tukey-Kramer post-hoc multiple comparison test was conducted ata confidence level of α = 0.05 to determine which modelsproduced significantly different errors. When the training andvalidation sets were the same (Test 1), the similarity, affine,and variable similarity transformations produced significantlylower error than the traditional fitting (F1(3,35) = 26.96,p1 < 0.0001, η21 = 0.2717), as one would expect from

173

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Traditional

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Similarity

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [

mm

]

Affine

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [

mm

]

Variable Similarity

Fig. 5. Transformed human motion (solid colored line) overlaid on robotdata (dashed line) for subject 6, motion 3 (test 1). The original human datais also displayed in light gray. In test 1, the training and validation sets arethe same.

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Traditional

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Similarity

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Affine

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Variable Similarity

Fig. 6. Transformed human motion (solid colored line) overlaid on robotdata (dashed line) for the within-trial validation test for subject 6, motion3 (test 2). The original human data is also displayed in light gray. Test 2splits each trial in half to form the training and validation sets.

the higher dimensionality of these fittings. When a mappingtrained on data from a given subject was tested on previouslyunseen motion data recorded from the same trial (Test 2) or adifferent trial (Test 3), the three data-driven mappings yieldedsimilar errors, with the variable similarity performing slightlybetter than the similarity and affine transformation. All threedata-driven mappings were again statistically significant im-provements over the traditional fitting (F2(3,35) = 17.33,p2 < 0.0001, η22 = 0.1462; F3(3,35) = 4.76, p3 = 0.0096,η23 = 0.0168). The errors yielded by the cross-subject vali-dation (Test 4) did not differ significantly from each other(F4(3,35) = 1.58, p4 = 0.2201). Though not significantlybetter, the variable similarity mapping is the only one thatperforms better than the traditional fitting in Test 4.

Figs. 5–8 show how the model trained in each one ofthe validation methods transforms a sample movement bySubject 6 to that of the robot. These plots make it clear thatthe variable similarity fitting can better match the features

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Traditional

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Similarity

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Affine

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Variable Similarity

Fig. 7. Transformed human motion (solid colored line) overlaid on robotdata (dashed line) for the leave-one-out validation test for subject 6, motion3 (test 3). The original human data is also displayed in light gray. Test 3involves a standard leave-one-out cross-validation within each subject.

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Traditional

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Similarity

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Affine

−500 0 5000

500

1000

X Position [mm]

Y P

ositio

n [m

m]

Variable Similarity

Fig. 8. Transformed human motion (solid colored line) overlaid on robotdata (dashed line) for the leave-one-out validation test across subjects formotion 3 (test 4). The original human data is also displayed in light gray.Test 4 involves leave-one-out cross-validation across subjects.

of human motion data to those of the robot motion data. Inthe same vein, Figs. 9–11 visually displays the distortionof space around each of the nine subjects’ bodies whenmapping human motion to robot motion under the data-driven similarity, affine, and variable similarity transforma-tions trained on the combined data of each subject. Thesefigures show that the similarity and affine transformationscan be viewed as local approximations to the total spatialwarping around the subject’s body. Thus, if the similarity andaffine transformations are trained on data recorded when thesubject’s hand is in a certain region, the resulting mappingsmay not be able to map human motion to robot motionwhen the subject’s hand moves to another region. Sincethe global position of the subject’s hand drifted over time,which was likely caused by the fact that they were usingproprioception to estimate their arm positions, it will beimportant to accurately model a human’s entire workspacewhen performing longer data captures.

174

0

0

Y P

ositio

n [m

m]

00

0

X Position [mm]

0

Fig. 9. Visualization of all nine subjects’ similarity mappings.

0

0

Y P

ositio

n [

mm

]

00

0

X Position [mm]0

Fig. 10. Visualization of all nine subjects’ affine mappings.

The fact that the variable similarity fitting performedslightly better than the traditional mapping in the cross-subject leave-one-out validation study means that some pa-rameters of this transformation are consistent across subjects.Though the variable similarity fittings shown in Fig.11clearly differ across subjects, there are some striking sim-ilarities among all of the identified mappings. Notably, thesesimilarities are in general agreement with the systematicrotational errors described by Ghez et al. [13] (Fig. 1).Fig. 11 shows how the human’s motion would have to betransformed to best match the robot’s motion. Thus, if thefindings from our study are consistent with those describedby Ghez et al., our transformation will appear to be aninversion of the distortions they described. Looking againat Fig. 11, we see that the angle of rotation error of thesubject is consistent with the previously reported results:the magnitude of the direction error grows with lateraldisplacement and is increasingly counterclockwise to theleft and clockwise to the right. Furthermore, the rotationalerrors made by subjects when the hand was between thebody midline and the right shoulder were fairly small. Withthe exception of points very near to the body for subjects1 and 3, the lateral location where the directional errorschange from clockwise to counterclockwise is within 30 cmof the shoulder at all points over all nine subjects, whichis consistent with the findings of Ghez et al. Additionally,much like their described distortions, our variable similarityfittings show a large dependance on the lateral position ofthe subject’s hand and a much smaller dependance on theextension of the subject’s hand.

VI. CONCLUSION AND FUTURE WORK

This paper is the first work to consider non-traditionaldata-driven motion mappings for teleoperation with the goal

0

0

Y P

ositio

n [m

m]

00

0

X Position [mm]0

Fig. 11. Visualization of all nine subjects’ variable similarity mappings.

of intuitiveness. We have created a standard paradigm thatallows us to determine motion mappings from the recordingsof a human mimicking a target robot autonomously movingthrough a trajectory. Simultaneously measuring the motionsof the human and the robot allows us to characterize how asubject systematically distorts space around his or her bodywhile imitating the robot. We believe these models can beused to allow the subject to naturally teleoperate the robot.

Three new motion mapping models were discussed in thispaper: similarity, affine, and variable similarity. To validatethese models, we trained and validated each model in fourtests: the same data set, portions of the same data, data fromother trials by the same subject, and finally data recordedfrom other subjects. In the tests where mappings were trainedand validated on motion data from the same subject, the threedata-driven motion mappings all yielded significantly lowererrors than the traditional motion mapping. The variablesimilarity fitting was the only motion mapping that yielded alower error than the traditional mapping for the cross-subjectvalidation study, though the difference was not significant.

The validation tests used for this study were an appropriatestarting point, but to see the true effectiveness of eachmotion mapping model, they will need to be implementedon a teleoperation platform and used by human operators.Therefore, we plan to run a user study in which subjects willmimic the PR2 as it moves through planar motions to allowus to find data-driven motion mappings for each individual,as done in this study. We will then implement these mappingson the PR2 and ask each subject to perform tasks viateleoperation. During this study we will be interested indetermining how constant the data-driven motion mappingsare across individuals, and whether the new motion mappingsaffect task performance or cognitive load.

We also plan on extending our data-driven motion map-ping models to three dimensions. All of the motion mappingsdeveloped in this paper can easily be extended to the threedimensional case. Both the similarity and affine transfor-mations require no additional work to find the best three-dimensional fitting. The variable similarity model can be fitin 3D by finding the best fit linear function for rotation,scaling, X , Y , and Z offset as a function of human X , Y ,and Z position. Because Ghez et al. [13] reported spatialdistortions only in the horizontal plane, we do not expectsignificant dependence on the third dimension.

Finally, we are interested in applying similar data-driven

175

motion mappings to assistive technology. Specifically, weplan on investigating how data-driven motion mappingswould allow a person with limited or deteriorated motion,such as a patient suffering from cerebral palsy or post-strokeparesis, to more easily control a target robot.

REFERENCES

[1] J. Casper and R. Murphy, “Human-robot interactions during therobot-assisted urban search and rescue response at the world tradecenter,” Systems, Man, and Cybernetics, Part B: Cybernetics, IEEETransactions on, vol. 33, no. 3, pp. 367–385, June 2003.

[2] R. Goertz and W. Thompson, “Electronically controlled manipulator,”Nucleonics, vol. 12, no. 11, pp. 46–47, November 1954.

[3] R. Goertz, “Mechanical master-slave manipulator,” Nucleonics,vol. 12, no. 11, pp. 45–46, November 1954.

[4] G. Niemeyer, C. Preusche, and G. Hirzinger, Telerobotics. Springer,2008, ch. 31, pp. 741–757.

[5] J. Romano, R. Webster, and A. Okamura, “Teleoperation of steerableneedles,” in Robotics and Automation, IEEE International Conferenceon, April 2007, pp. 934–939.

[6] C. Passenberg, A. Peer, and M. Buss, “A survey of environment-, operator-, and task-adapted controllers for teleoperation systems,”Mechatronics, vol. 20, no. 7, pp. 787–801, 2010, Special Issue onDesign and Control Methodologies in Telerobotics.

[7] T. Flash and N. Hogan, “The coordination of arm movements: Anexperimentally confirmed mathematical model,” The Journal of Neu-roscience, vol. 5, pp. 1688–1793, 1985.

[8] Y. Maeda, T. Hara, and T. Arai, “Human-robot cooperative manip-ulation with motion estimation,” in Proc. IEEE/RSJ InternationalConference on Intelligent Robots and Systems, vol. 4, 2001, pp. 2240–2245.

[9] B. Corteville, E. Aertbelien, H. Bruyninckx, J. De Schutter, andH. Van Brussel, “Human-inspired robot assistant for fast point-to-pointmovements,” in Proc. IEEE International Conference on Robotics andAutomation, April 2007, pp. 3639–3644.

[10] J. Gordon, M. Ghilardi, and C. Ghez, “Accuracy of planar reachingmovements. I. Independence of direction and extent variability.” Ex-perimental Brain Research, no. 99, pp. 97–111, 1994.

[11] C. Ghez, J. Gordon, M. Ghilardi, and R. Sainberg, “Contributionsof vision and proprioception to accuracy in limb movements.” inThe Cognitive Neurosciences, M. S. Gazzaniga, Ed. Cambridge,Massachusetts: MIT Press, 1994, pp. 549–564.

[12] M. G. F. Ghilardi, J. G. Gordon, and C. Ghez, “Learning a visuomotortransformation in a local area of work space produces directionalbiasesin other areas,” Journal of Neurophysiology, vol. 73, no. 6, pp. 2535–2539, 1995.

[13] C. Ghez, J. Krakauer, R. Sainburg, and M. Ghilardi, “Spatial repre-sentations and internal models of limb dynamics in motor learning,”in The New Cognitive Neurosciences, 2nd ed., M. Gazzaniga, Ed.Cambridge, Massachusetts: The MIT Press, 1999, pp. 501–514.

[14] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs,E. Berger, R. Wheeler, and A. Ng, “ROS: an open-source robot oper-ating system,” in Open-Source Software workshop of the InternationalConference on Robotics and Automation, 2009.

[15] M. Gleicher, “Retargetting motion to new characters,” in ACM SIG-GRAPH, 1998, pp. 33–42.

[16] N. Pollard, J. Hodgins, M. Riley, and C. Atkeson, “Adapting humanmotion for the control of a humanoid robot,” in Robotics and Automa-tion, IEEE International Conference on, vol. 2, 2002, pp. 1390–1397.

[17] B. Dariush, M. Gienger, B. Jian, C. Goerick, and K. Fujimura, “Wholebody humanoid control from human motion descriptors,” in Roboticsand Automation, IEEE International Conference on, May 2008, pp.2677–2684.

[18] L. Boutina, A. Eona, S. Zeghloula, and P. Lacouture, “From humanmotion capture to humanoid locomotion imitation application to therobots HRP-2 and HOAP-3,” in Robotica, vol. 29, 2010, pp. 325–334.

[19] N. Koenig and A. Howard, “Design and use paradigms for Gazebo, anopen-shource multi-robot simulator,” in Proc. IEEE/RSJ InternationalConference on Intelligent Robots and Systems, September 2004, pp.2149–2154.

[20] S. Hart and L. Staveland, “Development of NASA-TLX (Task LoadIndex): Results of empirical and theoretical research,” Human MentalWorkload, pp. 239–250, 1988.

[21] G. E. Martin, Transformation Geometry: An Introduction to Symmetry,J. H. Ewing, F. Gehring, and P. R. Halmos, Eds. Springer-Verlag,1982.

[22] P. H. Schonemann and R. M. Carroll, “Fitting one matrix to anotherunder choice of a central dialation and a rigid motion,” Psychometrika,vol. 35, no. 2, pp. 245–255, June 1970.

[23] J. P. Wann and S. F. Ibrahim, “Does limb proprioception drift?”Experimental Brain Research, vol. 91, pp. 162–166, 1992.

[24] L. E. Brown, D. A. Rosenbaum, and R. L. Sainburg, “Limb positiondrift: Implications for control of posture and movement,” Journal ofNeurophysiology, vol. 90, no. 5, pp. 3105–3118, November 2003.

[25] H. N. Zelaznik and D. Lantero, “The role of vision in repetive circledrawing,” Acta Psychologica, vol. 92, no. 1, pp. 105–118, June 1996.

[26] S. M. P. Verschueren, S. P. Swinnen, P. J. Cordo, and N. V. Dounskaia,“Proprioceptive control of multijoint movement: unimanual circledrawing,” Experimental Brain Research, vol. 127, pp. 171–181, 1999.

176