robot hand synergy mapping using multi-factor model and emg...

Robot Hand Synergy MappingUsing Multi-factor Model and EMG Signal

Sanghyun Kim, Mingon Kim, Jimin Lee, and Jaeheung Park

Graduate School of Convergence Science and Technology,Seoul National University, Korea

{ggory15,mingonkim,jmpechem,park73}@snu.ac.kr

http://dyros.snu.ac.kr

Abstract. In this paper, it is investigated how a robot hand can becontrolled from a human motion and an EMG signal in a tele-operationsystem. The proposed method uses a tensor to represent a multi-factormodel relevant to different individuals and motions in multiple dimen-sions. Therefore, the synergies extracted by the proposed algorithm canaccount for not only various grasping motions but also the different char-acteristics of different people. Moreover, a synergy-level controller whichgenerates motion and force of the robot is developed with postural syner-gies and an EMG signal. The effectiveness of the proposed new mappingalgorithm is verified through experiments, which demonstrate better rep-resentation of hand motions with synergies and greater performance ongrasping tasks than those of conventional synergy-based algorithms.

Keywords: Synergy, Robot Hand, Mapping, Multi-factor Model, EMG

1 Introduction

A robot hand can provide a great deal of manipulation capability to its user in atele-manipulation system. The method of controlling the robot hand with humanhand motion is one of the most important parts of such a system, as a humanhand can perform many types of operations given its number of joints, whereasa robot hand is limited in terms of motion compared to that of human handmotion. Thus, the functionality and controllability of dexterous robot handshave been investigated in an effort to overcome the difficulty stemming from thekinematic dissimilarity between robot and human hands.

Various kinematic maps between a human hand and a robot hand have beenproposed using the joint angles of the human hand [1], fingertips [2], and poses[3] or separate motion controllers using a supervisory control methods [4].

In particular, Santello et al. [5] demonstrated that there exist strong correla-tions between grasping postures of the hand. These correlation patterns, referredto as synergies, can be defined as a spatial configuration of the hand shape, asmore than 80% of all grasping postures can be described with only two synergies.Thus, how to extract synergies and how to use synergies have recently been stud-ied in attempts to reduce the high-dimensional data in the matrix representationto a lower dimensional space for human-like control and grasping [6]-[9].

2 Kim et al.

Data

Motion

Fig. 1: Illustration of a tensor including the factors of grasping motion and thefactors of individuals.

These approaches, however, still have limitations. First, these synergy map-ping schemes cannot account for individual characteristics, while Santello et al.[5] found that synergies differ from person to person. Hence, reconstructed mo-tion with synergies shows erroneous results for a user who does not undergo thetraining. Secondly, the grasping force cannot be represented using motion-basedsynergy-level controllers [7]. For example, it is difficult for a robot hand to graspa thin object, as a robot hand only follows human hand motion but not thegrasping force.

Hence the following questions are proposed: if the synergy is defined as thelinear combination of the grasping type and individual characteristics, can weobtain synergies that represent human hand motion more accurately? Addition-ally, if this is possible, can we extract synergies for a new user by only extractingthe factor of the new user? Finally, can we manipulate the robot hand dexter-ously, if the grasping force of the robot is generated from the intention of theuser? These issues constitute the main focus of this paper.

In this paper, a new type of mapping algorithm is proposed to answer thesequestions. Our goal is to extract the synergies of each user and to generate motionand grasping force in a robot using these synergies. The proposed algorithm usesa tensor composed of data relevant to different individuals and various motionsin multiple dimensions, as shown in Fig. 1. This tensor, which is regarded as amulti-factor model, is then decomposed into the two factors of the informationof the grasping motions and the individual characteristics. Thus, we can extractsynergies accounting for not only various grasping motions but also the differentcharacteristics of different people. Furthermore, the EMG signal of a user is usedto estimate the intention of the grasping force. This estimated grasping force isgenerated by the robot hand using synergies.

This paper is organized as follows. Section 2 introduces the method used toextract the postural synergies and control the robot hand using the multi-factor

Robot Hand Synergy Mapping Using Multi-factor Model and EMG Signal 3

model and the synergy-level controller. Section 3 presents the experimental setupand data sets to extract synergies. Section 4 discusses the result of the proposedalgorithm for extracting synergies compared with other methods, and the paperis concluded in Section 5.

2 Technical Approach

As shown in Fig. 2, the proposed algorithm consists of two parts: the extractionof the postural synergies with the multi-factor model, and the synergy-levelcontroller part for a robot with postural synergies from the multi-factor modeland the EMG signal.

In section 2.1, we introduce the method used to extract the postural syn-ergies of the user with the multi-factor model. The multi-factor model usingtensor decomposition is applied to separate the factors of the individuals andthe grasping hand motion. Therefore, when a new user performs several typesof training motions, the factor for the new user is computed by optimizationusing the multi-factor model. Consequently, the postural synergies for the newuser are extracted by combining the multi-factor model and the factor for thenew user. In subsection 2.2, we discuss the approaches used to overcome thekinematic dissimilarity between the robot hand and the user and to generatethe grasping force of the robot hand with the synergy-level controller. We usethe EMG signal to generate grasping force of a user, as there is a certain linearrelationship between the EMG signal and the grasp force [10].

The subsections below describe the details of the proposed algorithm. In ourpaper, bold lower case (a), bold upper case (A), and underlined characters (A)are used to denote the vector, matrix, and tensor, respectively.

2.1 Extracting Postural Synergies with the Multi-factor Model

In this section, we discuss the method used to extract the synergies of a user withthe multi-factor model. Several human hand motions are recorded for training.The training data set constitutes a tensor for the gallery in the proposed algo-rithm. The factors of individuals and grasping hand motions are then extracted

TCP/IP

Multi Factor Model

Screen Camera

Extracting the Factor of the User

EMG Signal

Synergy Controller

Master Slave

Computation of Synergies

Fig. 2: Schematic diagram using the multi-factor model and EMG signal forhand-teleoperation

4 Kim et al.

by the multi-factor model using tensor decomposition. In our experiments, thedimensions of the tensor for the gallery Y∈ RI×I×I indicate the number ofjoints (I1), the overall number of sample data obtained from each grasping type(I2), and the number of people (I3). The tensor is decomposed using the Tuckermodel [11], [12],

Y = G×1 A1 ×2 A2 ×3 A3 + E, (1)

where G∈ RJ×J×J is the core tensor, Ak ∈ RIk×Jk denotes the factor matricesof mode-k, and E ∈ RI×I×I is the error tensor.

Tensor G and the matrices Ak are calculated by the alternative least square(ALS) algorithm [12] to minimize the Frobenius norm F, as follows :

minG,A1,A2,A3

‖Y−G×1 A1 ×2 A2 ×3 A3‖2F

subject to G ∈ RJ×J×J , Ak ∈ RIk×Jk : orthonormal.(2)

Thus, the factor of joint A1 spans the space of the joint angles and the factor ofmotion A2 represents the space of the grasping motion denoting the principalmotion factor regardless of the individuals. The factor matrix of individuals A3

spans the space of the characteristics of an individual.Finally, as shown in Fig. 3, Eq. (1) can be reshaped so that it represents the

decomposition of only A2 and A3 by means of

Y ' G×2 A2 ×3 A3, where G = G×1 A1. (3)

The tensor G ∈ RI×J×J is a new core tensor.When a new user who has not participated in the multi-factor model performs

a few of the same types of motions existing in the training data, the factor forthe user, p, which denotes the relationship between the characteristics of thenew user and the individuals in the multi-factor model, is computed by

minp

‖F− G×2 A2,n ×3 pT‖1 subject to p ∈ RJ , (4)

where F ∈ RI×n is the motion data set of the new user, A2,n ∈ RI×n is thecorresponding factor matrix of motion in the multi-factor model, and n is thenumber of motions performed by the new user. Finally, the postural synergiesbi ∈ RI of the user and the corresponding coefficients c of each synergy arerepresented as (

b1 b2 ... bk

)= G×3 p

T, (5)

c =(b1 b2 ... bk

)+ × qh, (6)

where qh is the vector of the joint angles of a human. As noted in the Ap-pendix, the first synergy contains the greatest amount of information amongthe synergies because the multi-factor model provides the bases in the order ofimportance, like to the SVD algorithm [11], [12]. For example, if the first twosynergies are selected to represent the posture, the approximated joint angles ofthe human hand qh are represented as

qh =(b1 b2 0 ... 0

)× c. (7)


Y G

Gallery Data)

=Core Tensor )

New Core Tensor )

=

Fig. 3: Illustration of a tensor including the factors of grasping motion and thefactors of individuals.

2.2 Synergy-level Controller for Grasping Motion and Force

The synergy-level controller generates the motion and the force of the robot bymatching the synergies of the human to those of the robot. The synergies of theuser and the robot differ due to the level of kinematic dissimilarity. However, thecorresponding synergies of the robot hand can be computed by the assumptionthat the coefficients of the synergies of the human hand are identical to those ofthe robot hand. Thus, the synergies of the robot br,i are computed by(

br,1 br,2 ... br,k

)=(qr,1 qr,2 ... qr,k

)× c+, (8)

where, qr,i is the vector of the joint angles of the robot hand when the robot handis configured to match the corresponding human hand motion for each graspingtype, and where (+) refers to the pseudo-inverse. The size of each synergy vectorbr,i is equal to the number of joints of the robot hand.

Next, when the robot grasps an object, the grasping force of the robot canbe generated by increasing the coefficients of the synergies by the amount ofthe preprocessed EMG signal V (t) which is assumed to be proportional to theintention of the user with regard to their grasping force. Also, it should benoted that the coefficients of the synergies vary at approximately the same ratioduring one type of grasping motion. Therefore, the coefficients of the robot hand

6 Kim et al.

MCP

PIP

DIP

DIP

DIP

DIPPIP

PIP

PIP

MCPMCPMCPIP

MCPTMC

(a) (b)

Fig. 4: The comparison of kinematic structure between the human hand and therobot hand: (a) The joint location of hand; (b) The joint location of robot hand.

synergies, cr, are computed by

cr = c + K × V (t)× c

|c|, (9)

where K is a scalar component for gain and the vector c is the rate of changefor each coefficient. The grasping posture is maintained while the grasping forceis created using (9) and (10).

qr =(br,1 br,2 ... br,k

)× cr (10)

3 Experiment

The proposed algorithm is validated through experiments. On the master side,a motion capture system is used to track a human hand motion. The physicalhardware of the robot hand is used as a slave system. Finally, between the mastersystem and the slave system, there is a communication line (TCP/IP) to transferthe human hand motion information. The subsections below describe the detailsof the system configuration and the data sets.

3.1 System Overview

The human hand motion is tracked by a motion-capture system which uses thesoftware NEXUS (Vicon, Co. USA), twenty-four 5mm markers (Fig. 4a), and


Platform

Grasp

Small Diameter

Adducted Thumb

Medium Wrap

Light ToolPower Disk

Lateral PinchLarge

DiameterPrecision Disk

PowerSphere

Thumb‐3 or 4 Finger

Tripod

Thumb Index finger

Thumb‐2 Finger

Fig. 5: Grasp taxonomy according to Wei Dai [15].

fourteen cameras (Vicon T160 Camera). The frame rate was 100 frames persecond. On the other hand, the EMG sensor is placed on the extensor digitorumcommunis (the forearm) of the user. This surface EMG system, Trigo (Delsys,Co. USA) is used to obtain the EMG signal. The EMG signal is sampled at 1000Hz. During the preprocessing step, we used a linear envelope method to rectifythe amplitude of the signal [13]. To do this, we used a fourth-order ButterworthIIR filter. The cutoff frequency of the high-pass filter was 30 Hz and the cutofffrequency of the low-pass filter was 1 Hz. Finally, we experimentally determinedthe threshold of the EMG signal so that the noise caused by the movement ofthe hand could be ignored.

In the slave system, a fully actuated four-fingered robot hand, the Allegrohand [14], is used to achieve dexterous manipulation. The appearance and jointlocation of the Allegro hand are described in Fig. 4b.

3.2 Data sets of Human Grasping

According to the grasp taxonomy, there are fifteen grasping types [15] (Fig. 5).In our experiments, each grasping type (except for the platform) was collectedtwice. Thus, our data sets for human grasping were collected from twenty-eightdifferent grasping motions. In order to generate the multi-factor model, the datasets of five subjects (five men) are collected by the motion-capture system. Theaverage age and hand length, defined as the distance from the tip of the middlefinger to the midpoint interstylon line, of these participants were 27.3 ± 1.45years and 18.1 ± 2.14 cm, respectively. Finally, each grasping motion consistedof 100 frames through Dynamic Time Warping (DTW)[16]. Thus, the dimensionsof the tensor for the gallery were R×× in our experiments.

8 Kim et al.

Subject1 Subject2 Subject3 Subject4 Subject50

1

2

3

4

5

6

7

Mea

n A

bso

ult

e E

rro

r (D

EG

)

Ten2DPCA2DRPCA2DTen3DPCA3DRPCA3D

Fig. 6: The mean absolute error between the hand posture and the reconstructedposture with two and three synergies of the subjects who participated in thetraining.

4 Results

In this section, we experimentally validate the proposed algorithms for extractingpostural synergies and for generating the grasping force of the robot hand.

4.1 Extracting Postural Synergies with the Multi-factor Model

We validate the proposed algorithm for synergy compared to other synergy-extraction methods, in this case Principal Component Analysis (PCA) and Ro-bust Principal Component Analysis (RPCA) [17], [18]. PCA is one of the mostwidely used algorithm to extract synergies [5]-[9], and RPCA is a modified ver-sion which is robust with respect to the different tendencies of people whenused for training. Also, it is fair to compare with linear methods and our pro-posed algorithm, as these both algorithms are linear methods which reduce thedimensionality.

We measured the angular difference of each joint between the actual postureand the reconstructed posture to compare how well each algorithm reconstructsthe human hand motion using two or three synergies.

Fig. 6 shows the mean absolute error (MAE) between the hand posture andthe reconstructed posture with two or three synergies for people who partici-pated in the training for the gallery data. Thus, the gallery data set for eachalgorithm contained the motion data of the users. As shown in Fig. 6, the pro-posed algorithm produced a much closer posture to the hand motion than theother methods. This is because the proposed algorithm extracts the posturalsynergies for each training user with tensor representation.

Fig. 7 shows the MAE result of the extracted postural synergies for peoplewho did not participate in the training. Although the gallery data did not con-tain the motion data of the users, better hand motions were represented by theproposed algorithm. This is because the proposed algorithm accounts for the


Subject1 Subject2 Subject30

2

4

6

8

Mea

n A

bso

ult

e E

rro

r (D

EG

)

Ten2DPCA2DRPCA2DTen3DPCA3DRPCA3D

Fig. 7: The mean absolute error between the hand posture and the reconstructedposture with two and three synergies of the subjects who did not participate inthe training.

characteristic of the users. The synergies for the user are calculated by the lin-ear combination of the synergies of all training of the people in the multi-factormodel and using the factor of the user p.

4.2 Synergy-level Controller for Grasping Motion and Force

In this section, the result of the hand motion mapping to the robot hand is pre-sented by means of a simulation and experiments. First, we demonstrated thatthe robot can imitate human motions during a simulation using the physics-based simulation software RoboticsLab [19]. Fig. 8 shows the configurations ofthe robot hand when the operator grasped the drinking cup using the proposedalgorithm, the PCA-based algorithm, and the RPCA-based algorithm. The pro-posed algorithm showed a grasping motion that was more similar to the originalhand motion used in grasping the drinking cup than the other algorithms.

We also demonstrated the performance of the proposed algorithm when therobot hand tracks a human hand motion in real-time using the synergy-levelcontroller. Fig. 9 shows an image taken during the grasping of the tennis ballusing the first two synergies by the proposed algorithm, by the PCA algorithm,and by the RPCA algorithm. First, the posture of the robot hand using the pro-posed algorithm was much more similar to that of the operator than the postureof the robot using other algorithms when a thumb-index grasp was performed.Also, the ball was successfully grasped using only the proposed algorithm.

Second, the performance of the proposed algorithm with an EMG signal wasalso demonstrated through experiments. As shown in Fig. 10, the robot handproduced grasping force due to the EMG signal while the posture of the humanhand did not change.

10 Kim et al.

(a) (b)

(c) (d)

Fig. 8: The comparison of postures when the operator grasps the drinking cup(The power disk grasp); (a) The posture of human hand, (b) The reconstructedposture of robot hand using the proposed algorithm, (c) The reconstructed pos-ture of robot hand using the PCA-based algorithm, (d) The reconstructed pos-ture of robot hand using the RPCA-based algorithm.

(a) (b) (c)

Fig. 9: The comparison of postures when the operator grasps pen (thumb-indexgrasp); (a) The proposed algorithm, (b) PCA, (c) RPCA.

(a) (b)

Fig. 10: The comparison of the robot grasping; (a) Without the EMG signal, (b)With the EMG signal.


5 Conclusion

In this paper, a novel synergy-based mapping algorithm for a robot hand whichuses a multi-factor model and an EMG signal is proposed. The main experimen-tal points of insight are summarized as follows. First, we extracted synergies thatrepresent human hand motion more accurately by accounting for the character-istics of individuals. Second, we demonstrated that the synergies of a new usercould be obtained simply by extracting the factor pertaining to the new user.Third, the posture of the robot hand using the proposed algorithm was moresimilar to the human hand than that of using other algorithms. This thereforeenabled dexterous manipulation through tele-operation. Finally, the graspingforce of the robot can be generated by using an EMG sensor. Future works willinvolve extending the workspace by integrating the robot hand with an arm,and simplifying the system by replacing the motion-capture system with othercompact and practical solutions. Also, an algorithm to extract the grasping forceof a human by means of EMG will be investigated.

Acknowledgements

This work was supported by the Global Frontier R&D Program on Human-centered Interaction for Coexistence through the National Research Foundationof Korea. (NRFM1AXA003-2011-0032014)

References

1. Hong, J., Hollerbach, J., Jacobsen, S.: Calibrating a VPL dataglove for teleoperat-ing the Utah / MIT hand: In: IEEE Int. Conf. on Robotics and Automation, pp.1752–1755 (1989)

2. Rohling, R., Hollerbach, J.: Optimized fingertip mapping for teleoperation of dex-terous robot hands. In: IEEE Int. Conf. on Robotics and Automation, pp. 796–775(1993)

3. Griffin, W., Findley, R., Turner, M., Cutkosky, M.: Calibration and mapping ofa human hand for dexterous telemanipulation. In: ASME IMECE. Conf. HapticInterfaces for Virtual Environments and Teleoperator System Symposium, pp. 1–8(2000)

4. Ozawa, R., Ueda, N.: Supervisory control of a multi-fingered robotic hand systemwith data glove. In: IEEE/RSJ Int. Conf. Intelligent Robots and System, pp. 1606–1611 (2007)

5. Santello, M., Flanders, M., Soechting, J.: Postural hand synergies for tool use.Journal of Neuroscience, 10105–10115 (1998)

6. Ciocarlie, M. T., Allen, P. K.: Hand posture subspaces for dexterous robotic grasp-ing. Int. Journal of Robotics Research, 28(7), pp.851–867 (2009)

7. Matrone, G. C., Cipriani, C., Secco, E. L., Magenes, G., Carrozza, M. C.: Principalcomponents analysis based control of a multi-dof underactuated prosthetic hand.Journal of NeuroEngineering and Rehabilitation, 7(1) (2010)

12 Kim et al.

8. Ajoudani, A., Godfrey, S. B., Catalano, M., Grioli, G., Tsagarakis, N. G., Bicchi,A.: Teleimpedance control of a synergy-driven anthropomorphic hand. IEEE/RSJIntelligent Robots and System, pp. 1985–1991 (2013)

9. Romero, J., Feix, T., EK, C. H., Kjeelstrom, H.: Extracting postural synergies forrobotic grasping. IEEE trans. on robotics, 29(6), 1342–1352 (2013)

10. Pedersen, DR., Arora, JS., Brand, RA., Cheng, C.: Direct comparison of muscleforce predictionjs using linear and nonlinear programming. Journal of BiomedicalEnginearing, 109(3) pp. 192–199 (1987)

11. Cichocki, A., Zdunek, R., Phan, A. H., Amari, S.: Nonnegative matrix and ten-sor factorizations: Applications to exploratory multi-way data analysis and blindsource separation. Wiley, New York (2009)

12. Kolda, T. G., B. W.: Tensor decompositions and applications. SIAM Review, 51,455–500 (2009)

13. Konrad, P.: The abc of EMG. A practical introduction to kinesiological electromyo-graphy, 1 (2005)

14. SimLab.: Allegro Hand is a low-cost and highly adaptive robotic hand.http://www.simlab.co.kr

15. Dai, W., Sun, Y., Qian, X.: Functional analysis of grasping motion. IEEE/RSJIntelligent Robots and System, pp. 3507–3513 (2013)

16. Berndt, D. J., Clifford, J.: Using dynamic time warping to find patterns in timeseries. In KDD workshop, 10(16) pp. 359–370 (1994)

17. Jolliffe, I.: Principal component analysis. Wiley Online Library (2005)

18. Candes, E. J., Li, X.: Robust principal component analysis?. Journal of the ACM,58(3) (2011)

19. Simlab, http://www.rlab.co.kr

Appendix: Tensor Representation

This section is a condensation of [11] and [12]. For details of tensor, please readthe references.

A tensor is a higher-order generalization of the vector (first-order tensor) andmatrix (second-order tensor). When we used a matrix to represent motion dataset, the rows usually contained the channels for the joint angles, and the columnsfor motion samples. However, when we considered the multiple factors of humanand used a tensor framework, the motions were grouped by each factor so thatthey constituted an Nth-order tensor.

The order of tensor Y∈ RI×I×...×IN is N. The mode-n vectors of an Nth-order tensor Y are defined as the In-dimensional vectors obtained by vary-ing index In while keeping the other indices fixed. All mode-n vectors can bearranged together as column vectors to compose a mode-n flattening matrixYn ∈ RIn×(II,...,In−In+,...,In). The In-dimensional vectors of Yn are obtainedfrom tensor Y by varying index In while keeping other indices fixed.

The multiplication of a high-order tensor Y∈ RI×I×...×IN by a matrix A ∈RJn×In is a mode-n product of tensor Y by A, which is denoted as Y ×n A. Itcan also be expressed in terms of flattened matrices. The entries of the productare given as


(Y×n A)i1...in−1jnin+1...in =∑in

di1...in−1inin+1...inajnin (11)

The tensor decomposition of Y seeks for N orthonormal mode matrices as Eq.(12), which is obtained by HOSVD.

Y = G×1 U1 ×2 U2...×n Un (12)

The column vectors of An are the orthonormal basis vectors of the mode-n un-folding matrix Yn. Core tensor G governs the relationship among mode matricesUn.

Consequently, tensor representation is helpful to treat multi-factorizationproblem, as matrix is able to decompose factor using non-negative matrix fac-torization such as PCA and NMF.

robot hand synergy mapping using multi-factor model and emg...

Documents