Automatic Labanotation Generation

Based on Human Motion Capture Data

Hao Guo1, Zhenjiang Miao1, Feiyue Zhu2, Gang Zhang2, and Song Li2

1 School of Computer and Information Technology, Beijing Jiaotong University,Beijing, China

{12120339,zjmiao}@bjtu.edu.cn2 Center for Ethnic and Folk Literature and Art Development,

Ministry of Culture, Beijing, Chinazhufeiyue@aliyun.com,13801240583@139.com,13051270916@163.com

Abstract. As a kind of dance notation, Labanotation has been adoptedextensively as an analysis and record system for performing dances. Thisarticle aims to generate Labanotation automatically from human motioncapture data stored in BVH (Bio-vision Hierarchy) files. First, we con-vert motion capture data into position format. Then we analyze motionsseparately according to whether the motion belongs to supporting mo-tion or not. Using the obtained Primary Motion Segments, a sequenceof coded description of Labanotation - the Labanotation Data (LND) -is built. And finally, Labanotation is drawn and stored correctly on thebasis of LND.

Keywords: motion capture data, Labanotation, motion segment, mo-tion analysis, BVH.

1 Introduction

Labanotation is to dance what musical notation is to music. It is a vivid, scientificand logical analysis and record system for human body motion. And it plays animportant role in the intercultural communication for dance arts. But drawingof it is time-consuming and inconvenient because of high degree of freedom ofhuman body joints and dependencies of different body parts.

The automatic generation of Labanotation will simplify the drawing processin order to promote the communication of dance arts. It can also be used topreserve folk dance arts that are in danger of dying out. Besides, Labanotationsequences generated from motion capture data may serve as a kind of motionfeature for motion retrieval.

Calvert et al. [1] introduced several applications that assist in drawing La-banotation. Laban Writer [2] is the most commonly used application. There arealso similar applications such as Laban Editor [3]. Hachimura and Nakamura [4]proposed the spatial relationship between notation characters and body partsmovements. But it was limited to recognition of motions of arms and ignored legssupporting motions. Chen et al.[5] presented a system which transfers motion

S. Li et al. (Eds.): CCPR 2014, Part I, CCIS 483, pp. 426–435, 2014.c© Springer-Verlag Berlin Heidelberg 2014

Automatic Labanotation Generation Based on Human Motion Capture Data 427

Fig. 1. Staff and the characters of Labanotation: (a) columns for Labanotation (b) 27notation characters: place, forward, left forward, right forward, left, right, left back,right back and back. Each in the levels of low (black), middle (dot) and high (slash).

capture data to Labanotation. Shen, Li, Yu, Geng and Zhang [6],[7],[8] exploredthe transformation from motion capture data to Labanotation and motion retrialwith Labanotation. However, their generating results are distinct with actual La-banotation.

This article aims to generate Labanotation automatically and correctly frommotion capture data. With instruction of Professor Luo Bingyu, an expert onLabanotation, we gave full consideration in rules of Labanotation when carryingon our research. We parse BVH file and analyze human motions. Then thePrimary Motion Segment (PMS) defined as basic unit for composing each ofhuman action can be extracted. The LND is produced by these processes. Theoutput results are both pictures of Labanotation and LW4 files [9].

2 Labanotation and Motion Capture

Labanotation. Rudolf Laban created Labanotation in early 20th century. Ac-cording to comprehensive interpretations in [10] and [11], it is a kind of verticalstructure written and read from the bottom up. The staff is a format of 3 lineswith 11 columns. The middle line represents spine of human body as shown inFig.1(a). Laban divided human body motion into gaits and poses. Gaits in sup-porting columns represent motions of gravity center of human body. Poses innon-supporting columns express motions of body parts.

There are 27 primary characters of Labanotation expressing 27 orientations:9 directions and 3 levels. The prototype of each notation character is a rectan-gle, which means the place orientation. And the other characters derive fromit by cutting right parts of it, as shown in Fig.1(b). Details refer to [11], anauthoritative masterpiece.

Motion Capture. We used two kinds of optical motion capture system toobtain motion capture data, BVH file. The one with markers is for indoor use

428 H. Guo et al.

Table 1. Devices of OptiTrack Motion Capture System

OptiTrack Device Name Model Count

Hardware Camera and platform Prime 41 8Software Software platform and hardware key Motive: Body 1Accessories Suits and Markers 1

Calibration tools 1

Fig. 2. (a) Motion capture system with markers: cameras on top (b) Environment formotion capture without markers

Fig. 3. Skeleton structure for motion data

only. And the one without markers is used under various circumstances. Butmotion capture system with markers is more precise.

We were offered with OptiTrack, the motion capture system with markers, asFig.2(a) shows. It mainly contains devices in Table 1. And the motion capturewithout markers is designed and implemented by other members of our team[12], [13], [14]. It contains 8 cameras and 1 workstation, Fig.2(b).

A BVH file, motion data, is divided into 2 sections. A keyword HIERARCHYin the first line signifies the skeleton definition. The other section, saving motiondata, starts with MOTION. Motion data is a matrix, of which each row is aframe of data recording a posture of human body at the moment of capturing.

The hierarchy, a depth-first tree structure, defined in BVH file specifies themean-ing of each data in motion section. The basic unit in the hierarchy is ajoint, a braced block with attributes of offset, channels and children joints. 23joints make up the hierarchy of BVH file in this article shown in Fig.3.

Automatic Labanotation Generation Based on Human Motion Capture Data 429

Table 2. Two data structures

PMS LND

Duration start frame start time Durationend frame end time

Orientation direction direction Orientationlevel level

Body Part Laban Column

3 Motion Analysis

We defined two data structures: PMS as basic motion unit for motion data andLND. Shown in Table 2.

3.1 Data Converting

The meanings of joints in hierarchy are identified by the structure rather thanthe names of them. Joints in hierarchy should be calibrated with the humanbody part. According to Fig.3, we devised the calibrating method.

1. Traverse from ROOT depth-first. Find the first joint, R, with 3 childrenjoints, J1, J2 and J3. R is the joint of hips. And J1, J2 and J3 represent thestart of left leg, right leg and torso.

2. Compute each depth and joint count of branches that start with J1, J2 andJ3. Considering the symmetry of human body, 2 branches, starting withjoints A and B, share the same depth and joint count. The third joint, C, isthe start joint of chest branch.

3. Contrast initial offset of joint A and B. As the skeleton structure in BVHfaces to the direction of positive z-axis in Cartesian coordinates. The jointwith initial Offset.x < 0 is the start joint of right leg. And the joint withOffset.x > 0 is the start joint of left leg.

4. Traverse from C. Find a second joint with 3 children joints. Identify the startjoints of left arm, right arm and head with the same processes in 2 and 3.

The motion data in BVH files records human body motion with Euler Angle[15]. To facilitate motion analysis, we convert Euler Angle to position data.Assume a joint, P , and its child joint J with (x0, y0, z0) as initial relative offsetto P . Its relative offset changes to (x1, y1, z1) after P rotating. Suppose therotation matrix of P is M . Then the new relative offset after rotation is

Offset(x1, y1, z1) = M · Offset(x0, y0, z0) (1)

Considering all ancestor joints of J , offset of J (to P ) in world coordinate systemis computed. Assume Mi(i = 1, ..., A) represent the rotation matrix of ith parentjoint of J from bottom up.

Offsetnew = MA · ... · [M2 · (M1 · Offsetinitial)] (2)

430 H. Guo et al.

Fig. 4. The partition of space (a) four height space (b) 9 horizontal space correspondingwith 9 Labanotation directions (c)3 vertical space corresponding with 3 Labanotationlevels

3.2 Supporting Analysis

Supporting analysis is designed for analyzing gaits of human body. Notationcharacters in supporting columns of Labanotation describe the tendency of eachprimary motion rather than the posture after the motion.

Extraction of Support-DurationsWhen human body is moving, it is either supported or unsupported. Suppose theframe number of motion data is N . We defined the supporting state, SSi(i =1, ..., N). It signifies which part (or parts) of the body is (or are) supporting.A threshold is used to determine whether a body part supports body or not.Empirically, when a foot is higher than 3.4 inches from the ground, this foot isoff the ground and does not support the body. Each state lasts a period of time.So we integrate consecutive identical supporting states into a support-duration,SDi(i = 1, ...,M). Suppose M is the number of support-durations.

Extraction of Motion-Durations: Direction ObtainedMotion data is a high-dimension time series. The variation between adjacentframes represents the change of body posture. Thus, the velocity vi(i = 1, ..., N),which is the adjacent frame difference, can be used to express the tendency ofhuman body motion. We classified velocities into 9 directions by analyzing thehorizontal pointing of velocity vectors as Fig.4(b) shows. The circle in Fig.4(b)means PLACE direction that occurs when the angle between velocity and y-axisis among [0, 20) ∪ (160, 180].

This yields horizontal directions of orientations in PMS between every pair ofadjacent frames. Similar with the extraction of support-durations, consecutiveidentical directions are merged into motion-durations MDi(i = 1, ...,K).

Isolating Primary Motion Segments: Duration ObtainedThe way to isolate PMS from motion-durations and support-durations is shownin Fig.5. Duration of each PMS is supplied by this.

Respective Analysis: Levels and Moving Part ObtainedWe generalize 3 types of supporting motion based on the motion mode. Theyhave different features. The respective analysis completes each PMS.

Automatic Labanotation Generation Based on Human Motion Capture Data 431

Fig. 5. Isolating PMS from MD and SD

Fig. 6. Relationship between movements of foot and whole human body

Vertical up and down - This type of motion is recorded with the notationcharacter PLACE. The height that human body at is the basis for level judgmentas shown in Fig.4(a). According to our observation, the level is in MID levelwhen difference between the height human body at and its initial height locatesin (−1, 1) inch. Lower is in LOW level. And higher but no more than 3.4 inchesis in HIGH level. Besides, the moving part in vertical up and down is the bodypart supporting the body. This is determined according to SD above.

Stepping - The level of PMS is determined by the method shown in Fig.4(c).Moving part of PMS is the foot which results in this primary motion. The movingdirection of human body is the same with direction of this foot. As Fig.6 shows,when human body moves to right forward, in one case, the right foot stays stillwhile left foot moves the same direction. Thus, it is the left foot lead to humanbodys motion.

Leaping - The level of this motion is determined with the same way as step-ping motion. We distinguish the foot that drives this leaping motion with thesupporting state of previous frame. And this supporting foot is the moving partof PMS.

3.3 Non-supporting Analysis

Non-supporting analysis includes analyzing poses of five active branches: head,two arms and two legs. As each part is analyzed separately, the moving part inPMS needs no judgment. Duration is obtained from motion segmenting. Orien-tation derives from orientation determining.

Motion SegmentingMotion segmenting is aim to segment the motion data sequence at the borderbetween different motions. Barbic et al. [16] proposed 3 approaches, PCA, PPCAand an approach based on GMM, to segment motion capture data into distinctbehaviors. In this article, we use PCA method which is easy to implement withcertain accuracy.

432 H. Guo et al.

Fig. 7. Comparison of manual segmenting and PCA segmenting

Suppose a branch with m joints. Then each frame of data, xi(i = 1, ..., N), isa point in the space of M = m× 3 dimensions. We can get a diagonal matrix Σwith the Singular Value Decomposition (SVD). Suppose the diagonal elementsare σi(i = 1, ...,M). Project xi to the specified r-dimension space and defineprojection error.

e = ΣNi=1‖xi − x′‖2 = ΣM

j=r+1σ2j (3)

R = 1− e/(ΣMj=1σ

2j ) (4)

Set a threshold, τ = 0.95, compute the minimum r to make R > τ . Assume eirepresents the projection error for first i frames of motion capture data. ei willremain almost unchanged if the motion data describe a singular primary motion;but it changes rapidly at a frame between adjacent different primary motions.This frame is the border of different motions. Fig.7 shows the comparison ofPCA segmenting and manual segmenting.

Orientation DeterminingThere are Initial Joint and Terminal Joint for each branch. For example, theinitial joint of left arm is the joint of LeftUpArm. And the terminal joint ofleft arm is the joint of LeftHand. It is the relative position of Initial Joint andTerminal Joint that is used to determine the orientation of PMS. As shown inFig.4(b) and (c), the Initial Joint locates at the center. Then we inspect theposition of Terminal Joint. It must locate in one of the 27 space regions. Thus,the orientation in each PMS is obtained.

3.4 Build LND

Just as the definition of PMS and LND, they share the same structure. But thereis still a problem that is the redundancy of LND. It is not absolutely precisebecause of the deviations in processing, like errors of segmenting in Fig.7. So wemerge LND sequence under some rules.

Progressive motions. A singular continuous primary motion with different ori-entations in its moving may be determined as two primary motions, in Fig.8(a).

Sustaining motions. A continuous primary motion is divided into two becauseof deviation of segmenting, in Fig.8(b).

Nonexistent motions. Noises in motion capture data may result in nonexistentmotions, meaningless and too short to express a primary motion, in Fig.8 (c).

Cracked motions. Noises may also lead to a singular primary motion cracked,in Fig.8(d).

Automatic Labanotation Generation Based on Human Motion Capture Data 433

Fig. 8. Four merged situations: unmerged left and after merged right

Fig. 9. Method of drawing Labanotation characters

4 Generating Results

This project was developed on Microsoft Visual Studio 2008 with C++ programlanguage and OpenCV library under Windows system. It reads a piece of BVHfile. With some musical parameters, the project generates Labanotation afterparsing BVH file and analyzing human motion performed in it. We use MFC toolsto draw and display Labanotation. The character of PLACE is the prototype ofall other characters, shown in Fig.9.

Fig.10 shows two examples of generated Labanotation. The left piece describesa period of walk motion. And the right one describes motions of some times ofleaping with arms swing. To verify the usability of this project, we comparegenerated Labanotation with its original motions in Fig.11. This piece of Laban-otation is the left part of Fig.10. It records a period of walking which lasting 693

Fig. 10. Two examples of Labanotation, generated automatically

434 H. Guo et al.

Fig. 11. Comparison between Labanotation and original motions

frames with 60 FPS. In this motion duration, the human body walks forwardand strides sometimes.

As Fig.11 shows, 8 actions in Labanotation (left) and 8 original postures(right) are obtained by picking one about every 100 frames. Action 1 is theinitial state of human body. Compare it with Posture 1. Both legs support thehuman body. The right arm towards right while left arm biases front. This makesup an action just like Action 1 expressing. Similarly, Action 3, 4, 5, 6, 7 and 8can be compared separately with Posture 3, 4, 5, 6, 7 and 8. We can find eachpair of them matches well. Specially, it is move-less between Action 1 and 2. SoPosture 2 is the same with Posture 1.

According to the result of comparisons, the generated Labanotation can ex-press motions recorded in motion capture data correctly. Then we can save thegenerated Labanotation. Besides the picture displayed Fig.10 and Fig.11, it canalso be saved in LW4 file, defined by Laban Writer 4.0. It is a kind of text fileencoded in ASCII format.

5 Conclusions

The generating results indicate that we are able to generate Labanotation frommotion capture data correctly. We first used two types of motion capture systemto capture human body motion. Then the motion capture data in BVH files isprocessed. After analysis of both supporting and non-supporting motion, we getPMS as basic motion unit. Hence, LND, the coded description of Labanotation,is built. And finally, generating Labanotation automatically is implemented.

The next step of our work is to make the generating more accurate and developour own file format to save Labanotation with reader software for it.

Automatic Labanotation Generation Based on Human Motion Capture Data 435

Acknowledgments. Our work is supported by National Key Technology R&DProgram of China 2012BAH01F03. We thank Professor Luo Bingyu for instruc-tion on Labanotation.

References

1. Calvert, T., Wilke, W., Ryman, R., Fox, I.: Applications of computers to dance.IEEE Computer Graphics and Applications 25(2), 6–12 (2005)

2. Venable, L., Sutherland, S., Ross, L., Tinsley, M.: Laban Writer 2.0. The OhioState University, Department of Dance (1989)

3. Kojima, K., Hachimura, K., Nakamura, M.: Labaneditor: Graphical editor fordance notation. In: Proceedings of the 11th IEEE International Workshop on Robotand Human Interactive Communication, pp. 59–64. IEEE (2002)

4. Hachimura, K., Nakamura, M.: Method of generating coded description of humanbody motion from motion-captured data. In: Proceedings of the 10th IEEE Inter-national Workshop on Robot and Human Interactive Communication, pp. 122–127.IEEE (2001)

5. Chen, H., Qian, G., James, J.: An Autonomous Dance Scoring System UsingMarker-based Motion Capture. In: 2005 IEEE 7th Workshop on Multimedia SignalProcessing, pp. 1–4. IEEE (October 2005)

6. Shen, X., Li, Q., Yu, T., Geng, W., Lau, N.: Mocap data editing via movementnotations. In: Ninth International Conference on Computer Aided Design and Com-puter Graphics, p. 6. IEEE (December 2005)

7. Yu, T., Shen, X., Li, Q., Geng, W.: Motion retrieval based on movement notationlanguage. Computer Animation and Virtual Worlds 16(3-4), 273–282 (2005)

8. Zhang, S., Li, Q., Yu, T., Shen, X., Geng, W., Wang, P.: Implementation ofa notation-based motion choreography system. In: Zha, H., Pan, Z., Thwaites,H., Addison, A.C., Forte, M. (eds.) VSMM 2006. LNCS, vol. 4270, pp. 495–503.Springer, Heidelberg (2006)

9. LabanWriter 4 file format, http://dance.osu.edu/labanwriter/resources/labanwriter-4-file-format

10. Xiang, B.: The scientific notation of movement: The principle of science and func-tion of labanotation. Master’s thesis, Shanghai Normal University (2008)

11. Guest, A.H.: Labanotation: The system of analyzing and recording movement.Psychology Press (2005)

12. Wan, C.: Research on human body motion capture and pose estimation. Disserta-tion. Beijing Jiaotong University (2009)

13. Sun, Y.: Research on Touch-free Human Body Motion Capture Under MultipleViewpoints. Dissertation. Beijing Jiaotong University (2006)

14. Li, J.: Research on Multiview 3D Human Motion Capture. Dissertation. BeijingJiaotong University (2013)

15. Dunn, F., Parberry, I.: 3D math primer for graphics and game development. CRCPress (2011)

16. Barbic, J., Safonova, A., Pan, J.Y., Faloutsos, C., Hodgins, J.K., Pollard, N.S.: Seg-menting motion capture data into distinct behaviors. In: Proceedings of the 2004Graphics Interface Conference, pp. 185–194. Canadian Human-Computer Commu-nications Society (May 2004)