[ , ] autonomous human robot interactive skills

8/10/2019 [ , ] Autonomous Human Robot Interactive Skills

1/14

Autonomous Human-Robot Interactive Skills

A Collaboration Project between KAIST and Stanford University

Principle Investigators

Prof. Ju-Jang Lee (KAIST) and Prof. Oussama Khatib (Stanford University)

Participating Researches

KAIST

Dr. Sudath R. Munasinghe

Mr. Chang-Mok Oh

Stanford University

Mr. Jaeheung Park

Mr. James Warren


2/14

1. Introduction

Objectives: The objective of this project is to develop basic autonomous skill primitives

for robots to be able to inhabit and corporately behave in human-populated environments.

These primitives will be organized in a task behavior library (TBL), allowing for intuitive

task-level robot programming. TBL will provide a set of standard task templates that can

be used in various robot behaviors by instantiating them with the appropriate control

primitives, parameters, and properties.

Motivation: Humans have a variety of sensing and perception capabilities including,

visual, auditory, and tactile, which allow them to outperform machines and compensate

for the generic lack of control resolution. There is an increased interest in exploring

methodologies to mimic, or actually transfer human skills to robotic systems, and to

make those systems capable of performing complex tasks.

Methodologies: The transferred human skills would be essential for robotic systems to

behave in human-populated environments, and also to perform complex tasks that are

ascribed to human capabilities. Proper behavior in human-populated environments is

particularly essential for service robots, where the capability to perform complex human-

like tasks is particularly essential for robotic substitutes for humans.

We have concentrated on two major approaches to implement advanced robotic

capabilities; (1) model-based approach, and (2) human demonstration. Model-based

approach takes an insight into the specific tasks(s), and tries to describe and model the

skilled human behavior precisely. Using these models, human skills can me implemented

on robots with required strategies and parameters. Human demonstration is a direct skill

transfer based on sensory-motor relationships, and motion patterns of human motions

such as hand manipulation, gait-skills etc. Human demonstration can motion pattern can

be easily recorded by a motion capture facility.


3/14

2. Model-based Approach

The major difficulty with the extension of a robotic task to human-populated

environments lies in the various uncertainties of the real world. Our approach to deal with

these constrained motion uncertainties is to rely on sensor-based compliant strategies. In

assembly task for example, contacts can be recognized when they occur and take the

advantage of them in compliant way to guide the object in the assembly task. Sensor data

can be used during motion to adapt the path to the actual geometry of the moving objects

(e.g. sliding along the surfaces). Sensing can also be used to decide when to stop a

motion and then to select the next move if the goal has not been achieved. The

manipulation task primitives are parameterized by a compliance frame, operational point,

and other task related parameters. By selecting these parameters appropriately, we can

instantiate the basic skill primitives in many different ways to adapt to the needs of a

specific task. We have implemented such strategies for several simple tasks including

insertion, object stacking, and surface following. In this project, we are focusing on new

strategies for compliant motion tasks in order to develop models for advanced primitives

that allow complex tasks to be specified at a higher level of abstraction.

2.1 Advanced Skill Primitives

2.1.1 Mobile Manipulation

Obstacle Avoidance and Real-Time Path Planning:

Most motion planning algorithms presumes that the environment is completely known at

planning time [1]. Most of them actually assume that the world will not change in the

future, implying that all obstacles are static. Some of them allow for moving obstacles,

with known or predictable motion functions. To generate a motion those algorithms build

a representation of the obstacles in the configuration space of the robot. The

configuration space is the space describing all possible spatial positions and arm

configurations of the robot. If a particular configuration, described by the spatial positionof the robot and its arm configuration is in collision with the environment, it does not

correspond to a physically possible configuration. If we are dealing with robots with

many joints, the configuration space is high-dimensional and the computation of

configuration space obstacles, or even approximations to it, is computationally very

costly and can take from multiple seconds to hours. Planning algorithms that also allow

for obstacles moving on a known trajectory augment the configuration space with another

dimension: time.


4/14

Motion planning is performed by finding a continuous curve connecting the initial and

the final configuration of the robot. Sounds simple, but it can get pretty complex. This all

works fine until an unforeseen obstacle appears, or a known obstacle does not move as it

was assumed during planning. Now the motion generated by the motion planning

algorithm might not be valid any more. The environment has changed and the

assumptions that lead to that particular motion do not hold any more. The solution is to

plan a new motion from scratch, given the new information about the environment. If a

robot is supposed to accomplish a task in a space that changes frequently, however, this is

not practical any more.

For a robot to be able to inhabit human-populated environments, it is required that the

robot be able to cope with unpredictable changes in the environment. To allow the

execution of motion plans in dynamic environments the global planning process can be

augmented with a fast, reactive obstacle avoidance component. With this augmentation,

an initial motion plan can be generated under the assumption that all obstacles are known

[2]. During execution the robot deviates from the planned path, guided by potential fields

caused by previously unknown or moving obstacles. The planner attempts to rejoin the

original path at a later point that remains unaffected by changes in the environment.

Deviation from the pre-planned path, however, can result in local minima that require

replanning [3]. To overcome this problem, the concept of elastic bands was introduced

[4] in that a path is represented as a curve, called elastic band, in configuration space,

and it is incrementally modified according to the position of obstacles in the environment

to maintain a smooth and collision-free path. Figure 1 illustrates the elastic path being

planed for PUMA robot on a mobile base.

Fig. 1 Elastic path-planning for obstacle avoidance [Stanford mobile platform]

At each workspace location of the mobile manipulator, collision-free space around each

rigid link is determined by integrating the collision free bubbles along the spine of that

rigid link. These collision-free hyperspaces make the protective hull [5], and by covering

entire elastic band with overlapping protective hulls we can guarantee that the path it


5/14

represents is collision-free. However, as the number of degrees of freedom of the robot

increases, the estimate of the local free space around a configuration would require

consideration of an excessive amount of bubbles, which affects real-time performance.

By defining bubbles in robot workspace, considering the distances to of the obstacles will

greatly reduce this drawback. Let )(p be the function that computes the minimum

distance from a point pto any obstacle, then the workspace bubble of free space around p

is defined as { })(:)( pqpqp


6/14

as given by

+=

+

)()( 1111

1

int

,

i

j

i

j

i

j

i

ji

j

i

j

i

j

cjidd

dk ppppF , where

1+=

i

j

i

j

i

jd pp is the

distance in the initial original trajectory. These forces cause the elastic stripe to

contract and to maintain a constant ratio of distances between every three

consecutive configurations. Meanwhile, the obstacles cause external forces due to

their potential fields


7/14

For mobile manipulation, some degrees of freedom are used for task execution

while others can be used to achieve task-independent motion behavior. This is

achieved by unifying motion and force control behaviors of redundant

manupulatiors [7]. Redundant manipulator dynamics is given by

0)]()([)( += qqIFq

TTT JJJ where the first term on RHS is the torques

due to the forces action at the end-effector, whereas the second term on RHS

affects internal motion. This decomposition can be used to achieve advanced

motion behaviors such as mobile manipulation. To ensure the execution of a task

specified in a particular task frame f, the internal and external forces are mapped

into the null space of the Jacobian Jf associated with the task frame. This

corresponds to the sets of tasks where the end-effector is required to move on a

certain trajectory and the redundant degrees of freedom can be used for obstacle

avoidance. We adapt the adapt the operational space formulation [8] to realize mobile

manipulation, in that the position and orientation of the end-effector is controlled

independent of the base motion and other redundant joints, which can be used for

obstacle avoidance. However, task execution has to be suspended temporarily when

and while task-consistent motion behavior is infeasible due to kinematic

constraints or change in the environment. Therefore, it is necessary to develop a

criterion to determine task suspension and resumption. Let

)]()([))(( qqqTTT JJIJN = be the dynamically consistent nullspace mapping

of the Jacobian J(q) associated with the task. The coefficient0

0))((

=

qJNc

T

corresponds to the magnitude ratio of the torque vector 0 before and after

mapping into the nullspace, thus indicates how well the behavior be implemented

within the nullspace of the task. We use an experimental lower limit scc < at

which it becomes desirable to suspend the task in favor of the behavior. Task

suspension is gradually, as a transition. The reverse transition takes place to

resume the task at an upper value rcc > . Unnecessary transitions can be avoided

by introducing a dead-band rs cc < . The mobile manipulation experiment with

the Stanford mobile platform is shown in Fig. 4, in that the end-effector maintains


8/14

its straight-line position profile (which is the specified task) within the entire

motion, while the mobile base adapts repulsive curvatures to avoid an incoming

obstacle.

Fig. 4 Mobile manipulation: keeping end-effector along a straight-line (task) while

avoiding collision with unforeseen obstacles

Figure 5 shows (1) Real-time obstacle avoidance without specified end-effector task, (2)

Mobile manipulation with end-effector task (without suspension), and (3) Mobile

manipulation with task suspension

Fig. 5 Mobile manipulation: Task suspension and resumption (3rdtrial)

First trial shows only obstacle avoidance under no task specified at the end-effector.

Econd trial has a task specified to keep the end-effector along a straight line. The obstacle

doesnt come extremely close, therefore the end-effector task is maintained witout


9/14

suspension. However, in the third trial, the task has to be suspended for a while until the

obstacle moves away.

The task and posture decoupling can be used to realize skilled behavior as illustrated in

Fig. 6

Fig. 6 PUMA 560 in vacuuming the floor, opening a door, and ironing a fabric.

These skilled behaviors involve a specific task defined at the end-effector in terms

of position, orientation and contact force. The task is satisfied independent of the

overall posture of the arm.

2.1.2 Human-Robot Cooperative Skills

In human-Robot cooperative skills, our investigation is to develop protocols for tactile

communication and guided motion skills for human-robot cooperative tasks. Guided

motion involves tight cooperation achieved through compliant motion actions, or looser

free-space motion commands. The robot for instance, may support a load while being

guided by the human to an attachment. These motion skills rely on the interaction skills

and the guided-motion primitives. The issues involved in human-robot cooperation havesimilarities with those associated with multi-robot cooperation. The investigation of

guided motion primitives is influenced by the decentralized control behaviors we have

developed for cooperative robots. In decentralized corporation of two robots, each robot

relies on a model based on an augmented load that takes into account the inertial

properties and the dynamics of the other robot. Relying on the interaction skills and using

a simplified model of the human arm inertial properties, we are going to investigate a


10/14

similar approach to human-robot cooperation. This approach will provide the basis for

the development of effective human-robot guided-motion primitives.

Our approach is based on the concepts of virtual linkage [9] and augmented object [10].

The virtual linkage characterizes the internal forces of the cooperative task, whereas

augmented object describes the closed-chain dynamics of the system. The two concepts

are briefly described as follows:

Virtual linkage and augmented objects:

The model for internal forces for multi-grasp manipulation is illustrated in Fig. 7 below.

Fig. 7 Virtual linkage concept for multi-robot cooperation in three-grasp manipulation

task

In this model, grasp points are connected by a closed, non-intersecting set of virtual links.

For N-grasp manipulation task, the virtual linkage model is a 6(N-1) degree of freedom

mechanism that has 3(N-1) linearly actuated members and Nspherically actuated joints.

By applying forces an torques at the grasp points we can independently specify internal

forces in the 3(N-2) linear members and 3Ninternal moments at the spherical joints. The

internal forces of the object are then characterized by these forces and torques in a

physically meaningful way. The relationship between applied forces, their resultants, and

internal forces is

=

N

res

f

f

GF

FM

1

int

, where resF is the resultant force at the operational


11/14

point, intF is the internal forces, if is the forces applied at the ith grasp point, and

G is the grasp description matrix. The inverse of the grasp description matrix

provides the forces required at the grasp points to produce the resultant and

internal forces

=

int

1

1

F

FG

f

fres

N

M acting on the object. The resultant force on the

object describes the required motion behavior, whereas internal forces refer to the

specified task at the object. Once these two quantities are specified, the individual

grasp-point forces can be determined using 1

G . Further, the grasp point forces

can be decomposed into corresponding motion and task components for each

robot as taskimotionii ,, fff += and implemented using a decentralized control

structure as shown below

Fig. 8 Decentralized control for autonomous robots in cooperative tasks

Figure 9 illustrates the cooperative behavior of two PUMA 560 robots based on

the augmented object and virtual linkage concepts. The decoupled task and

posture (motion) control is also included in the decentralized control structure.


12/14

Fig. 9 Two PUMA 560 robots in a cooperative task

The decoupled motion and force (task) control together with the decentralized

behavior make the skill primitive for a robot to b able to cooperate with other

robots and human operators in a highly skilled manner [11]

3. Direct Skill Transfer

Direct skill transfer can be generally applied to a variety of human skills. It does not

require an insight, or a model of the particular skill. Human demonstrated data can be

learned by a neural network, and then implemented on robots. In this approach, we view

human skills in terms of spacio-temporal motion patterns. We could define human skills

as proper combination of joint motion profiles.

Our approach here is to capture the kinematics of human demonstration and analyze it to

reveal the underlying organization of the skill human motions. We are first considering

how a human leaves a cup on a table quite easily, without damaging both of the objects,

and also in no time of thinking and planning. On the other hand, it is a very difficult task

for a robot to perform without the three-dimensional vision information, and advanced

contact force sensory system that the humans are equipped with. Object manipulation

under constrained environments requires compliance motion, in that the contact

information itself is used as important information in guiding the object towards the

target position. Therefore, the motion-capture profiles should be supplemented with

vision, and contact information, and then investigate the high level organization of human

skilled behaviors. This part of the project is still being carried out at KAIST.


13/14

4. Strengthening KAIST-Stanford Relationship

The project indeed has been a very successful venture for both KAIST and Stanford

University. Both sides visited each other twice in the duration of two years. Our visiting

of each other actually strengthened our personnel relationships, and mutual

understanding. In addition to the project work, we have learned a lot about what is going

on the other side. It enlightened us about the cutting-edge research in robotics. After

working on this project, now we have strong relationship and better grounds for

promoting advanced future collaborations.

5.

Summary and Conclusion

This project has been dedicated for developing basic autonomous capabilities for robots

to be able to inhabit human-populated environments. We have particularly concentrated

on skill primitives on (1) Mobile manipulation and (2) Cooperative behavior. With regard

to mobile manipulation skills, we have developed methods for (1a) Decoupling force and

motion control by null space mapping, (1b) Real-time obstacle avoidance by elastic strip

and potential fields. With regard to cooperative skills, we have developed (2a)

Compliance control with virtual linkages and augmented object, and (2b) Decentralized

control structure for autonomous behavior.

Another parallel thread of this project has been devoted to revealing the organizational

structure of skilled human behavior by way of analyzing captured motion of human

demonstration, augmented with contact and vision information. However, this work is

still in progress, and we are mot is a position to publish any results on that as of yet.

The primitives that we have developed for mobile manipulation and cooperative skills

can be stored in a task behavior library (TBL) and the complex behaviors that a robot

would need to inhabit human-populated environment can be instantiated from this TBL.

6. Acknowledgement

This collaboration work was funded under the Brain Korea 21 project between KAIST

and Stanford University.


14/14

References

[1] J. C. Latombe, Robot Motion Planning Kluwer Academic Publisher, Boston 1991.

[2] J.P.H. Steel, G. P. Starr, mobile robot path planning in dynamic environments, in

Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, vol. 2,

pp. 922-925, 1988.

[3] W. Choi, J .C. Latombe, A reactive architecture for replanning and executing robot motions

with incomplete knowledge, in Proceedings of the IEEE/RSJ International workshop on

Intelligent Robots and Systems, vol. 1, PP. 24-29, 1991

[4] S. Quinlan and O. Khatib, Elastic bands: Connecting path planning and control, in

Proceedings of the IEEE International Conference on Robotics and Automation, vol. 2 , pp.

802-807, 1993

[5] S. Quinlan, Real-time modifications of collision free paths, Ph.D. thesis, Stanford

University, 1994

[6] O. Broke, O. Khatib, and S. Viji, Mobile manipulation: Collision-free path modification and

motion coordination, in Proc. Proceedings of the 2nd International Conference on

Computational Engineering in Systems Applications, vol. 4, pp. 839-845, 1998.

[7] O. Khatib, A unified approach for motion and force control of robot manipulators: The

operational space approach, IEEE Journal of Robotics and Autonation, vol. RA-3, no. 1, pp.

43-53, Feb., 1987

[8] O. Khatib, A unified approach motion and force control of robot manipulators: The

operational space formulation, International Journal of Robotics and Automation, 3(1):43-

53, 1987

[9] O. Khatib, Object manipulation in a multi-effector robot system in R. Bolls and B. Roth

(Eds.),Robotics Research 4, pp 137-144, MIT Press

[10] D. Williams and O. Khatib, The virtual linkage: a model for internel forces in multi-grasp

manipulation, in Proceedings International Conference on Robotics and Automation, vol. 1,

pp. 1025-1030[11] O. Khatib, K. Yokai, O. Nrock, K. Chang, and A. Casal, Robots in human environments:

Basic autonomous capabilities, in archives Robotics Laboratory, Department of computer

science, Stanford University, Stanford, CA 94086, USA

[ , ] autonomous human robot interactive skills

Documents