symbiotic human-computer interaction - engineeringpetriu/rose2006 -symbiotic_hci-v5-post.pdf ·...

Symbiotic Human-Computer

Interaction

Emil M. PetriuUniversity of Ottawa, Ottawa, ON, Canada

http://www.site.uottawa.ca/~petriu/

Thomas E. WhalenCRC, Ottawa, ON, Canada

Abstract

The presentation will address basic principles and discuss examples of symbiotic human-computer interaction for the next evolutionary stage of the computing technology.

It is expected that current "human-computer interaction" (HCI) interfaces will evolve into more efficient multimedia HCI symbiotic partnerships in which humans will contribute human-specific capabilities complementing those of the computer.

It will be a symbiotic relationship in which each partner will lead in some cases and provide assistance in others. The leader/assistant role of a partner will be decided on the basis of maximizing the overall efficiency of the symbiotic team.

The classic measurement process: an early example of human-transducer cooperation

TRANSDUCER

MEASURAND(weight, voltage, power,…)

2.83 Kg

Robots emulate humans

The definition given by M. Bradley: "Robotics is the intelligent connection of the perception to action" considers robotics from a system integration perspective, indicating how robots are doing things.

Programmable robots (manipulators,vehicles) provide the action function.

A variety of sensors provide the perceptioncapability.

Computers provide the framework for integration/connection as well as the intelligence needed to coordinate in a meaningful way the perception and action capabilities.

Robot Controller - Hierarchical Architecture

Artificial intelligence level, where the program will accept a command such as Pick up the bearing and decompose it into a sequence of lower level commands based on a strategic model of the task.

Control mode level where the motions of the system are modelled, including the dynamic interactions between the different mechanisms, trajectories planned, and grasp points selected. From this model a control strategy is formulated, and control commands issued to the next lower level.

Servo system level where actuators control the mechanism parameters using feedback of internal sensory data, and paths are modified on the basis of external sensory data. Also failure detection and correction mechanisms are implemented at this level.

ACTUATORS

SENSORS(EXTEROCEPTORS)


MULTI-SENSORFUSION

HYBRID DELIBERATIVE/REACTIVE

CONTROL

WORLD MODEL

DISTRIBUTEDCOMPUTINGFRAMEWORK:

HIGH LEVELINFORMATIONEXCHANGE MECHANISMS:XML, Grammars,...

MULTI-CARRIERCOMMUNICATION

MECHANISMS

WIR

ELE

SS

CO

MM

UN

ICA

TIO

N N

ETW

OR

K

ROBOT CONTROL SYSTEM

Model (human’s image) of the real world as he/she perceives it troughhis/her sensory organs Off-Line Programing

by Human

Traditional human-computer interaction

VideoC a m e r a Robot

Arm

Tact i l e S e n s o r s

Man ipu la ted Ob jec t

H e a d M o u n t e d D i s p l a y

Hap t i cF e e d b a c k

V i r tua l mode l of the object man ipu la tedin t he phys i ca lwor ld

VideoC a m e r a Robot

Arm

Tact i l e S e n s o r s


V ideoC a m e r a Robot

Arm

Tact i l e S e n s o r sTac t i l e S e n s o r s








Video and haptic virtual reality interfaces allow a human operator to interactively control a remote robot manipulator

equipped with video camera and tactile sensors.

Tele-Robotic Manipulation

Robot hand and tendon driven compliant wrist

System architecture

(a) The robotic tactile sensor , and (b) the human tactile feedback

Early (80s) Video and HapticTelerobotic System

E.M. Petriu, D.C. Petriu, V. Cretu, "Control System for an Interactive Programmable Robot," Proc. CNETAC Nat. Conf. Electronics, Telecommunications, Control, and Computers, (in Romanian), pp. 227-235, Bucharest, Romania, Nov. 1982.

……

Tactile Sensor Interface

HapticRobot

InterfaceROBOT(k)

OBJ(i)

Robot Arm Controller

C y b e r G r a s p ™ C y b e r T o u c h ™

H a p t i c H u m a n Inter face U S E R ( k )

N E T W O R K N E T W O R K

…

{ [ 3 D ( j ) & F ( k , j ) ] , t }

A V A T A R H A N D ( k )

. . .

. . .

O B J ( N ) O B J ( 1 )

3 D G e o m e t r i c & E las t i c

C o m p o s i t e M o d e l o f O b j e c t{( x

p, y

p, z

p,E

p) | p = 1 , . . , P }

O B J ( j )

Virtual Operation TheaterVirtual Operation Theater

Composite Haptic Interaction Vector between User (k) and Object (j)

A p p l i c a t i o nS p e c i f i c

In te rac t i veAct ion

S c e n a r i o

…

……


HapticRobot

InterfaceROBOT(k)

OBJ(i)



HapticRobot

InterfaceROBOT(k)

OBJ(i) OBJ(i)






N E T W O R K N E T W O R K

…

{ [ 3 D ( j ) & F ( k , j ) ] , t }


. . .

. . .

O B J ( N ) O B J ( 1 )



p, y

p, z

p,E

p) | p = 1 , . . , P }

O B J ( j )





S c e n a r i o

{ [ 3 D ( j ) & F ( k , j ) ] , t }


. . .

. . .

O B J ( N ) O B J ( 1 )O B J ( 1 )



p, y

p, z

p,E

p) | p = 1 , . . , P }

O B J ( j )



p, y

p, z

p,E

p) | p = 1 , . . , P }

O B J ( j )

3 D G e o m e t r i c & E las t i c3 D G e o m e t r i c & E las t i c


p, y

p, z

p,E

p) | p = 1 , . . , P }{( x

p, y

p, z

p,E

p) | p = 1 , . . , P }

O B J ( j )





S c e n a r i o

…

It allows any user USER(k) (k=1 ,... ,NU) to experience the haptic hands-on feeling while dexterously manipulating remote physical objects via a given robotic telemanipulator ROBOT(k) (k=1 ,... ,NR) equipped with haptic and video sensors. An impedance-reflecting virtual operation theatre contains the composite geometric & haptic models of the kinematic and dynamic interaction behaviour of all manipulated objects . Haptic transport protocols allow to compensate for network lag and preserve synchronization in shared object manipulation.

Distributed Interactive Telemanipulation Systemunder development at the University of Ottawa

The robotic telemanipulation system has a bilateral architecture allowing to connect the human operatorand the telerobotic manipulator as transparently as possible.

Conformal (1:1) mapping of human & robot sensory and perception frameworks

Using a head mounted display for augmented visual virtual reality and a haptic feedback system, the human operator controls the operation of a remote robot manipulator equipped with video camera and tactile sensors placed in robots hand.

The tactile sensors provide the cutaneous informationat the remote robotic operation site. The joint sensors of the robot arm provide the kinesthetic information.

A tactile human-computer interface provides the cutaneous feedback allowing the human operator to feel with his/her own sense of touch the same sensation as that acquired by the remote robot hand from its artificial tactile sensor. A robot-like kinematicstructure provides the kinesthetic feedback for the haptic human-computer interface.

Human-Computer Interaction in the Tele-Robotic Manipulation System

ACTUATORS



MULTI-SENSORFUSION

HYBRID DELIBERATIVE/REACTIVE

CONTROL

WORLD MODEL

DISTRIBUTEDCOMPUTINGFRAMEWORK:

HIGH LEVELINFORMATIONEXCHANGE MECHANISMS:XML, Grammars,...

MULTI-CARRIERCOMMUNICATION

MECHANISMS

WIR

ELE

SS

CO

MM

UN

ICA

TIO

N N

ETW

OR

K

ROBOT CONTROL SYSTEM

Model (human’s image) of the real world as he/she perceives it troughhis/her sensory organs

MULTI-MEDIAHCIINTERFACES

Off-Line Programingby Human

Real-Time Interactive

Programingby Human

Human-Computer interaction for symbiotic operations

The time clutch concept is used to disengage synchrony between operator specification time and tele-robot operation time during path specifications.In order to avoid fatal errors and reduce the effect of the communication delay, we are using a distributed virtual environment allowing to maintain a shared world modelof the physical environment where the telemanipulationoperation is conducted].

Another critical requirement is the need to maintain the synchronism between the visual and the hapticfeedback. While being two distinct sensing modalities, both haptic perception and vision measure the same type of geometric parameters of the 3D objects that are manipulated.

Visual-, geometric-, force-, and touch –domain human -feedback devices

Biologically inspired robotic manipulator consists of arobot arm an instrumented passive-compliant wrist and a 16-by-16 tactile probe. Position sensors placed in jointsand on the instrumented passive-compliant wristprovide the kinesthetic information. The compliant wristallows the robot-hand equipped with tactile sensors to accommodate the constraints of the explored object surface and thus to increase the amount of cutaneoustactile information.

The tactile probe is based on a 16-by-16 matrix of Force Sensing Resistor (FSR) elements spaced 1.58 mm apart on a 6.5 cm2 (1 square inch) area. The FSR elements exhibit exponentially decreasing electrical resistance with applied normal force: the resistance changes by two orders of magnitude over a pressure range of 1 N/cm2 to 100 N/cm2.

The elastic overlay has a protective damping effect against impulsive contact forces and its elasticity resets the transducer system when the probe ceases to touch the object. However, they may cause considerable blurring distortions in the sensing process if they are not properly controlled. We avoided it by replacing the one-piece pad with a custom-designed elastic overlay consisting of a relatively thin membrane with protruding round tabs.

.

16-by-16 median

filtered tactile image of a washer.

The kinesthetic sensing provides information about the positions and velocities of the kinematic structure of the hand.

The human kinesthetic capability has a much lower frequency band than the cutaneous perception provided by the fingertips.

Human Kinesthetic Perception

• Uses 18-22 linear sensors – electrical strain gauges;• Angles are obtained by measuring voltages on a Wheastone bridge;

• 112 gestures/sec “filtered”.• Sensor resolution 0.5 degrees, but errors accumulate to the fingertip (open kinematic chain);

• Sensor repeatability 1 degree• Needs calibration when put on the hand

CyberGlove®

Immersionn_3D Interaction <http://www.immersion.com/>

CyberGrasp™ Pack

CyberForce ™

CyberGrasp™


Commercial Virtual Hand Toolkit for CyberGlove/Grasp providing the kinesthetic human-computer interface

The cutaneous/tactile sensing provides information about contact force, contact geometric profile andthe temperature of the touched object.

There are various cutaneous mechanoreceptors located in the outer layers of the skin. These receptors are specialized nervous cells or neurons. The free nerve endings are the most numerous and play an active role in the perception of pain, cold and warmth.

These mechanoreceptors have preferential frequency response characteristics: the highest sensitivity for the Pacinian Corpuscles (PC) units is around 250-300 Hz but they respond from 30 Hz to very high frequencies. The Rapidly Adapting (RA) units effective frequency range is between 10 and 200 Hz, with more sensitivity below 100 Hz. The Slowly Adapting (SA) units respond at low frequencies, under 40-50 Hz.

( The human kinesthetic function has a much lower frequency band.)

Human Cutaneous/Tactile Perception

Two-point limen test: 2.5 mm fingertip, 11 mm for palm, 67 mm for thigh (from [Burdea& Coiffet 2003] ).

[Burdea& Coiffet 2003] G. Burdea and Ph. Coiffet, Virtual Reality Technology,(second edition), Wiley, New Jersey, 2003

CyberTouch™


Tactle/Cutaneous Feedback to the Human Operator

H U M A N - H A N D

TM TM

TACTILE SENSATION RECONSTRUCTION

TM = Tactile Monitor

TS TS

TACTILE IMAGE ACQUISITION

ROBOT - HAND

TS = Tactile Sensor

A tactile monitor placed on the operator's palm should allow the human teleoperator to virtually feel by touch the object profile measured by the tactile sensors placed in the jaws of the robot gripper.

Cutaneous tactile HCI developed at the University of Ottawa in the early 90s. It consists of an 8-by-8 array of electromagnetic vibrotactile stimulators. The active area is 6.5 cm2 (same as the tactile sensor). The figure shows a curved edge tactile feedback .

Ø Human operator and intelligent sensor-based systems work together as symbionts(cyborgs) each contributing the best of their specific abilities.

Human-Computer Symbiont (Cyborg) Systems

Ø Proper control of these operations require humnan-computer interfaces capabilities allowing the human operator to experience the feeling of virtual immersion in the working environment.

Interfaces enhancing the performance figures of the human’s natural capabilities

eye glasses, binoculars, IR night vision device, HMD for augmented VR, ..

hearing amplifier

NORMALBASIC

HUMNANBEING

Hand

Leg

Ear

Eye

Brain PDA

ARTIFICIALLY ENHNCED HUMAN = SYNBIONT ?

footwear, skates, bike,exoskeleton,..

gloves (baseball glove), hand tools (pliers)

Prosthetic interfaces enhancing the performance figures of the human’s natural capabilities

(including survivability)

Artificial Hand

Knee Joint +Titanium Knee Joint

Ear +Hearing Aid

(Implant)

Eye +Artificial Cornea

CRIPPLEDHUMNAN

BEINGPacemaker

ARTIFICIALLY ENHNCED HUMAN= SYNBIONT

Interfaces enhancing the performance figures of the human’s natural capabilities

eye glasses, binoculars, IR night vision device, HMD for augmented VR, ..

PDA

gloves (baseball glove), hand tools (pliers)

ARTIFICIALLY ENHNCED HUMAN = SYNBIONT !

footwear, skates, bike,exoskeleton,..

Artificial Hand

Knee Joint +Titanium Knee Joint

Ear +Hearing Aid

(Implant)

Eye +Artificial Cornea

HEALTYOR

CRIPPLEDHUMNAN

BEINGPacemaker

Brain

Eye

Hand

Human beings are only valuable in this symbiotic partnership to the degree that their capabilities complement those of the computers:

(i) humans are still far more intelligent than any computer, are able to act on incomplete or ambiguous instructions, able to adapt to a variety of computer interfaces, and able to interact directly with other humans,

(ii) humans are mobile being able to leave the vicinity of the computer, perform complex tasks in a variety of different environments, and then return to report on the outcome,

(iii) humans can recognize visual, auditory, olfactory, gustatory, and haptic stimuli,

(iv) humans are dexterous, which allows them to precisely manipulate a wide variety of objects,

(v) humans are emotional, varying the characteristics of response, depending on the global state of each individual.

E.M. Petriu, T.E. Whalen, "Computer-Controlled Human Operators," IEEE Instrum. Meas. Mag., Vol. 5, No. 1, pp. 35 -38, 2002.

Humans are very high-bandwidth creatures:

- their visual system is capable of perceiving more than a hundred megabits of information per second, and

- their largest sense organ, the skin is capable of perceiving nearly that much as well.

- human speech conveys information in the form of intonation and inflection as well as the actual words uttered.

People communicate through "body language“which includes facial expressions and eye movements.

q Symbionts will combine intrinsic machine-sensing reactive behavior with higher-order human-oriented world-model representations of the immersive virtual reality.

Human-to-human communication and cooperation require a common language and an underlying system of shared knowledge and common values. In order to achieve a similar degree of human-to-computer interaction and cooperation, a symbiotic framework should be developed to allow for the management of heterogeneous functions and knowledge.

Cyber/Machine Society/World

{Intelligent Robot Agents}

Cyber/Machine Concept Representation Language

HumanSociety/World

{Human Beings}

Human Concept

Representation Language




HumanSociety/World

{Human Beings}

Human Concept


Asimov’s laws of the robotics:

1st law: “A robot must not harm a human being or, through inaction allow one to come to harm”.

2nd law: “A robot must always obey human beings unless that is in conflict with the 1st law”.

3rd law: “A robot must protect itself from harm unless that is in conflict with the 1st and 2nd law”.




HumanSociety/World

{Human Beings}

Human Concept


SymbiontSociety/World

{Cyborgs} SymbiontConcept



Human Concept


SymbiontConcept


Multi-CulturalHuman &Cyber&

SymbiontHyper-Society

World

Hyper-Society CommonConcept

Representation Meta-Language

Multi-CulturalHuman &Cyber&

SymbiontHyper-Society

World

Hyper-Society CommonConcept


Human Concept


Asimov’s laws of the robotics:

0th law: "A robot may not injure humanity or, through inaction, allow humanity to come to harm."

1st law- updated: “A robot must not harm a human being or, through inaction allow one to come to harm, unless this would violate the 0th law."

2nd law: “A robot must always obey human beings unless that is in conflict with the 1st law”.

3rd law: “A robot must protect itself from harm unless that is in conflict with the 1st and 2nd law”.__________[*] I. Asimov, Robots and Empire, Doubleday & Co., NY 1985, p.291

BORGHyper-Society

World

BORG Hyper-Society Common Concept


Moral, Ethical, Theological, Legal,Biological, Psychological Social, Economic,…. Challenges in a BORG Hyper-Society World

[Normal Human Partner] + [Pacemaker-fitted Human Partner]= [Acceptable Married (incl. Lovers)_Couple]

[Normal Human Partner] + [Advanced Augmented Symbiont Partner]= [Acceptable Married (incl. Lovers)_Couple] ?

[Normal Human Partner] + [Robot Partner]= [Acceptable Married (incl. Lovers)_Couple] ???

=> Absolutely ! … according to Asimov’s “Prelude to the Foundation”

Human( no machine

allowed)Society/World

{Human Beings}

Human Concept


No Robots !!! No No Robots !!! No SymbiontsSymbionts !!! !!! . . or an alternative è

HumanSociety/World

{Human Beings}

Human Concept


The GALACTIC EMPIRE:The GALACTIC EMPIRE:

q Robots banned . . but Eto Demerzel.

q Hari Seldon’s Psychohistory.

q The FOUNDATION => SECOND GALACTIC EMPIRE SECOND GALACTIC EMPIRE

. . .

Publications

• A.-M. Cretu, J. Lang, E.M. Petriu, “A Composite Neural Gas-Elman Network that Captures Real-World Elastic Behavior of 3D Objects,” Proc. IMTC/2006, IEEE Instrum. Meas. Technol. Conf., pp. 1063-1068, Sorrento, Italy, April 2006.

• A.-M. Cretu, E.M. Petriu, “Neural-Network –Based Adaptive Sampling of Three-Dimensional-Object Surface Elastic Properties,” IEEE Trans. Instrum. Meas.," Vol. 55, No. 2, pp. 483 - 492, 2006.

• A.-M. Cretu, E.M. Petriu, G.G. Patry, “Neural-Network-Based Models of 3 -Objects for Virtualized Reality: A Comparative Study,” IEEE Trans. Instrum. Meas.," Vol. 55, No. 1, pp.99 - 111, 2006.

• P. Payeur, C. Pasca, A.-M.Cretu, E.M. Petriu, “Intelligent Haptic Sensor System for Robotic Manipulation,” IEEE Trans. Instrum. Meas., Vol. 54, No. 4, pp. 1583 –1592, 2005.

• T.E. Whalen, D.C. Petriu, L.Yang, E.M. Petriu, “Interactive Scripts for HapticVirtual Environments,” Proc. HAVE 2005 - IEEE Intl. Workshop on Haptic, Audio and Visual Environments and their Applications, pp. 143-147, Ottawa, ON, Canada.t. 2005.

• E.M. Petriu, S.K.S. Yeung, S.R. Das, A.-M. Cretu, H.J.W. Spoelder, “Robotic Tactile Recognition of Pseudorandom Encoded Objects,” IEEE Trans. Instrum. Meas., Vol. 53, No. 5, pp. 1425-1432, 2004.

• E.M. Petriu, T.E. Whalen, R. Abielmona, A. Stewart, "Robotic Sensor Agents," IEEE Instrum. Meas. Mag., Vol. 7, No. 3, pp. 46 -51, 2004.

• T.E. Whalen, D.C. Petriu, L. Yang, E.M. Petriu, M.D. Cordea, “Capturing Behaviour for the Use of Avatars in Virtual Environments,” CyberPsychology & Behavior, Vol. 6, No. 5, pp. 537-544, 2003.

• P. Rusu, E.M. Petriu, T.E. Whalen, A. Cornell, H.J.W. Spoelder, “Behavior-Based Neuro-Fuzzy Controller for Mobile Robot Navigation,” IEEE Trans. Instrum. Meas., Vol. 52, No. 4, pp.1335- 1340, 2003.

• E.M. Petriu, "Neural Networks for Measurement and Instrumentation in Virtual Environments,” in Neural Networks for Instrumentation, Measurement and Related Industrial Applications, (S. Ablameyko, L. Goras, M. Gori, V. Piuri Eds.), NATO Science Series, Computer and System Sciences Vol. 185, pp.273-290, IOS Press, 2003.

• E.M. Petriu, T.E. Whalen, "Computer-Controlled Human Operators," IEEE Instrum. Meas. Mag., Vol. 5, No. 1, pp. 35 -38, 2002.

• E.M. Petriu, T.E. Whalen, V.Z. Groza, “Haptic Perception System for Robotic Tele-Manipulation,” Proc. INES 2002, 6th International Conference on Intelligent Engineering Systems 2002, pp. 51-55, Opatija, Croatia, May 2002.

• E.M. Petriu, D. Ionescu, D.C. Petriu, S.K. Yeung, Ph. Lavoie, N. Trif, “Absolute Position Measurement Applications of Pseudo-Random Encoding,” Proc. ETIM'96 IEEE Intl. Workshop on Emergent Technol. for Instrum. Meas., pp.119-126, Como, Italy, 1996.

• S.K. Yeung, E.M. Petriu, W.S. McMath, D.C. Petriu, "High Sampling Resolution Tactile Sensor for Object Recognition," IEEE Trans. Instrum. Meas.,Vol. 43, No. 2, pp.277-282, 1994.

• W.S. McMath, M.D. Colven, E.M. Petriu, S.K. Yeung, "Tactile Pattern Recognition Using Neural Networks," Proc. IEEE&SICE IECON'93 Conf., pp. 1391-1394, Maui, Hawaii, 1993.

• E.M. Petriu, W.S. McMath, "Tactile Operator Interface for Semi-autonomous Robotic Applications," Proc. Int. Symposium on Artificial Intell. Robotics Automat. in Space, i-SAIRS'92, pp.77-82, Toulouse, France, 1992.

• E.M. Petriu, W.S. McMath, S.K. Yeung, N. Trif, "Active Tactile Perception of Object Surface Geometric Profiles," IEEE Trans. Instrum. Meas., Vol. 41, No. 1, pp.87-92, 1992.

• S.K. Yeung, W.S. McMath, E.M. Petriu, N. Trif "Three Dimensional Object Recognition Using Integrated Robotic Vision and Tactile Sensing," Proc. IEEE&RSJ Int. Workshop on Intell. Robots and Systems IROS'91, pp.1370-1373, Osaka, Japan, 1991.

• E.M. Petriu, D. Petriu, V. Cretu, "Multi-Microprocessor Control System for an Experimental Robot with Elastic Joints," Proc. Nat. Conf. Cybernetics, (in Romanian), Bucharest, Romania, 1981

symbiotic human-computer interaction - engineeringpetriu/rose2006 -symbiotic_hci-v5-post.pdf ·...

Documents