human-robot emotional and musical interactions...human-robot emotional and musical interactions...

Human-Robot Emotional and Musical

Interactions Massimiliano Zecca

Professor of Healthcare Technology [email protected]

Special lecture for the International Journal of Advanced Robotic Systems

Outline of the presentation

•  Emotional robotics •  Musical robotics •  Wearable bioinstrumentation

The pyramid of life

Increasing population Stable population Decreasing population

The pyramid of life is becoming more and more unstable

http://www.ipss.go.jp/index-e.html

1930 – the past

Over 65 <5%

2005 – the present

Over 65 about 20%

2050 – the future

More than 30% of the population in Japan will be over 65 years old!!!

Seamless mobility

Natural communication

Safety technology

Pseudo-lite

GPS

Environmental information structuring

Pseudo-lite (outdoor)

Autonomous environment recognition RFID tag

RT furniture

Interaction with physical environment

Position　identification

Robot as physical agent

Emotional robot WE-4RII

Paul Ekman’s classification of emotions

•  Ekman’s list of basic emotions (1970’): •  anger •  fear •  surprise

ANGER DISGUST FEAR HAPPINESS SURPRISE SADNESS

The facial expressions of the 6 primary emotions are common to all cultures

FACS (Facial Action Coding System)

•  Face elements

1.  Miwa’s model 2.  Kismet’s

model 3.  Circumplex

model 4.  Plutchick’s

Wheel of Emotions

1 2

3 4

Emotional models

Previous Studies

Tactile and Audition Olfaction

Motion Perception Depth Perception Light Perception Facial Expression

WE-2 (1995) WE-3 (1996) WE-3R (1997) WE-3RII (1998)

WE-3RIII (1999) WE-3RIV (2000) WE-3RV (2001)

Facial Color

WE-4 (2002)

New Model

External Environment

Internal Environment

Autonomic Reflex

Robot

Sensing

Sensing Personality

Emotion Need

Motion

Motion

Expression Personality

Emotion Need

Emotion Mood

Need

Consciousness

Mental Model for Humanoid Robot

Motion Reflex Sensing

Recognition Behavior Intelligence

WE-4RII emotional expressions

WE-4RII Various behaviors

WE-4RII consciousness

Big issues (1)

The robot works, but… •  What about the people interacting with the robot? •  Is s/he enjoying the interaction? •  What about her/his emotions?

Musical Robots WF-4 and WAS-3

Interacting with humans at musical level

Long term research objective

natural and intuitive human-robot complete musical interaction

Waseda Flutist Robot N.4 Refined VI

WF-4RVI (2010)

1600

[mm

]

Lungs

Lips

Arms

Neck Fingers

Eyes

Lungs 2-DOF

Vibrato 1-DOF

Eyes 3-DOF

Arms 14-DOF

Lips 3-DOF

Fingers 12-DOF

Neck 4-DOF

Tonguing 1-DOF

DOF Configurations

Total : 41‐DOF X

Z

Y

Waseda Flutist Robot N.4 Refined VI

WAseda Saxophonist No.3

WAS-3

1400

[mm

]

x

z y

Oral cavity／1-DOF Tongue ／1-DOF

Pump ／ 1-DOF

Valve ／　 1-DOF

y x

z

Lips／2-DOF

Air Flow

x z y

Eyes／2-DOF

Right／8-DOF

179[

mm

]

Left／11-DOF

185[

mm

]

Lung

Waist /1-DOF

Hands／19-DOF

計／28-DOF

WAseda Saxophonist

Big issues (2)

•  What about the people interacting with the robot? •  Is s/he enjoying the interaction? •  What about her/his emotions?

•  What about the interaction with other musicianso and artists in a small group?

•  And hat about the interaction within a big orchestra?

Wearable sensing Common need: Understand the human being

Main Scientific interest

•  Observation, analysis, and understanding of the human being •  extreme and exquisite example of robotic system, •  clarification of the basic mechanisms underlying

human's control of their bodies •  extremely important and helpful tool for the society for

realizing better health-care systems, human-support devices, teleoperation methods, and so on.

Main Research Activities

•  OBJECTIVE: •  to develop a Wearable Bio-

instrumentation System capable of monitoring and integrating the human motion with several physiological indices

•  Core research tool for understanding the basic mechanisms underlying human’s control of their bodies

•  Ultimate objective: •  to improve and enhance the quality of life of elderly and disabled people

House & Health care

Wearable Bio-

instrumentation System

Research approach and activities


•  WB-4 and WB-4R Inertial Measurement Units

•  WB-EMG wireless sensor

Industrial cooperation → WB Evolution

WB-1

2004

WB-1R

2005 2006

WB-2 WB-2R

2007

WB-3

2008

WB-4

2010

Z X

Y

Z X

Y

Z X

Y

Z X

Y

WB-4 WB-4

Best Paper Finalist Award, ROBIO 2007 Best Paper Award, AIM 2009 Best Paper Finalist Award, IRIS 2010

Best Presentation Award, WWLS 2010 Best Paper Finalist Award, ROBIO2010 Best Paper Award, ROBIO2012

2014

?WB-5?

12mm

WB-4 Wireless IMU (7g)

Bluetooth

Mother board

WB-4 Wireless IMU

q  Bluetooth module: Zeal-‐C01 (Musenka Company) q  Li-‐Polymer ba3ery : KPL501717 (3.6V@80mAh, Kayo Inc.) q  WB-‐4 IMU power consump>on: 3.0V@80mA q  Con>nuous working >me: 1 hour q  Data: ±8 [G], ±1600 [deg/s] @200Hz

WB-EMG Sensor Specifications

Circuits WB-‐EMG Board Gain 2800

PCB Size (mm) 30.5x17.5x11mm Voltage Supply 3.7V (Poly Li-‐ion baRery) Sampling Rate 1KHz ResoluUon 12-‐bit

Low Frequency Cutoff 19Hz

High Frequency Cutoff 457Hz

Transmission Frequency 2.4 GHz (Bluetooth)

Power line Noise RejecUon Method

Amplifier Common-‐Mode RejecUon Driven Right Leg

DC Input RejecUon Passive DifferenUal Filter

DC Offset Removal from INA Feedback Integrator

Movement ArUfact RejecUon Input Filter, Feedback Integrator

IC Packages 8-‐MSOP, 14-‐TSSOP

Passive Components 16

AcUve Components 2

Components Count 18

ICMA2013, best student paper finalist


•  Orientation estimation •  Wavelet Based EMG

Denoising •  EMG-driven muscle model

Quaternion-based Extended Kalman Filter

1)( −+= kTkkk

Tkkk RHPHHPK

Effect of R-Adaptive algorithm

ü  Perfect bias canceling No dynamic error rejecUon ü  Dynamic error rejecUon Drie

ü  Dinamic error RejecUon ü  No Drift

Denoising: Baseline Adaptive Denoise Algorithm (BADA)

Successes Limita>ons ü  ReducUon of false acUvaUons 92% vs original

signal and 10% vs Standard Dohono Technique

Ø  Heavy ComputaUonal Load of Wavelet Algorithm -‐> difficult to implement Real Time

ü  No negaUve impact the signal original shape Ø  Firmware currently implemented only on MATLAB

L. Bartolomeo, M. Zecca, et al, “Wavelet thresholding technique for sEMG denoising by baseline estimation” in International Journal of Computer Aided Engineering and Technology, vol. 4, no. 6, p. 517, 2012 L. Bartolomeo, M. Zecca, et Al., “Baseline Adaptive Wavelet Thresholding Technique for sEMG Denoising”, AIP Conference Proceedings Vol. 1371, ISSN Print+Online: 0094-243X, ISSN Online only: 1551-7616, ISBN: 978-0-7354-0931-6, 2011

Proposed EMG+IMU Driven Musculoskeletal Model

Muscle ContracUon Dynamics + Hill Model

a(t) F EMT (θ,t)

Joint angles: θ(t)

EMG

Model Parameters (EsUmated from CalibraUon)

PH

IMU LMT (θ,t)=f (θ,t)

LMT (θ,t)

NRMSE=7.3 ± 1.0 Normalized Root Mean Square Error

Successes Limita>ons ü  Fast (i7 Dual Core Intel <0.2s for 30s exercise)

with only 1% NRMSE degradaUon -‐> possibility of Real Time Analysis

Ø  Firmware currently implemented only on MATLAB

ü  Synchronized IMU + EMG Ø  EKF opUmized only for the upper limbs

Synchronized


•  Human-Robot interaction

•  Healthcare •  And many

many more…



Autonomic Reflex

Robot

Sensing

Sensing Personality

Emotion Need

Motion

Motion


Emotion Need

Emotion Mood

Need

Consciousness

Original Mental Model





Autonomic Reflex

Robot

Sensing

Sensing Personality

Emotion Need

Motion

Motion


Emotion Need

Emotion Mood

Need

Consciousness

Modified Mental Model



Sensing

Human-Robot Interaction Example:

Stress management

Artist

Performance rule Modulation of the musical output

of the robot by an artist`s performance

Performance rule

Robot Performance feedback

Interaction System General Structure

Performance rule

Robot Performance feedback Artist


Performance Start Synchronization

Dance performance: very strong expression of instinctive motion based on musical performance.

Robot Performance feedback

Artist

Instinctive interaction

Performance rule

Dancer


Instinctive Interaction with a Dancer


Directing a musical robot

Music

Robot (WF4-‐RVI)

Conductor Data Preprocessing

Ictus DetecUon

Tempo CalculaUon

Mood DiscriminaUon

MIDI Controller

WB-‐4

Beat SegmentaUon

Normal, Legato, Marcato

Tempo

Traditional training model

Skill Trainer Trainee

Evaluator/Expert

Score

Current universal evaluation model

Advice

Methodology proposal

Instrument / human moUon measurements

MoUon analysis and skill classificaUon

QuanUtaUve feedback

Training plaqorms Subject

Expert doctor

to develop a general skill evalua>on system •  separated from the training devices •  objecUve and quanUtaUve based on moUon analysis •  adapUve to various training devices, surgical applicaUons and

clinical pracUce

Medical applications – Laparoscopy (1/3)

Qualitative analysis

Novices r-ECU

Experts r-ECU

Novices, differently from the experts, use the right forearm only during the second part of the exercise


Quantitative analysis

Expert Surgeons Muscle Activation

Muscles present different behavior in different exercises 0

20

40

60

80

100

r-‐ECU r-‐FCR r-‐BB r-‐TB r-‐TP l-‐TP l-‐ECU l-‐FCR l-‐BB l-‐TB

Force [N]

Dry Boxes In vivo Experiment

** p<0.01

**

**

**

**

**

**


Successes Limita>ons ü  General system able to provide both

qualitaUve and quanUtaUve results Ø  EKF opUmized only for the upper limbs

ü  ApplicaUon in a complete curriculum for laparoscopic training

Ø  Current HW setup invasive and uncomfortable

Take home message

Take home message

•  We need to work hard for developing better robots

•  More important, we need to work harder for understanding ourselves J

Human-Robot Emotional and Musical

Interactions Massimiliano Zecca

Professor of Healthcare Technology [email protected]

Special lecture for the International Journal of Advanced Robotic Systems

human-robot emotional and musical interactions...human-robot emotional and musical interactions...

Documents