neural approach for personalized emotional model in human-robot interaction mária virčíková,...

Neural Approach for Personalized Emotional Model in Human-Robot

Interaction

Mária Virčíková, Martin Pala, Peter Smolar, Peter SincakTechnical University of Kosice, Slovakia

IEEE WCCI 2012 Brisbane, Australia

Slovakia (5million inhabitants)

2/33

3

Center for Intel. Technology, Technical University of Kosice, Slovakia, EU

Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI

Structure of the Talk

A. Motivation & System

Overview

B. What is

unique about our

approach?

C. Technology - research

D. Results &

Future

3/33

4

How can we make the human - machine

(humanoid) interaction with

easier and more enjoyable for people ?



A. Motivation & System Overview

4/33

Visual events

Audible events

Touch events

… events

Exte

rnal

eve

nts

Battery state

Motors, processors state In

tern

al e

vent

s

Events Memory

Events Control Reasoning Control

Cognition Emotions

Output robot behavior



Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI5/33

6


Robotic ExpressionsEmotion recognition

Machine

User

Part of Human-Robot Communication


Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 6/33

7




Can this Robot Evoke my Emotions ???? Influence my decision?

Can it enhance Human-Robot or Robot-Human communication ?

7/33

8

“The question is NOT whether intelligent machines CAN HAVE emotions,

but whether machines CAN BE intelligent without any emotions.”

M. Minsky

“The question is NOT whether intelligent machines CAN EVOKE HUMAN

EMOTIONS, but whether machines

CAN BE intelligent without this ability .” M. Virčíkova at al.




8/33

9

B. What is UNIQUE about our approach?



10



User 1 SYSTEM

User 3

User 2

B. What is unique about our approach?

different people express same feelings in different ways

10/33

11

User 1 SYSTEM

User 2



B. What is unique about our approach?

8 basic emotions whole emotional spectrum

Evolving Robots behaviors during interaction with users

Easily preprogrammed for any humanoid-type robot

11/33

12

4. Emotional Technology: Experiment

User 1 SYSTEM

User 3

User 2

“Is the personalized form of the emotion E better understandable than

the pre-programmed form of the emotion E?”



13

C.Technology - Research



14


Em

oti

on

model by R

. Plu

tchik


Despite different forms of expression there are prototype patterns, that can be identified.

There is a small number of basic emotions. All other emotions are mixed or derivative states. All emotions vary in their degree of similarity to one

another. Each emotion can exist in varying degrees of intensity

or levels of arousal.

C. Technology - Research

Plutchik’s psychoevolutionary model

14/33

15

Em

oti

on

model by R

. Plu

tchik


Despite different forms of expression there are prototype patterns, that can be identified.

There is a small number of basic emotions. All other emotions are mixed or derivative states. All emotions vary in their degree of similarity to one

another. Each emotion can exist in varying degrees of intensity

or levels of arousal.


Plutchik’s psychoevolutionary model:computational implementation

NN recognition

Fuzzy Logic

Overlapped clusters create many types of mixed emotions from basic ones


15/33

16


Em

oti

on

model by R

. Plu

tchik



16/33

17


User 1



membership-function ARTMAP NN

o clustering technique with supervisiono incremental learningo calculates the membership function (MF) of the sample from the feature space to the fuzzy class

17/33

18


User 1 SYSTEM

User 3

User 2

Input pattern = M-dimensional vector (I1, , , , IM )

Ii = position of one point of the body in time



: Inputs

motion capture system

18/33

19

C. Technology - Research: Learning Process



When receives a feature vector, deduces the best-matching category by evaluating a distance measure against all memory category nodesEach new pattern is compared with all of the nodes (categories)

1. Memory empty - no categories recorded2. 1st pattern to arrive - the 1st category created - saved in the memory3. 2nd pattern reaches the network, a comparison between the arrived pattern and the saved category(s) is established4. NN compares the current input with a trained class representation (emotion) using training set

19/33

20


User 3



If new user starts to interact with the system, his/her emotions are added. The network is not learning from beginning, only this specific cluster is added to the original topology

20/33

21




• not a crisp decision on whether the input sample is or not a member of a particular class

• a vector of values that describes the degree of membership of this sample to all of the classes (emotions)

MF ARTMAP Result of classification

Fuzzy Plutchik’s model of emotional responses

21/33

22


User 1 SYSTEM

User 3

User 2

1. input layer - 3D coordinates of body postures in time2. comparison layer - partial membership functions 3. recognition layer - clusters - emotional expression of each

of the user . Adaptation to the concrete user4. mapfield layer - classes from previous clusters – from

different users5. cognitive layer - similarities between classes



23


User 1

User 3

User 2

the degree of membership of the emotion of the new user to the existing class is 1 only 1 cluster for both of the users for Joy

the degree of membership of his/her expression to the existing expression of Joy is 0 2 clusters for expressing Joy will exist



24


User 1 SYSTEM

User 3

User 2following statement can be deducted:

“expression of Anger of User A is similar to the expression of Surprise of User B with the value of the coefficient of similarity 0.8.”

coefficient of similarity is fuzzified:

“class A is very similar/similar/not similar to the class B.”



25

User 1 SYSTEM

X1x2Xn-1

Xn

Joy



X1x2Xn-1

Xn

Sadness

The importance of personalization – possible feedback for psychology

n- dimensional space , n= 48 x time

expressions in n-dimensional space

25/33

26

D. Results & Future


Maria Vircikova: Neural Approach for Personalized Emotional Model in HRIMaria Vircikova: Neural Approach for Personalized Emotional Model in HRI 26/33

27

D. Results: Basic emotions

User 1 SYSTEM

User 3

User 2

Personalized better

Equal to understand

Preprogrammed better



28

D. Results: Mixed emotions

SYSTEM

User 3

User 2

Personalized better

Equal to understand

Preprogrammed better



29

SYSTEM

User 3

User 2



Children’s Dpt. of Oncology, Hospital of

Kosice

Several Schools in Kosice (age 6-12)

Center for People with

AutismGeriatrics (elder

people)

D. Results

29/33

30

SYSTEM

User 3

User 2



31

Future possible consideration ......

SYSTEM

User 3

User 2



expand our database of emotional expressions …CLOUD ROBOTICS!

31/33

32

Summary (the last slide … )

SYSTEM

User 3

User 2



Personalization of the robot to different users

Replicable model from human body to humanoid body

Overlapped clusters create many types of mixed emotions from 8 basic emotions.

For new user – NN not learning from beginning, only this specific cluster is added to the topology.

Information about the data structure. The cognitive layer - similarity of clusters (how different

people express their emotions.)

Our modified MF ARTMAP neural network enables:

32/33

33

VideoSystem in action



34

Fuzzy relation for 2D input space

User 1



axes x and y - dimensions of the input space axe z - degree of membership of the samples to the fuzzy set

(fuzzy clusters )

35



Fuzzy cluster

Fuzzy cluster is considered as a fuzzy set in multidimensional feature space.

j=1, 2, …, m - index of the dimension of input space

U = X1 x X 2 x...x X - input spacen - number of samples in the training set

i = 1, 2, ... , n - index of the current samplep - number of previously formed fuzzy

clustersk = 1,2,…,p - index of the current fuzzy

clusterA(xi) - value of membership of sample xi to kEk and Fk - parameters of the fuzzy cluster k

Xsk - center of the cluster k

36



more complex shapes of real clusters... the fuzzy relation may:

1. not sufficiently cover the samples which belong to the fuzzy class2. cover even samples which do not belong to the fuzzy cluster.

… problem

… reason

fuzzy relation cannot adapt sufficiently to the training samples

37



Modified fuzzy relation

• multidimensional Gaussian probab. distribution based on the covar. matrix • able to adapt to the shape of the cluster made up from the input patterns

Σ covariance matrix of input samples in the dimensional feature space

X vector of input sample values µ vector of mean values

38

User 1 SYSTEM

User 3

User 2

Overl

ap

pin

g o

f cl

ust

ers



40

similarity between classesJeffries-Matusita(JM) distance

(if JM = 0 ... Identical clusters)



Cognitive layer

i, j - class identifiers of compared classes α - Bhattacharyya distance for multidimensional Gaussian

distributions

41

coefficient of similarity of each of the classes

0 - different classes 1 – identical classes



Ci,j number of clusters of the class i similar to our class j

Ci, Cj number of clusters of class i and j

neural approach for personalized emotional model in human-robot interaction mária virčíková,...

Documents

neural approach

personalized emotional

eu maria vircikova

hri user

system user

human emotions

emotional s

human robot