Towards Automated Recognition of Human EmotionsUsing EEG

Haiyan Xu

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, Universityof Toronto

July 12, 2012

Research Objective

To investigate whether EEG signal is feasible for affect detection analysis,especially using consumer grade EEG devices.

1/23

Outline

1 Overview of Affect Classification

2 Affect Detection System OverviewPreprocessing: Filtering Based on Instantaneuous FrequencyFeature AnalysisChannel Selection: Genetic Algorithm (GA)

3 Experimental Setup

4 Simulation Results

5 Summary of Research Contributions

6 Future Works

2/23

Overview of Affect Classification

Overview of Affect Classification

Affective Computing

Emotion is an unique, personal expression that differs under socialcontext, culture background and personal experience

Affective is the raw neurophysiological expression of emotion

Affect Sensitive Applications

Human Machine Interface (HMI)

Health and rehabilitation applications

Multimedia content indexing and retrieving

3/23

Overview of Affect Classification

Research Motivations

Prior Assessment Methods

Facial expression, Voice, Physiological signals: heart rate, GSR, ..

Disadvantages

Not available due to inability to express emotions (autism disorders)

Not reliable due to noisy source (crowded place)

Interference due to non-emotional factors

Affect Computing and EEG

EEG signal originates from the Central Nervous system

Brain networks in the limbic system are associated with affectexpression

Works with inaccessible and non-coorperative cases (autism disorders)

Less influenced by non-emotional factors4/23

Overview of Affect Classification

Technical Challenges of Affect Detection Analysis UsingEEG

General Challenges

EEG signals are originated from a non-linear, non-stationary processes

Most signal processing systems are linear, use predefined bases, andassume stationary signals

Lack of understanding on the dynamics of brain networks andcorrelation with emotional states

Challenges Faced by Using Consumer-grade EEG Devices

Much fewer number of electrodes available and conventional spatialanalysis is not applicable

Optimal sensor configuration for such applicaiton is not studied

Constraints on computational complexity

5/23

Overview of Affect Classification

Research Contributions

Main Contributions

Novel signal representation for EEG based on Instantaneuous FrequencyAutomatic determination of optimal sensor configuration for affectapplication that can be generalized easily to other applications

Additional Contributions

First person to design and implement a compelete EEG signalprocessing system for affect detection at our universityInvestigated the implications on how key parameters. (e.g., samplingrate) affects system performance

6/23

Affect Detection System Overview

System Overview

Test EEG Signals

7/23

Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency

Preprocessing: Novel Filtering Based on InstantaneuousFrequency

PreprocessingInput signals Feature Analysis Classifiers

Output Class labels

Purposes: to reduce noises, artefact and other external interferences

Fourier and Wavelet Based Methods use a priori basis

Multivariate Empirical Mode Decomposition (MEMD) isInstantaneuous Frequency based

To obtain meaningful IF, Hibert Transform requires signal to bemonocomponent, zero-mean locally [1]Data-driven, suitable for non-stationary signals from non-linearprocessesDecompose original signal into multiple time-varying frequencycontent, Intrinsic Mode Functions (IMFs)

[1] L. Cohen. Time-Frequency Analysis, Prentice Hall, Englewood Cliffs (1995).

8/23

Affect Detection System Overview Preprocessing: Filtering Based on Instantaneuous Frequency

Preprocessing Cont.: MEMD as Filter bank

MEMD is useful as time-varying filtering technique (preprocessing),particularly for multicomponent signals.Frequencies of interest

Alpha wave 8− 12HzBeta wave 13− 30Hz

Reconstruct EEG signal using sum of IMFS

100

101

102

10−8

10−6

10−4

10−2

Frequency

PS

D in

Lo

g s

cale

IMFs obtained using Multivariate EMD

5 10 15 20

0.20.40.6

5 10 15 200.20.40.6

5 10 15 20

0.20.40.6

5 10 15 20

12

5 10 15 20

0.20.4

5 10 15 20

0.20.40.6

5 10 15 20

0.20.4

5 10 15 200

0.20.40.6

5 10 15 20

0.20.40.6

5 10 15 200.10.20.3

5 10 15 20

0.20.4

Time

Instataneuous Amplitude and Averaged Frequency of each IMFs

166.15

107.17

64.12

25.05

14.93

8.67

6.95

5.51

4.81

4.52

8.09

9/23

Affect Detection System Overview Feature Analysis

Feature Analysis

PreprocessingInput signals Feature Analysis Classifiers

Output Class labels

Purpose

Dimension and computational complexity reduction

Increase class separability

Feature Analysis Algorithms Used

Oscillation pattern variation in the time domainSix statistical features: e.g., Mean, Std, Skewness, Kurtosis..Higher Order Crossings (HOC): zero crossing counts with iterativefiltering process

Event-related energy variationSpectral domain: narrow band energy (1Hz resolution, 8− 30Hz range)Time-spectral: wavelet-based energy and entropy analysis

10/23

Affect Detection System Overview Channel Selection: Genetic Algorithm (GA)

Channel Selection: Genetic Algorithm (GA)

For 54 channels, there are 254

combinations, impossible for exhaustivesearch

Genetic Algoritm is a non-rankingglobal optimization method

Binary string representation forchannel informationCorrect classification rate was used asfitness functionChannels were selectd on results from10 runs of GA

Application of GA

Initial Population

EEG Features

Continue Evolution?

Optimal Features

Fitness Calculation

Crossover

Mutation

UpdatedPopulation

No

Yes

11/23

Affect Detection System Overview Channel Selection: Genetic Algorithm (GA)

Classifiers

PreprocessingInput signals Feature Analysis Classifiers

Output Class labels

Purpose: Determine optimal decision boundaries between classes

Classifiers

Two simple classifiers were used to reduce computational complexity

Linear Discriminate Analysis (LDA): optimal linear boundary betweenclassesk Nearest Neightbors (kNN): majority voting, Euclidean distance wasused as the distance metric

10 fold cross validation process was used to obtain the averagedsimulation results shown in later sections

A portion of the training samples are used for training, and theremaining, unseen samples used for testing purposes

12/23

Experimental Setup

Experimental Setup: Database Description

Emobraine-eNTERFACE06

Validated public dataset, ideal forresults comparison

5 subjects, aged 22-38 ,

3 classes of affects, stimuli:IAPS images

Biosemi Active II: 64 channelsEEG headset

3 sessions and 30 trials eachsession

Valence

NegativelyExcited

Positively Excited

Calm

Arousal

13/23

Experimental Setup

Experimental Setup: Experiment Description

setup

Three experiments were conducted:with and without channel reduction

Two testing cases:

Suject Specific: Training andtesting samples are from the samesubjectCross Subject:Training and testingsamples are from all subjects

Raw EEG Signal

Multivariate EMD

Genetic Algorithm

Classification:kNN, LDA

Feature Analysis

Emotions

Referenced Emotive

EEG Reconstruction

14/23

Experimental Setup

Experimental Setup: Emotive Epoch v.s Biosemi Active II

Biosemi Active II is a medicalgrade EEG headset

Emotive Epoch is a popularconsumer grade EEG headset

Device Biosemi Active 2 Emotive EPOC SDK

Data Format EDF MAT

Resolution 24 bits ADC 16 bits (14 bits effective)

Sampling Rate 1024Hz 128 SPS (2018 Hz internal)

Channels 64 14

Channels in common AF3, F7, F3, FC5, FC6, F4, F8, AF4

15/23

Simulation Results

Simulation Results Using all Channels

Cross-Subject Emotion Recognition

Classifier Statistical Narrow-bands Power HOC Wavelet

5NN 81.39 82.62 90.77 77.44

LDA 59.18 63.49 79.64 55.90

This results show that EEG signal is feasible for affect analysis evenwith simple classfiers

kNN classifier provides better results which implies the features arenot linearly seperable

HOC is most representative for affect analysis

16/23

Simulation Results

Simulation Results with Channel Reduction Referecend toEmotive

Table: Cross-subject Recognition rate using only 8 electrodes

Classifier Statistical Narrow-bands Power HOC Wavelet

5NN 68.15 78.15 89.64 58.87

LDA 38.77 39.90 43.13 37.74

Currently in market consumer grade EEG headset is feasible for affectdetection applications

HOC and kNN combination provides the best performance

17/23

Simulation Results

Channels selected using GA

Given the cost and usability, consumer grade EEG headsets typicallyhave less than 14 channels.

Selected 8 channels

FPzFP2

AF4 AF8AFzAF3

FP1

AF7

Fz F2 F4F8F6

F1F3F5F7

FCz FC2 FC4 FC6 FC8FC1FC3FC5FC7

Cz C2 C4 C6 T8T7

CPz CP2 CP4 CP6 TP8TP7

C5 C3 C1

CP5 CP3 CP1

Pz P2 P4 P6P8

P10

P1P3P5P7

P9 POzPO3PO7PO4 PO8

OzO2O1

CMS DRL

Iz

Selected 14 channels

FPzFP2

AF4 AF8AFzAF3

FP1

AF7

Fz F2 F4F8F6

F1F3F5F7

FCz FC2 FC4 FC6 FC8FC1FC3FC5FC7

Cz C2 C4 C6 T8T7

CPz CP2 CP4 CP6 TP8TP7

C5 C3 C1

CP5 CP3 CP1

Pz P2 P4 P6P8

P10

P1P3P5P7

P9 POzPO3PO7PO4 PO8

OzO2O1

CMS DRL

Iz

18/23

Simulation Results

Simulation Results Summary

Channels 5NN LDA

All 54 channels 90.77 79.64

8 - Referenced to commercial device 89.64 43.13

Channel selected using GA18 89.23 62.0510 88.77 56.316 89.79 53.21

The experimental results indicate the following:

Recordings from higher number of channels give more classificationaccuracy

Headsets with much smaller number of electrodes are feasible foraffect detection analysis, however the choice of classifier is veryimportant

Linear classifier provides better classification accuracy on EEG sensorconfiguration selected through GA

19/23

Simulation Results

Recognition Rates Compared to the State of The ArtResults

Studies Database PreprocessingMethod

Features Maximum CorrectClassificaiton Rate

Khalili et al.[1]

eNTERFACE06,IASP, 3 emotions

4−45Hz bandpassfilter

Statistical 40%( LDA), 51%(kNN)

Comparing results from Khalili study, our approach is able to increasethe performance from 51% to 81.39% through the use of an moreeffective preprocessing algorithm, and a maximum recognition rate of90.77%

Limitations: lack of international standard database for comparingresults

[1] Z. Khalili and M. Moradi, Emotion detection using brain and peripheral signals,” in

Biomedical Engineering Conference, 2008. CIBEC 2008. Cairo International, pp. 1-4,

dec. 2008.20/23

Simulation Results

Effects of Sampling Rate on Correct Recognition Rate

Sampling rate vs. correct recognition rate using all electrodes andLDA, kNN classiffier for all four feature types

256 512 10240

10

20

30

40

50

60

70

80

90

100

Sampling Frequency

Cor

rect

Rec

ogni

tion

Rat

e

Sampling Frequency vs. Correct Recognition Rate (all electrodes, LDA)

statisticalnarrow−bandHOCwavelet−based

256 512 10240

10

20

30

40

50

60

70

80

90

100Correct Recognition Rate vs. Sampling Rate (all electrodes, kNN)

Sampling RateC

orre

ct R

ecog

nitio

n R

ate

statisticalnarrow−bandhocwavelet−based

Down sampling degrades the performance of linear classifier morethan kNN

Performance using HOC features degrades significantly when downsampled from 512Hz to 256 Hz

21/23

Summary of Research Contributions

Summary of Research Contributions

Applied a novel signal processing algorithm, Multivariate EmpiricalMode Decomposition (MEMD) as a time-varying filtering technique

Defined emotion specific channels using Genetic Algorithm (GA),which will be useful for future headset design

Implemented a framework for Affect detection using EEG signals, aEEG signal processing system was designed and 4 feature extractionalgorithms along with 2 classifiers were implemented for affectdetection

Proposed solutions for various practical issues, such as emotionelicitation, sampling rate implications, artefact reduction

22/23

Future Works

Future Works

To generalize my research findings

Proposed system should be tested with a larger dataset under naturalconditionsOptimal fusion techniques at the feature and decision level should beincluded for better performance

To broaden the application scenarios

Multimodality analysis should be studied, e.g., using face images fordata labeling real-time testingReal time processing concerns should be addressed

23/23

Future Works

Publications

1 Haiyan Xu, Konstantinos N. Plataniotis, Affect Recognition UsingEEG Signal, 2012 IEEE International Workshop on MultimediaSignal Processing, Banff (Canada), 2012 (accepted).

2 Haiyan Xu, Mohammad Shahin Mahanta, Chris Aimone,Konstantinos N. Plataniotis, ”A Real Time Portable System forClassification of Meditation States Using EEG Signals” 2012IEEE International Workshop on Multimedia Signal Processing, Banff(Canada), 2012 (under review).

3 Haiyan Xu, Gaurav Jain, Konstantinos N. Plataniotis, ”AutomatedAffect Detection using Facial Images and EEG Signals”, TheIEEE Transactions on Affective Computing (TAC).(To be submitted)

24/23

Genetic Algorithm

Crossover Point

Parent #1 1 0 0 1 0 1 0 0 1 1

Parent #2 0 1 0 1 1 1 1 0 1 1

Child #1 0 1 0 1 0 1 0 0 1 1

Child #2 0 1 0 1 1 1 1 0 1 1

Parent #1 1 0 0 1 0 1 0 0 1 1 Child #1 1 0 0 1 1 1 0 0 1 1

Mutation Point

Mutation Process

Crossover Process

Initial Population [1001010101] ……… [010100111]

24/23

Experimental Setup Flowchart

Initial Signal

MEMD-IMFs Extraction

IMFs Selection

Signal Reconstruction

Statistical Narrow-band wavelet HOC

MEMD

Filte

ring

Classification

Affect Recognition Rate

Featu

re An

alysis

0

0

1

1

0

1

0

0

0

0

0

0

0

LDA, kNN

Feature Vector

Pos. exited, Neg. excited, Neutral

24/23

Multivariate EMD Algorithm

24/23

Multivariate EMD Algorithm

Algorithm 2. Multivariate extension of EMD Algorithm (MEMD)

1. Choose a suitable pointset for sampling on an (n − 1) sphere.2. Calculate a projection, denoted by Pθk (t)Tt=1, of the input signal {v(t)}Tt=1 along the directionvector xθk , for all k (the whole set of direction vectors), giving pθk (t)Kk=1 as the set of projections.

3. Find the time instants{tθki

}corresponding to the maxima of the set of projected signals

pθk (t)Kk=1.

4. Interpolate[tθki , v(tθki )

]to obtain multivariate envelope curves eθk (t)Kk=1.

5. For a set of K direction vectors, the mean m(t) of the envelope curves is calculated as

m(t) =1

K

K∑k=1

eθk (t) (1)

6. Extract the ’detail’ d(t) using d(t) = x(t)−m(t). If the ’detail’ d(t) fulfills the stoppagecriterion for a multivariate IMF, apply the above procedure to x(t)− d(t), otherwise apply it tod(t).

23/23

Feature Anlalysis Comparison

Strength of each feature analysis algorithm

statistical:

HOC:intuitive, analyzing the up and down movement of the EEGsignal.

Process is iterative, the higher the order, the more computationalexpensiveFilter the time domain signal (backward difference operator). count thenumber of zero crossingsA measure of EEG oscillation; the more pronounced the oscillation, thehigher the expected number of zero-crossings is and vice versa.Zerocrossings and spectrum: number of zero crossings indicates thedominate frequency

Wavelet-based Entropy analysis: the entropy feature is basically anon-linear in nature and captures the nonlinearity of the EEG signalsover different emotions than other statistical features.

23/23

Statistical FeaturesThe statistical features used to form the proposed FVs are defined as (Xi , i = 1 · · ·N is the raw N-sample EEG signal) givenin the following.

1 The mean of the raw signal

µx =1

T

T∑t=1

X (t) = X (t) (2)

2 The standard deviation of the raw signal

σx =

√√√√ 1

T

T∑t=1

(X (t)− µx)2 (3)

3 The mean of the absolute values of the first differences of the raw signal

δx =1

T − 1

T−1∑t=1

|X (t + 1)− X (t)| (4)

4 The mean of the absolute values of the first differences of the standardized signal

δx =1

T − 1

T−1∑t=1

∣∣X (t + 1)− X (t)∣∣ =

δxσx

(5)

5 The mean of the absolute values of the second differences of the raw signal

γx =1

T − 2

T−2∑t=1

|X (t + 2)− X (t)| (6)

6 The mean of the absolute values of the second differences of the standardized signal

γx =1

T − 2

T−2∑t=1

∣∣X (t + 2)− X (t)∣∣ =

γxδx

(7)

23/23

Higher Order Crossings

Higher Order Crossing Features Let X1,X2, ...,XN be a zero-meanstationary time series, the zero-crossing count in discrete time is defined asthe number of symbol changes in the corresponding clipped binary timeseries

Zt =

{1, ifXt ≥ 00, ifXt < 0

(8)

The number of zero-crossings, denoted by D, is defined in terms of Zt

D =N∑t=2

[Zt − Zt−1]2, 0 ≤ D ≤ N − 1 (9)

23/23

Wavelet-based Features

Daubechies fourth-order orthonormal bases (db4) was employed tocalculate the wavelet coefficients at the lth scale, CX (l , n), thatcorrespond to the alpha band (812Hz)and Beta band (13− 30Hz) wereused to estimate the wavelet energy and wavelet entropy, given by

ENGl =2S−l−1∑n=1

|CX (l , n)|2 ,

N = 2S , 1 < l < S . (10)

ENTl = −2S−l−1∑n=1

|CX (l , n)|2 log(|CX (l , n)|2),

N = 2S , 1 < l < S . (11)

The parameters of 10 and 11 were used as a feature vector fw (i.e.,f jw ,i = [ENGl ,ENTl ]).

23/23