1

© 2005, it - instituto de telecomunicações. Todos os direitos reservados.

Cardiotechnix 2014 - Rome, October 26

Pervasive ECG: Emotion and Identity at your Fingertips

Ana Fred Instituto de Telecomunicações Instituto Superior Técnico

2

ECG in Health Assessment

3

ECG in Diagnosis

4

Synergies between Cardiology and Technology

Cardiology

Signal Acquisition / Measurement

Digital Recording

Automated Analysis

5

Automated ECG Interpretation

Outcomes

• Interpretation • Test reporting • Computer-Aided Diagnosis

Tools

• Pattern Recognition • Artificial Intelligence • Knowledge Bases

Signal Acquisition

Devices

• ECG Data • Medical Expertise

6

ECG Measurement

7

ECG Measurement

In hospital signal acquisition

8

ECG Measurement

Ambulatory devices: holter

9

ECG Measurement

Portable devices

Wearable health monitoring systems

Wearable health monitoring systems

Pervasiveness / transparency

12

Technology: Early days

One-lead setup• Chest mounted sensor• Three electrodes (pre-gelled)• Placement on the subject

13

Technology: Today

Next generation one-lead setup• Hand palms/fingers sensor• Two electrodes (no gel)• Integrable in everyday items

Outline of our contributions• Custom circuitry design• Algorithms and hardware setup

Non-metallic electrode materials• Electrolycras• Capacitive sensors

X

14

The technology is already here!

•  Commercial systems

•  Case with sensors communicating via Bluetooth with the smart phone

15

The technology is already here!

•  DIY systems that allow monitoring of physiological data

BITalino

www.bitalino.com

16

BITalino: www.bitalino.com

ECG

ACC

EMG

EDA

Embedded Systems

collaboration with

Auto Industry

Bike Sharing collaboration with

18

Pervasive ECG

On-the-person Off-the-person

19

Physiological Computing

Study and development of interactive systems that sense and react to the human body

20

Physiological Computing – Most basic sort

21

Physiological Computing – bio-cybernetic loop

Goal: translate patterns of physiological activity into meaningful interaction

P

R

Q S

T

Electrical activity of the heart

•  Health monitoring •  Emotion detection •  Person identification

Electrocardiography | ECG

23

Physiological Computing – ECG-based Authentication

subsampling

features

System Database Matcher

User ID

Decision policy

Accept Reject

24

Physiological Computing – ECG-based Identification

subsampling

features

System Database Matcher

User ID

25

Physiological Computing – Emotional Assessment

features

Assess psychological states

Trigger real-time adaptation

26

Physiological Computing and Pattern Recognition

Sensor

Pre-

processing

Feature

Extraction

Decision

Rule object

Typical Structure of a PR System

27

Pattern Recognition

•  Finding patterns in data •  Theory and Methods to solve real world problems

28

Physiological Computing and Pattern Recognition

Sensor

Pre-

processing

Feature

Extraction

Decision

Rule object

Emotion assessment

…

Sadness Joy Surprise

29

Physiological Computing and Pattern Recognition

Sensor

Pre-

processing

Feature

Extraction

Decision

Rule object

User identification

ID1 … IDn

P

R

Q S

T

Electrical activity of the heart

•  Health monitoring •  Emotion detection •  Person identification

Electrocardiography | ECG

31

Heart Rate Variability

•  Field of Psychophysiology: •  HRV is related with emotional arousal

• Emotional strain • Anxiety • Stress

32

The Physiology of Emotion: Lie Detectors

Polygraph: Machine commonly used in attempts to detect lies. •  Measures several arousal responses that accompany

emotion •  perspiration •  heart rate •  blood pressure •  breathing changes •  Galvanic skin response (gsr)

1921

33

The Physiology of Emotion: Lie Detectors

Polygraph: Machine commonly used in attempts to detect lies. •  Measures several arousal responses that accompany

emotion •  perspiration •  heart rate •  blood pressure •  breathing changes •  Galvanic skin response (gsr)

20 subjects 20 to 60 years old Two emotion elicitation protocols consisting of video sequences to elicit: •  Anger •  Amusement •  Surprise •  Sadness •  Fear •  Disgust •  Neutral

Filipe Canento, Ana Fred, Hugo Silva, Hugo Gamboa, André Lourenço, Multimodal Biosignal Sensor Data Handling for Emotion Recognition, IEEE Sensors, 2011

Emotional Assessment

Emotional Assessment, early days …

36

… now, off-the-person approach

VitalyPlay Game station remote controller with ECG and EDA signal acquisition capabilities

37

HiMotion Project: Concentration Test

•  Follow, line by line, the a list of numbers and identify pairs of numbers that add to 10

•  When in a line the subject can correct the marked/unmarked pairs •  A test page contains a matrix of 20 lines per 40 columns of numbers

The game is inspired in a concentration test from a MENSA

38

Evidence Accumulation Clustering (EAC)

➔ Combine the results of multiple clusterings into a single data partition by viewing each clustering result as an independent evidence of data organization •  Learns pairwise similarity between patterns, unveiling the underlying pattern

structure

Produce a clustering

ensemble (CE) Combine Evidence Extract the final data partition

Validate the combined partition P*

iKi CCCP },,,{ 21 …=

P1

P2

... PN

Clustering Ensemble

CE

Combined Solution P*

Learned Paiwise Similarity

(co-assocs)

Signal processing: Filtering

Segmentation

Mean wave computation: 10 heart beats

Feature Extraction

ECG

Mean Wave

subsampling

features

Feature extraction and

data representation

ECG signal acquisition

Concentration test (ECG signal generation)

Time Series: Acquisition and Processing

Feature extraction and

data representation

ECG signal acquisition

Concentration test (ECG signal generation)

Time Series: Acquisition and Processing

Evidence Accumulation

Clustering ensemble generation

Learning Similarities

Co-association matrix Representation

Temporal evolution of the clusters

time

time

TimeSeries

Feature Vectors of the Temporal Data Samples

41

State Detection: Visual Inspection

I.Samples of each temporal segment belong to a single cluster

II. Samples of each temporal segment belong to distinct combinations of clusters

BioPrint Lisbon, July 2, 2012 @Ana Fred 2012

42

Challenge: Continuous Time Evolution

time

time

time

43

Challenge: Continuous Time Evolution

Clusters Time evolution start

start

end

end

Feature extraction and

data representation

ECG signal acquisition

Concentration test (ECG signal generation)

Time Series: Acquisition and Processing

Evidence Accumulation

Clustering ensemble generation

Learning Similarities

Co-association matrix Representation

Temporal evolution of the clusters

Detection of differentiated temporal

states

Elimination of transient time activity

time

time

TimeSeries

Feature Vectors of the Temporal Data Samples Genetic Algorithm

P

R

Q S

T

Electrical activity of the heart

•  Health monitoring •  Emotion detection •  Person identification

Electrocardiography | ECG

Biometrics •  not practical for everyday use

•  prone to artificial replicas

•  cannot tell if the person is alive

•  only recognize momentarily Con

vent

iona

l

47

ECG vs Conventional Approaches

Key Card PIN Finger-print Face Iris Keystroke ECG

Reliable? J J K J K J J J Easy to use? J J J J J K J J

Hard to “steal”? L L L L L J K J Knows if the person is alive? L L L L L L L J

Practical for everyday use? J J J J J L K* J Can be used continuously? J L L L K L J* J

Tells health status information? L L L L K L L J Reveals emotion? L L L L K L L J

Pervasive acquisition

On-the-person Off-the-person

ECG Biometrics

•  Universality •  Performance •  Measurability •  Acceptability •  Circumvention •  Permanence •  Uniqueness

Discernibility of templates from different subjects (i.e. inter-subject variability)

Study Sample Size ECG Lead AP (%) IP (%)

Zhang and Wei, 2006 502 I NA 85.3

Odinaka et al., 2010 269 Electrodes on lower ribcage 0.37 (EER) 99

Shen et al., 2010 168 I (hands) NA 95.3

Safie et al., 2011 112 I 94.54 (AUR) NA

Irvine et al., 2003 104 NA NA 91

AP: Authentication Performance IP: Identification Performance EER: Equal Error Rate

AUR: Area Under Curve NA: Not Available

ECG | Uniqueness

51

ECG-based Biometrics: Fiducial Approach

52

ECG: Feature Extraction

Signal processing Filtering Segmentation

Feature Extraction

Feature Selection

subsampling

53 features

Mean wave: 10 sequential non-overlapping heart beats

53

Experimental Results

•  Variation of the number of test patterns

•  Majority voting sequential classifier combination

H. Silva, H. Gamboa and A. Fred. Applicability of Lead V2 ECG Measurements in Biometrics. In Intl. Educational and Networking Forum for eHealth. 2007

H. Silva, H. Gamboa and A. Fred. One Lead ECG Based Human Identification with Feature Subspace Ensembles. In Machine Learning and Data Mining in Pattern Recognition, 2007

Identification rate of 99.70% using a group of 3 patterns

Identification rate of 99.9% using a group of 9 patterns

54

ECG-Based Authentication

EER for different number of mean ECG beats Hugo Gamboa, "Multi-Modal Behavioral Biometrics based on HCI and Electrophysiology",

PhD Thesis, Instituto Superior Técnico, May 2008 http://www.lx.it.pt/˜afred/pub/thesisHugoGamboa.pdf

55

ECG: Non-Fiducial Approach

Signal processing Filtering Segmentation

Quantization

Cross parsing

Mean wave: 10 sequential non-overlapping heart beats

56

Classifier:

1 -NN Classifier Training/Evaluation method:

50-fold cross validation

Results

57

Methods

•  Fiducial •  Non-fiducial •  Partially fiducial

3-NN classifier cosine distance

R

Q S

T P

58

Methods | Database

•  Collaboration with hospital specialized in cardiac issues

•  12-Lead ECG system - Philips PageWriter Trim III

•  10 seconds, 500 Hz, 16 bit •  4 332 records / 2 055 subjects •  832 normal records / 618 subjects •  Average age: 65 years

59

Authentication

Identification

•  Leave-one-out cross validation 4 templates per subject 10 runs

•  Population subsets: 5, 10, 20, 30, 40, 50 subjects 150 random groups

EER: Equal Error Rate FAR = FRR

EID = #errors / #tests

Methods | Evaluation

60

Authentication EER Authentication ROC

Case EER (%)

EID (%)

618 Subs. 9.01 15.64

Results

ECG Biometrics Timeline

FundamentalResearch

ACCESS

CONTROL

CONSUMER

ELECTRONICS

DataProtection

MultiBiometrics

ContinuousBiometrics

Customization HealthStatus

EmotionDetection

(Biel et al., 2001)(Kyoso and Uchiyama, 2001)

(Biel et al., 2001) L. Biel, O. Petterson, L. Phillipson, and P. Wide. ECG analysis: A new approach in human identification. IEEE Trans Instrumentation and Measurement, 50(3):808–812, 2001.(Kyoso and Uchiyama, 2001) M. Kyoso and A. Uchiyama. Development of an ecg identification system. In Proc. of the 23rd Annual Int’l Conf. of the IEEE EMBS, volume 4, pages 3721 – 3723 vol.4, 2001.(Israel et al., 2005) S. Israel, J. Irvine, A. Cheng, M. Wiederhold, and B. Wiederhold. ECG to identify individuals. Pattern Recognition, 38(1):133–142, 2005. (Silva et al., 2007) Hugo Silva, Hugo Gamboa, and Ana Fred. Applicability of lead v2 ECG measurements in biometrics. In Proceedings of the International eHealth, Telemedicine and Health ICT Forum (Med-e-Tel), 177–180, 2007.(Agrafioti et al, 2011) Foteini Agrafioti, Jiexin Gao, and Dimitrios Hatzinakos. Biometrics, chapter Heart Biometrics: Theory, Methods and Applications, Biometrics. InTech, 2011.(Lourenço et al., 2011) A. Lourenc ̧o, H. Silva, and A. Fred. Unveiling the biometric potential of Finger-Based ECG signals. Computational Intelligence and Neuroscience, 2011, 2011 (Silva et al., 2011a) Silva, H.; Lourenço, A.; Fred, A. L. N.; JBF Filipe; "Clinical Data Privacy and Customization via Biometrics Based on ECG Signals", Proc Information Quality in eHealth - USAB, Graz, Austria, Vol. , pp. 121 - 132, November, 2011.(Silva et al., 2011b) H. Silva, A. Lourenc ̧o, R. Lourenc ̧o, P. Leite, D. Coutinho, and A. Fred. Study and evaluation of a single differential sensor design based on electro-textile electrodes for ECG biometrics applications. In IEEE Sensors Conf., 2011(Silva et al., 2012) Silva, H.; Lourenço, A.; Fred, A. L. N.; "In-Vehicle Driver Recognition Based on Hands ECG Signals", Proc ACM SIGART SIGCHI International Conf. on Intelligent User Interfaces – IUI, pp. 25 - 28, February, 2012.(Silva et al., 2013) Silva, H.; Lourenço, A.; F. A. L. Canento; Fred, A. L. N.; "ECG Biometrics: Principles and Applications", Proc International Conf. on Bio-inspired Systems and Signal Processing - Biosignals - INSTICC, , Spain, 2013.(Lourenço et al., 2013) Lourenço, A.; Silva, H.; C. C. Carreiras; Fred, A. L. N.; "Outlier Detection in Non-Intrusive ECG Biometric System", Proc International Conf. on Image Analysis and Recognition, Póvoa de Varzim, Portugal,, 2013.

(Israel, 2005)

(Silva et al., 2007)

(Agrafioti et al., 2011) (Silva et al., 2012)

(Lourenço et al., 2013)(Silva et al., 2013)

(Lourenço et al., 2011)(Silva et al., 2011a)(Silva et al., 2011b)

on-the-person

off-the-person(Vitalidi)

ECG Biometrics Timeline

Security (Lourenço et al. 2011)

(Silva et al. 2011)

(Lourenço et al., 2011) A. Lourenc ̧o, H. Silva, and A. Fred. Unveiling the biometric potential of Finger-Based ECG signals. Computational Intelligence and Neuroscience, 2011, 2011 (Silva et al., 2011a) Silva, H.; Lourenço, A.; Fred, A. L. N.; JBF Filipe; "Clinical Data Privacy and Customization via Biometrics Based on ECG Signals", Proc Information Quality in eHealth - USAB, Graz, Austria, Vol. , pp. 121 - 132, November, 2011. (Silva et al., 2012) Silva, H.; Lourenço, A.; Fred, A. L. N.; "In-Vehicle Driver Recognition Based on Hands ECG Signals", Proc ACM SIGART SIGCHI International Conf. on Intelligent User Interfaces – IUI, pp. 25 - 28, February, 2012. (Silva et al., 2013) Silva, H.; Lourenço, A.; F. A. L. Canento; Fred, A. L. N.; "ECG Biometrics: Principles and Applications", Proc International Conf. on Bio-inspired Systems and Signal Processing - Biosignals - INSTICC, , Spain, 2013. (Canento et al., 2013) F. A. L. Canento; Lourenço, A.; Silva, H.; Fred, A. L. N.; "On Real Time ECG Algorithms for Biometric Applications", Proc International Conf. on Bio-inspired Systems and Signal Processing - Biosignals, 2013. (Lourenço et al., ?) Lourenço, A.; C. C. Carreiras; Silva, H.; Fred, A. L. N.; "Template Selection for ECG Biometrics", Submitted to the Journal of Integrated Computer Aided Engineering.

Customization (Silva et al. 2013) (Silva et al. 2012)

Aliveness Det. (Canento et al. 2013)

Continuous Recognition

(Lourenço et al. ?)

Health Status

Emotion Detection

addressed ongoing

63

TEAM - Collaborations

Students: •  Hugo Silva •  André Lourenço •  Carlos Carreiras •  Priscila Alves •  José Guerreiro •  Marta Santos •  Diana Batista

•  Francisco David •  Filipe Canento •  Joana Santos

Hospital Santa Marta •  Rui Cruz Ferreira •  Rui César das Neves •  Eduardo Antunes

Cruz Vermelha Portuguesa •  Marta Aires de Sousa •  Nuno Morujo

64

QUESTIONS?