Paper Title (use style: paper title)

Minimally Invasive Assessment of Mental Stress based on Wearable Wireless Physiological Sensors and Multivariate Biosignal Processing

978-1-5386-9301-8/19/$31.00 ©2019 IEEE

Riccardo Pernice Department of EngineeringUniversity of PalermoPalermo, Italyriccardo.pernice@unipa.it

Mariolino De CeccoDept. of Industrial Engineering, University of Trento Povo, TN, Italymariolino.dececco@unitn.it

Giandomenico NolloDept. of Industrial Engineering, University of Trento Povo, TN, Italygiandomenico.nollo@unitn.it

Alessandro Busacca Department of EngineeringUniversity of PalermoPalermo, Italyalessandro.busacca@unipa.it

Matteo ZanettiDept. of Industrial Engineering, University of Trento Povo, TN, Italymatteo.zanetti@unitn.it

Luca FaesDepartment of EngineeringUniversity of PalermoPalermo, Italyluca.faes@unipa.it

Abstract—The development of connected health technologies for the continuous monitoring of the psychophysical state of individuals performing daily life activities requires the aggregation of non-intrusive sensors and the availability of methods and algorithms for extracting the relevant physiological information. The present study proposes an integrated approach for the objective assessment of mental stress which combines wirelessly connected low invasive biosensors with multivariate physiological time series analysis. In a group of 18 healthy subjects monitored in a relaxed resting state and during two experimental conditions inducing mental stress and sustained attention (respectively, mental arithmetic and serious game), we collected simultaneously multichannel EEG, one lead ECG, respiration and blood volume pulse. From these signals, synchronous physiological time series were extracted measuring the δ, θ, α, and β EEG amplitudes, the heart period, the sampled respiratory activity and the pulse arrival time. For each condition, five minute windows of each of these seven time series were characterized with measures in the time domain (mean, standard deviation) and in the information domain (self entropy, measuring time series regularity). We show that the dynamical activity of the different physiological systems is affected in a different way by the alteration of the psychophysical state of the subjects induced by stress, and that the measures in the two domains can elicit complementary information about mental stress and sustained attention. These results advocate the feasibility of connected health technology for minimally invasive, automatic classifiers of different levels of mental stress in real life scenarios.

Keywords—physiological signals, EEG, stress assessment, time series analysis, wearable devices

Introduction

People undergoing prolonged cognitive activities are often affected by fatigue periods and mental stress [1], which can be defined as a non-specific response of the body or the mind to any demand of change [2]. In particular, stress arises when the body is no more able to react properly to excessive physical or psychological demands [3]. Stress is dependent on individual’s physiological and psychological status [4], and is often determined by circumstances and events that are novel and unpredictable. Environmental factors that trigger stress are usually referred as stressors [4]. The number of active stressors and the length of exposure of the body to them determine the consequences of stress [3].

Sustained cognitive load can provoke a stress condition known as mental stress [1]. In worst conditions, mental stress represents one of the factors which can ultimately increase the risk of heart attacks, depression and stroke [2]. On the other hand, mild mental stress can decrease the responsiveness of the central-peripheral regulatory system causing a poorer health [2], inability to perform further efforts, or even sudden changes in mood [1]. For these reasons, mental stress can be directly correlated to the quality of life of individuals [5]. When mental fatigue arises, the subject is not able to maintain his/her task performance at a suitable level and this can be very dangerous especially for workers in industry or for drivers, since it can cause incidents [1], [6]. Especially for elderly or disabled subjects, the inability to perform a certain task due to mental stress can be quite frustrating [5]. Therefore, carrying out accurate monitoring of stress level can be useful for disease prevention and individuals’ wellbeing. The development of suitable signal processing techniques could allow in the future a real time stress assessment, thanks also to the more and more widespread employment of minimally invasive portable or wearable devices for physiological parameter monitoring (as an instance see [7]–[11]). This would be useful especially within an Ambient Assisted Living (AAL) approach, that is promising in facilitating the elderly to live longer in residential environments, with their families [12].

Different techniques have been employed in the literature to monitor and assess the stress level. For instance, in the automotive field [6], different driver monitoring systems are presented which rely on the continuous observation of changes in facial expression, and of eye/head movements. Since stress causes changes in autonomic nervous system (ANS) activity [13], several research works have focused on the analysis of features extracted from Heart Rate Variability (HRV) [14], [15], which represents an important index for assessing cardiac health and the ANS status [14], [16]. HRV analysis is indeed useful to assess the sympathetic and parasympathetic ANS activities (e.g. through frequency domain analysis) and to extract from them information on the psycho-physiological status of a subject [14], [15]. In fact, different research works have correlated the variation of HRV features to mental stress, either considered alone or in combination with features derived from different physiological systems, including the cardiovascular, respiratory and hormonal systems [7], [17]–[21]. In addition, a number of research works have investigated the feasibility of exploiting electroencephalography (EEG) signals to assess stress status [1], [2], [22], finding suitable indexes that reflect the amplitude of the different brain rhythms and reflect specific spatial locations where EEG activity is recorded in the scalp. In particular, in [22] authors demonstrated that EEG signals can be used to reliably discriminate between different multiple levels of mental stress.

The discussion above clearly documents the importance and the need of a so-called “network physiology” approach, whereby the dynamical activity of multiple organ systems is monitored simultaneously through the measurement of physiological time series, to assess thoroughly the psycho-physiological state of an individual and its modifications elicited by stress. In a recent work, we pointed out the feasibility of exploiting this concept in a minimally invasive way by combining wearable sensors connected wirelessly with multivariate physiological time series analysis [5]. Here, we extend the study investigating the feasibility of combining cardiovascular and respiratory indexes with features derived mapping the EEG activity at multiple scalp sites, in order to identify the features which are more suitable to allow multilevel stress detection.

Materials and methods

The rationale of this work is to employ multiple low cost wearable biosensors monitoring the activity of different organ systems in a minimally intrusive way to quantify varying physiological states (see Fig. 1(a)). Moreover, exploiting the spatial distribution of the multichannel EEG, the monitoring of brain activity is performed at multiple scalp sites (Fig. 1(b)).

In the present study, 18 young healthy participants, with age ranging between 18 and 30, were monitored continuously during three different experimental conditions: a resting condition induced by watching a relaxing video (REST, 12 minutes), a stressful condition induced by the execution of a mental arithmetic test (MENTAL, 7 minutes), and a condition of sustained attention induced by instructing subjects to play a serious game (GAME, 7 minutes). Physiological monitoring was achieved employing a sensorized t-shirt, a wristband and a multichannel headset. The t-shirt was provided by Smartex (Prato, Italy) and was used to record the ECG at a sampling frequency of 250Hz and the respiration rate at 25 Hz. The E4 Empatica (Milano, Italy) wristband, instead, was employed to acquire the blood volume pulse (BVP), at 64 Hz. Finally, EEG signals were recorded using the 14 channels Emotiv EPOC PLUS (San Francisco, CA) wireless headset, 256 Hz per channel. Particular attention was paid to the correct positioning of the EEG electrodes [5], shown in Fig. 1(b). All the data were transmitted wirelessly to a personal computer via Bluetooth, and particular care was paid to correctly synchronize all the signals (see [5] for further details).

The acquired signals were processed as follows to extract synchronous time series (represented with a sampling time of 1 s) quantifying the dynamical activity of brain and peripheral systems. Starting from EEG signals, the power spectral density (PSD) was computed employing a sliding window of 2 s and a 50% of overlap, and calculating within each window the EEG power relevant to the δ (0.5-3 Hz), θ (3-8 Hz), α (8-12 Hz) and β (12-25 Hz) frequency bands. The spatial distribution of each EEG time series was obtained repeating the same procedure at each EEG electrode location.

Schematic representation of (a) the brain and peripheral time series monitored according to a network physiology concept, and of (b) the spatial distribution of the monitored brain signals.

Peripheral physiological signals consisted in the electrocardiogram (ECG), the respiratory signal, and the BVP signal. With regard to ECG, a high-pass filter (half power frequency of 1 Hz) and a low-pass filter (half power frequency of 20 Hz) were employed to remove baseline wander and high-frequency noise, respectively; afterwards, R peaks were extracted using the template matching algorithm [23], and RR intervals were computed on a beat-to-beat basis. The respiration time series (RESP) was obtained sampling the respiratory signal at the occurrence of each R peak in the ECG. Finally, the dynamical activity of the cardiovascular system was monitored computing the time series of the pulse arrival time (PAT), i.e., the time delay between ECG and blood volume pulse [9], [24], as the temporal distance between the R peak of the ECG and the corresponding point of maximum derivative in the BVP signal [24]. For each signal, time series of 300 points were selected, reducing the transient behaviors between the conditions and testing the stationarity by assessing the stability of mean and variance [5].

Afterwards, the time series of brain and cardiovascular signals were analyzed using measures in both time and information-theoretic domains, with the aim of extracting features descriptive of the state of each physiological system in correspondence of each analyzed time window. Specifically, in the time domain the series were described taking the average value (MEAN) and the standard deviation (STD), respectively representing the average level and the overall variability of the time series. In addition, the level of regularity of the time series, obtained studying the repetitiveness in time of patterns extracted from the series, was measured in the information domain. In detail, information-theoretic analysis was performed computing the amount information stored in the time series, quantified by the so-called self entropy (SE) [25]. The self-entropy is the part of the entropy of the present sample that can be derived from the past samples. In this work, we employed the k-nearest-neighbor estimator of the SE which guarantees bias compensation [26]; we refer also to [11], [27] for further detailed information.

Statistical analysis was applied on the distributions of each of the three indexes, computed from each of the seven physiological time series, in order to assess the significance of the differences of the index across the three considered states (REST, MENTAL, and GAME). Specifically, the Student’s signed rank test was applied to compare the distributions between the different conditions, assessing whether they come from a distribution of the same median. The null hypothesis of equal median was rejected (i.e., p-value < 0.05) when a lack of agreement between the two distributions under test was found.

Results and Discussion

Figure 2 shows the color-coded values of the mean EEG power in the δ, θ, α, β bands computed across scalp locations in the three analyzed conditions. Markers represent graphically the electrode locations and the results of Student’s t-test between distributions.

Maps of mean EEG power (index MEAN) in the δ, θ, α, β bands, represended as the mean value across subjects computed in the three experimental conditions (REST, MENTAL, GAME). Markers are positioned at each electrode location, and are colored according to the results of statistical analysis (white: p<0.05 vs. MENTAL vs. REST or GAME vs. REST); # in the right panels indicate p<0.05 MENTAL vs. GAME.

Results show that during REST the power of and θ rhythms is distributed mostly in the frontal scalp regions, while the power of and β rhythms is distributed mostly in the right hemisphere [28]. The mean EEG tends to become higher after the transition from REST to MENTAL, the increase being statistically significant for the θ waves especially in the frontal regions, and for the β waves in occipital regions. The increase of the θ power, and the increase of the β power in the occipital regions, are in agreement with previous findings in the literature [29], [30]. On the other hand, variations are generally less evident during GAME, with almost no statistical differences observed in comparison with REST, and significantly lower EEG power often observed in comparison with MENTAL.

Similar remarks can be made with regard to the standard deviation of EEG power in the δ, θ, α, β bands computed across scalp locations in the three analyzed conditions (Figure 3). Indeed, comparing Figure 3 with Figure 2 one can see that an increase in the mean power is often accompanied by an increase in its variability across time. Taken together, these results suggest that stress elicits bursts of EEG activity in several scalp regions, which are spectrally located in the θ band as regards the frontal regions, and in the β band as regards the occipital regions.

Maps of standard deviation of EEG power (index STD) in the δ, θ, α, β bands, represented as the mean value across subjects computed in the three experimental conditions (REST, MENTAL, GAME). Markers are positioned at each electrode location, and are colored according to the results of statistical analysis (white: p<0.05 vs. MENTAL vs. REST or GAME vs. REST); # in the right panels indicate p<0.05 MENTAL vs. GAME.

Figure 4 depicts the color coded values of the self entropy calculated across scalp locations in the three analyzed conditions for each of the δ, θ, α, β EEG power time series. Results suggest that the brain time series tend to be more regular in the regions of the left hemisphere and of the frontal lobe, compared with the other brain regions, especially regarding the and θ rhythms and also the other rhythms during mental stress. However, no statistically significant differences in the SE index were detected moving from REST to MENTAL. On the other hand, an overall decrease of SE, mostly involving the frontal scalp regions, was observed during GAME, with significant differences relevant to the and rhythms in comparison with REST, and the and rhythms in comparison with MENTAL. These findings suggest that sustained attention induced by serious games increases the complexity of the brain power dynamics in a way that is not documented for the more stressful condition elicited by mental arithmetic.

Maps of the self entropy of EEG power (index SE) in the δ, θ, α, β bands, represented as the mean value across subjects computed in the three experimental conditions (REST, MENTAL, GAME). Markers are positioned at each electrode location, and are colored according to the results of statistical analysis (white: p<0.05 vs. MENTAL vs. REST or GAME vs. REST); # in the right panels indicate p<0.05 MENTAL vs. GAME.

Figure 5 reports the distributions across subjects of the MEAN, STD and SE indexes computed for the cardiac, respiratory and cardiovascular time series in the three analyzed conditions.

Analysis of the RR intervals shows a significant decrease of the MEAN index (indicating higher heart rate) during mental stress which is instead not present during serious game. This result documents that the mild stress evoked by serious games is not able to elicit the tachycardia documented in this and previous studies (e.g. [17], [20]) during the stronger stress evoked by mental arithmetic. The standard deviation and self entropy of RRI significantly decrease both during mental stress and serious game conditions compared with rest, indicating that both mild and stronger stress levels reduce the variability of heart rate as well as the regularity of the cardiac dynamics. These results, partly observed before when comparing rest and stress states [19], indicate the usefulness of measures derived from heart rate variability to discriminate these different physiological states.

The analysis of the respiratory time series indicates an increase of the mean breath amplitude during stress conditions (statistically significant comparing GAME with REST), and a marked decrease of the respiratory variability (index STD) during GAME compared both to REST and to MENTAL. Moreover, the regularity of breathing reflected by SE decreased during stress, the difference being statistically significant only during MENTAL.

Finally, the analysis of the PAT time series shows a decrease of its mean values during the condition of mental stress, which might be related to the reduced Pre-Ejection Period previously observed in conditions during mental arithmetic workload [31].

Distributions (expressed as mean and standard error across subjects) of the three considered indexes (MEAN, STD, SE) computed for the RR, RESP and PAT time series in the three experimental conditions (REST, MENTAL, GAME). Results of the statistical analysis (Student’s t-test, p<0.05 between distributions) are reported by the lines connecting pairs of distributions.

Conclusion

Our analysis demonstrated that increasing levels of mental stress, induced by serious game and mental arithmetic, evoke different responses in the dynamical activity of different organ systems which can be detected by different metrics derived from time series analysis, thus posing the basis for a multilevel assessment of stress based on the concept of network physiology. In particular, we observed that time-domain and information-theoretic measures applied to the time series of specific brain wave amplitudes when measured in specific anatomical locations distinguished distinctly the EEG response to mental arithmetic and serious game, thus favoring their differential classification. In addition, the analysis of the dynamical activity of the peripheral organ systems provided robust indicators for the distinction between rest and stress, with some additional discriminative capability of mental stress with respect to sustained attention.

Future research in this direction should be devoted to extract more features descriptive of rest and stress states, also incorporating coupling indexes revealing the dynamical interaction within and between organ systems, and to exploit such extended set of features to design automatic classifiers of the stress states. These automatic classifiers, also taking advantage of the minimal invasiveness of the biosensors, would constitute an important step towards the development of connected health technologies for the continuous monitoring of the psychophysical state of subjects in real life scenarios.

Acknowledgment

The research has been supported by the grant ASTONISH, H2020-EU.2.1.1.7. – ECSEL, University of Palermo.

References

[1]C. Zhang and X. Yu, “Estimating mental fatigue based on electroencephalogram and heart rate variability,” Polish J. Med. Phys. Eng., vol. 16, no. 2, pp. 67–84, 2010.

[2]F. Al-Shargie, M. Kiguchi, N. Badruddin, S. C. Dass, A. F. M. Hani, and T. B. Tang, “Mental stress assessment using simultaneous measurement of EEG and fNIRS,” Biomed. Opt. Express, vol. 7, no. 10, pp. 3882–3898, Sep. 2016.

[3]C. M. Vander Weele, C. Saenz, J. Yao, S. S. Correia, and K. A. Goosens, “Restoration of hippocampal growth hormone reverses stress-induced hippocampal impairment,” Front. Behav. Neurosci., vol. 7, p. 66, Jun. 2013.

[4]R.-P. Juster et al., “A transdisciplinary perspective of chronic stress in relation to psychopathology throughout life span development,” Dev. Psychopathol., vol. 23, no. 3, pp. 725–776, 2011.

[5]M. Zanetti, L. Faes, M. De Cecco, M. Valente, G. Guandalini, and G. Nollo, “Assessment of mental stress through the analysis of physiological signals acquired from wearable devices,” Lect. Notes Electr. Eng., vol. in press, 2018.

[6]Q. Wang, J. Yang, M. Ren, and Y. Zheng, “Driver Fatigue Detection: A Survey,” in 2006 6th World Congress on Intelligent Control and Automation, 2006, vol. 2, pp. 8587–8591.

[7]D. Hernando, S. Roca, J. Sancho, Á. Alesanco, and R. Bailón, “Validation of the Apple Watch for Heart Rate Variability Measurements during Relax and Mental Stress in Healthy Subjects,” Sensors , vol. 18, no. 8. 2018.

[8]D. Agrò et al., “PPG embedded system for blood pressure monitoring,” in 2014 AEIT Annual Conference - From Research to Industry: The Need for a More Effective Technology Transfer (AEIT), 2014, pp. 1–6.

[9]D. Oreggia et al., “Physiological parameters measurements in a cardiac cycle via a combo PPG-ECG system,” in 2015 AEIT International Annual Conference (AEIT), 2015, pp. 1–6.

[10]Y. Yu, S. Chen, C. Huang, L. Ko, and C. Lin, “Portable and wireless EEG device used in sleep quality tracking,” in 2015 IEEE International Conference on Consumer Electronics - Taiwan, 2015, pp. 86–87.

[11]V. Vinciguerra et al., “PPG/ECG Multisite Combo System Based on SiPM Technology,” in Sensors. CNS 2018. Lecture Notes in Electrical Engineering book series (LNEE), vol. 539, Andò B. et al. (eds), Ed. Springer, Cham, 2019, pp. 353–360.

[12]A. Dimitrievski, E. Zdravevski, P. Lameski, and V. Trajkovik, “A survey of Ambient Assisted Living systems: Challenges and opportunities,” in 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP), 2016, pp. 49–53.

[13]S. D. Kreibig, “Autonomic nervous system activity in emotion: A review,” Biol. Psychol., vol. 84, no. 3, pp. 394–421, 2010.

[14]F. Shaffer and J. P. Ginsberg, “An Overview of Heart Rate Variability Metrics and Norms,” Front. public Heal., vol. 5, p. 258, Sep. 2017.

[15]M. Malik et al., “Heart rate variabilityStandards of measurement, physiological interpretation, and clinical use,” Eur. Heart J., vol. 17, no. 3, pp. 354–381, Mar. 1996.

[16]U. R. Acharya, K. P. Joseph, N. Kannathal, L. C. Min, and J. S. Suri, “Heart rate variability,” in Advances in cardiac signal processing, Springer, 2007, pp. 121–165.

[17]J. Taelman, S. Vandeput, E. Vlemincx, A. Spaepen, and S. Van Huffel, “Instantaneous changes in heart rate regulation due to mental load in simulated office work,” Eur. J. Appl. Physiol., vol. 111, no. 7, pp. 1497–1505, 2011.

[18]G. M. Schneider, D. W. Jacobs, R. N. Gevirtz, and D. T. O’Connor, “Cardiovascular haemodynamic response to repeated mental stress in normotensive subjects at genetic risk of hypertension: evidence of enhanced reactivity, blunted adaptation, and delayed recovery,” J. Hum. Hypertens., vol. 17, p. 829, Nov. 2003.

[19]M. Valente et al., “Univariate and multivariate conditional entropy measures for the characterization of short-term cardiovascular complexity under physiological stress,” Physiol. Meas., vol. 39, no. 1, p. 14002, 2018.

[20]R. Pernice et al., “Comparison of Short Term Heart Rate Variability Indexes evaluated through Electrocardiographic and Continuous Blood Pressure Monitoring,” Med. Biol. Eng. Comput., 2019.

[21]G. Acerbi et al., “A Wearable System for Stress Detection Through Physiological Data Analysis BT - Ambient Assisted Living,” 2017, pp. 31–50.

[22]F. Al-shargie, T. B. Tang, N. Badruddin, and M. Kiguchi, “Towards multilevel mental stress assessment using SVM with ECOC: an EEG approach,” Med. Biol. Eng. Comput., vol. 56, no. 1, pp. 125–136, 2018.

[23]G. Speranza, G. Nollo, F. Ravelli, and R. Antolini, “Beat-to-beat measurement and analysis of the RT interval in 24 h ECG Holter recordings,” Med. Biol. Eng. Comput., vol. 31, no. 5, pp. 487–494, 1993.

[24]M. Gao, N. B. Olivier, and R. Mukkamala, “Comparison of noninvasive pulse transit time estimates as markers of blood pressure using invasive pulse transit time measurements as a reference,” Physiol. Rep., vol. 4, no. 10, p. e12768, May 2016.

[25]J. S. Richman and J. R. Moorman, “Physiological time-series analysis using approximate entropy and sample entropy,” Am. J. Physiol. Circ. Physiol., vol. 278, no. 6, pp. H2039–H2049, Jun. 2000.

[26]L. Faes, D. Kugiumtzis, G. Nollo, F. Jurysta, and D. Marinazzo, “Estimating the decomposition of predictive information in multivariate systems,” Phys. Rev. E, vol. 91, no. 3, p. 32904, 2015.

[27]W. Xiong, L. Faes, and P. C. Ivanov, “Entropy measures, entropy estimators, and their performance in quantifying complex dynamics: Effects of artifacts, nonstationarity, and long-range correlations,” Phys. Rev. E, vol. 95, no. 6, p. 62114, Jun. 2017.

[28]B. Kornrumpf, O. Dimigen, and W. Sommer, “Lateralization of posterior alpha EEG reflects the distribution of spatial attention during saccadic reading,” Psychophysiology, vol. 54, no. 6, pp. 809–823, 2017.

[29]M. Gärtner, S. Grimm, and M. Bajbouj, “Frontal midline theta oscillations during mental arithmetic: effects of stress,” Front. Behav. Neurosci., vol. 9, p. 96, Apr. 2015.

[30]T. Kakizaki, “Occipital midline EEG and subjective rating of task difficulty as indices of mental task strain,” Eur. J. Appl. Physiol. Occup. Physiol., vol. 56, no. 2, pp. 163–168, 1987.

[31]S. L.-O. Martin et al., “Weighing Scale-Based Pulse Transit Time is a Superior Marker of Blood Pressure than Conventional Pulse Arrival Time,” Sci. Rep., vol. 6, p. 39273, Dec. 2016.