Multimodal Data Collection Frameworkfor Mental Stress Monitoring

Saewon KyeJunhyung MoonJuneil LeeInho ChoiDongmi CheonKyoungwoo LeeDepartment of Computer ScienceYonsei UniversitySeoul, Republic of Korea(saewon.kye; jh.moon.cs; juneil.lee; inho.choi; dmcheon; kyoung-woo.lee) @yonsei.ac.kr

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.UbiComp/ISWC’17 Adjunct , September 11 – 16, 2017, Maui, HI, USA.Copyright © 2017 ACM ISBN 978-1-4503-5190-4/17/09 $15.00https://doi.org/10.1145/3123024.3125616

AbstractAccurate recognition of people’s responses to stress andtimely management of stress is one of the important as-pects of maintaining good health. However, recognizingsuch reactions is difficult since people react to stressfulevents in various ways. Accordingly, we propose a multi-modal stress monitoring framework to examine people’sphysiological and behavioral reactions to stressors. Ourframework consists of (i) multimodal data collection relatedto mental stress, (ii) stress inducing experiment in a labo-ratory, and (iii) signal examination software in real-time andafter the data collection. Through preliminary experiments,we have examined how people show various reactions todifferent kinds of stressful tasks. Finally, based on the pro-posed framework, we will establish a database contributingto the mental health sensing research.

Author KeywordsStress recognition; Wearable devices; Physiological signalmonitoring; Activity monitoring

ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g., HCI)]:Miscellaneous; J.3 [Computer Applications]: Life and medi-cal Science - Consumer Health

IntroductionOne of the key aspects in maintaining good health is throughmanaging mental stress. Continuous exposure to stresscan cause various health problems, such as depression,cardiovascular disease, etc [12]. Recognizing when peo-ple are getting stressed and properly relieving them canbe an effective method to maintain good health (referencerequired).

In efforts to monitor people’s mental health, various ap-proaches have examined people’s physiological and be-havioral responses to stressful events [4, 6, 13, 5, 8, 3, 10].However, since these studies rely on one or two wearabledevices, the proposed systems are not able to examinepeople’s physiological and behavioral responses to stres-sors at the same time. People may show various kindsof responses to the same stressors and the same personmay react in different way during different stressful environ-ments. For example, when people are presenting in front ofa large crowd, some might sweat while others might shaketheir hands. Further, a person might breathe irregularlywhen he or she meets someone for the first time whereasthe person might breathe increasingly fast during a presen-tation. Accordingly, people experiencing stressful eventsshould be examined in multiple perspectives using varioussensors.

Accordingly, we propose a multimodal framework for mon-itoring people during stressful events. Our framework col-lects people’s physiological and motion sensor data. In theframework, three commercial wearable devices, EmpaticaE4, Zephyr Bioharness Module, and LG Watch Style, anda IP (Internet Protocol) camera collect 8 different types ofsensor data at the same time. As for the data collection re-lated to stress, we have utilized popular stressors in existingresearch to induce stress to participants [1, 11, 7]. In ad-

dition, we also have developed a program, which monitorssensor data collected during the experiments. The programreceives various signals from the wearable devices, checksthe connectivity between wearable devices and the server,and visualizes the collected data comprehensibly. Basedon the proposed framework, we plan on constructing a pub-lic on-line database which broadens the understanding ofmental stress and further contribute to the affective comput-ing and mental health sensing communities.

Related Works on Stress RecognitionExisting research have used various wearable devices toestimate mental stress in several contexts. Typically usedphysiological sensors embedded in wearable devices areEDA (electro-dermal activity), PPG (photoplethysmogram),ECG (electrocardiogram), and Resp (respiration); whichmeasure skin conductance, heart rate, heart activity, andbreathing rate, respectively. In addition, motion sensorsfrom wearable devices, image, and audio sensors examinepeople’s behavioral features, such as movements, facialexpressions, voice and so on.

To continuously and unobtrusively monitor such signals, ex-isting research have utilized various wearable devices. Ta-ble 1 compares the state-of-the-art studies on stress mon-itoring against the proposed framework. Those researchhave appropriately captured whether people were stressedor not, but can be insufficient to cover various responsesagainst stress. For example, [3, 8, 9] are not able to monitorpeople’s behavioral responses during experiment, such asshaking one’s hands, since accelerometer and gyroscopeare excluded in the framework. Further, [4] fails to moni-tor people who show little facial expression and have nohabitual movements under stressful episodes. In [6], move-ment data are just used to remove noisy information andthey fail to capture behavioral features caused by stressors.

Table 1: Under the proposed framework, people’s physiologicaland behavioral reaction can be comprehensively examined

Reference EDA PPG ECG Acc (wrist) Acc (chest) Resp Image AudioProposed method D D D D D D D D

Mozos, 2017 [10] D D D

Chan, 2016 [3] D D

Koo, 2016 [8] D

Gjoreski, 2016 [5] D D D

Wu, 2015 [13] D D

Hovsepian, 2015 [6] D D D

Gao, 2014 [4] D

[10] does not investigate changes in heart activity under bystressful incidents and further, [13] lacks in considering theskin conductivity of people.

To properly examine various responses of people understressful environment, we have develop a multimodal datacollection framework which consists of 8 different sensorswidely used in stress recognition research as describedin Table 1. The data from the experiment will be used toanalyze people’s various kinds of reaction to stress and willbe further used to develop a stress recognition model whichwill reflect individual characteristics.

Figure 1: The wearable devicesused in the proposed frameworkshould minimize discomfort andrestriction on a person’s movement

Comprehensive Data Collection FrameworkData collection methodIn the proposed framework, the Empatica E4 Wristband, LGWatch Style, Zephyr Bioharness Module, and the IP (Inter-net Protocol) camera examine people’s reactions to stres-sors during the stress inducing experiment. The three wear-able devices can be easily and comfortably worn, and isembedded with the sensors used in existing stress recogni-tion research. Table 2 describes the sensors in each deviceand body parts to be worn on. Figure 1 shows one partic-ipant wearing all devices. The location of each device arechosen so that the data from each sensor can be accurately

acquired and can be comfortably worn by a person. Also,the IP camera is installed at a corner of the experimentalroom to record the test participant at a distance.

The Empatica E4 Wristband is worn on non-dominant handto monitor skin conductance, heart rate, and movement ofthe hand. One of the advantages of Empatica E4 is thatEDA signals are measured from the wrist rather than fin-gers. Typically, other wearable devices measure skin con-ductance from fingers which limits one’s activity while wear-ing the sensors. To minimize the discomfort in wearing thedevice, we utilize Empatica E4 Wristband of watch-typeband. Further, LG Watch Style is a state-of-the-art smart-watch embedded with accelerometer and gyroscope. Sincesmartwatches are widely used in everyday life, we hypoth-esize that it is one of natural ways to monitor a person’shand movement in both labs and during daily life. As forthe placement of each device, since physiological signalsmay contain noise data if there are movements during dataacquisition, we have decided to wear the Empatica E4 onnon-dominant hand. LG Watch Style is only embedded withmotion sensors. Accordingly, the smartwatch is worn onthe dominant hand where the test participants will makemore movements compared to that of non-dominant hand.

Table 2: Devices can be comfortably worn and are embeddedwith sensors used in existing stress recognition research

Device Sensor Features LocationEmpatica E4 EDA Skin conductance Non-dominant hand

PPG Heart rateAccelerometer Movement

Temperature sensor Body TemperatureLG Watch Style 3-axis Accelerometer Directional movement of hand Dominant hand

3-axis Gyroscope Rotation angle of handZephyr Bioharness Respiration sensor Respiration rate Around chest

ECG Heart rate variabilityTemperature sensor Body Temperature3-axis Accelerometer Posture

IP Camera Image sensor Video of experiment Facing participantAudio sensor Voice and noise

Figure 2: During the experiment, each test participant will go through 20 minutes of relaxation period and 20 minutes of stress induced period.

Zephyr Bioharness Module is a chest-worn device close tothe heart. The device is widely used for monitoring heartactivity of athletes during training and in games. Becausethe device can accurately measure physiological signals inthe presence of violent movements, we have hypothesizedthat the device can also be used in people’s ordinary day.Additionally, an IP camera is installed in the experimentalroom to monitor participants at a distance. This capturesnot only several physical movements and facial expressionsbut also voice. We aim to acquire meaningful data for esti-mating stress.

Experimental scenarioTo understand how people react against stressful episodes,we have designed a laboratory experiment based on theprotocols used in (references). Our experimental scenarioconsists of setup, rest, stress inducted, recovery, and de-briefing stages. In the setup stage, participants will be giveninstructions on the tasks that they will be performing andhow to wear the devices. To remove any potential bias thatcan be caused from prior knowledge, we disclose the realpurpose of the experiment at the debriefing stage. Duringthe debriefing stage, they will be told the goal of the experi-ment and detach the devices. As for inducing stress to peo-ple, the experiment participants of the conventional workshave performed utilized cognitive and socio-evaluativeworks. Since it has been demonstrated that they are suc-

cessful in inducing stress [1, 11], we base the design of ourexperiment on such tasks. Figure 2 shows the experimen-tal timeline for one participant excluding the setup and thedebriefing phases.

After the setup phase, the data collection from three wear-able devices and the IP camera will start and the partici-pants will move on to the rest period to neutralize their emo-tions before stress induction. In the rest period, the partici-pants will be watching a relaxing video for 10 minutes. Thevideo is a clip showing natural environment, such as flyingbirds or a lake, with slow-tempered relaxing music.

For the first stressor of our experiment, we utilize "e-crossing"[7] where the participants exercise a cognitive task. In thisperiod, the participants are given a piece of paper filled withtexts and are instructed to cross out as many "e"’s as possi-ble. This task lasts for 5 minutes and we call this sub-periodas "e-crossing1". After "e-crossing1" is finished, the partic-ipants are given another piece of paper with different textsand they would conduct a similar task. In this section, theyare instructed not to cross out "e" if there are three or more"e"’s in one word. For example, the participants should notcross out "e"’s in the word "between" since three "e"’s arein this word. This period is called "e-crossing2" and lasts for5 minutes. According to [7], people enter the stage of ego-depletion during the tasks and eventually receive stress by

(a) The incoming data from all three wearable devices can bemonitored in real-time

(b) The form visualizes the database and each graph can bemanipulated in various ways

Figure 3: The program is a centralized data collection software where the devices’ connectivity, incoming data and the constructed databasecan be viewed

performing such cognitive task.

After finishing the "e-crossing", the participant is involved ina socio-evaluative task, an interview. An interview has beenwidely utilized for inducing stress [1, 11]. Moreover, the par-ticipants are forced to speak in a non-native language toinduce even more stress. For the first part of this stage,which we refer as "Preparation" stage, the participants aregiven a piece of paper with the interview questions and pre-pare the answers for 5 minutes. After 5 minutes, a nativespeaker will come into a room and will conduct the interviewfor 5 minutes. The interviewer will ask previously handedout questions and some follow up questions until the giventime is over.

Once the whole stress induction periods of "e-crossing" and

the interview are finished, the participants are given a mate-rial to read for 10 minutes to recover from the stress. Duringthis period, which we refer as "Recovery", the participantswill be given a magazine which contains emotion neutralcontents. Once this period is over, we stop collecting thesensor data as well.

Data Examination ToolTo easily check the connection status between wearabledevices and a host device, and to intuitively examine in-coming signals, we have developed a C# application. Theapplication running on the host device (i) starts a connec-tion with each device and receives data, (ii) monitors theincoming data in real time, and (iii) organizes the data intoa database and visualizes them. Figure 3 presents mainpages of our application utilized in the experiments.

(a) "e-crossing" experiment increases participant’s overall EDA andheart rate

(b) During the interview experiment, the overall tension that theparticipant receives increases

Figure 4: The participant’s physiological signals change while performing the proposed stress inducing tasks

Figure 3a shows the form in which the three wearable de-vices can be connected to the server and connectivity ofthe devices during the experiment can be monitored. A usercan connect to each device and starts data collection in thetop left corner of the screen. Empatica E4 and Bioharnessmodule communicate with the server using Bluetooth. Asfor the LG Watch style, the device communicates data overTCP (Transmission Control Protocol) wireless. Once eachdevice is successful in connecting to the server, the colorof the corresponding device’s circle will change to blue. Assoon as the host device starts to receive the data, the pro-gram plots the data on the graph and saves the data to csvfile simultaneously. As for the Bioharness module, since themanufacturer provides limited access to their API for devel-opers, we utilize the sample bluetooth communication soft-ware provided by the company. The program monitors if thedevice is correctly acquiring signal and the sensor data aresaved within the device. The collected data is later exportedto the server.

Each sensor data is saved at a separate csv file. Later,

the collected data are organized into the database and arelater visualized in Figure 3b. On the box located at the leftof the form, there shows the list of the participants whereeach participant is identified as an unique subject num-ber. When each subject number is clicked, the collecteddata are shown in the form. The personal information isshown on the box located at the top of the form. Note thatthese data represents statistical information only for theresearch purposes and cannot be utilized to break one’sprivacy. Each sensor data is visualized in a graph. Oncethe start button is pressed, the graph shows how the sen-sor data changes according to time. The speed in whichthe program displays the data can be manipulated. Further,the program can be fast-forwarded or reversed to any giventime chosen by the program user.

Preliminary Experiment ResultTo examine the feasibility of the proposed framework andeffectiveness of the proposed stressors, we have conductedpreliminary experiments. Four people participated in theexperiment and they went through the rest period and either

"e-crossing" period or interview period while wearing allthree devices.

Figure 4 displays the graphs of EDA values measured fromEmpatica E4 and the heart rate calculated from Zephyr Bio-harness module. Figure 4a is the graph of the participant’schange in physiological signals during "e-crossing" and theparticipant in Figure 4b went through interview. Throughthe preliminary experiments, we were able to examine thechanges in physiological data during stress induction peri-ods. In both scenarios, both of the particiapnts’ EDA valueand heart rate value during stressor activities were higherthan those of during Rest period.

In addition to the increases in the values of EDA and heartrate, we were able to observe how people have various re-actions to different types of stressors. Compared to the par-ticipant who performed the interview, the participant whoperformed "e-crossing" showed more abrupt increased inthe EDA values. Such peaks are called as Skin Conduc-tance Responses (SCRs) and such phasic EDA reactionsoften occur when people are exercising cognitive tasks,such as decision making or anticipation [2]. Even thoughboth tasks increased the overall level of EDA values, moreSCRs can be examined during "e-crossing" than during theinterview since people more frequently involve in decisionmaking during "e-crossing" than during an interview.

(bpm) AVG MAX MIN

Rest 67 82 58e1 74 87 64e2 73 85 65

Rest 81 93 64Prep 110 134 76Intw 107 139 78

Table 3: The participant’s heartrate were higher during the stressinducing tasks compared to that ofduring rest period.

We have also examined how people’s heart rates are in-fluenced during each task. The average heart rate of bothparticipants were higher during stress inducing period thanthat of rest period. Table 3 shows the average, the mini-mum and the maximum heart rate of each period for eachexperiment. During the stress induction period, the overallheart rate was higher than that of during rest period. How-ever, the heart rate of participant in Figure 4a were rela-tively steady compared to that of the participant in Figure

4b. We hypothesize that the participant in Figure 4b wereable to control this heart rate by controlling how he breatheswhile he was talking.

ConclusionIn this work, we have proposed the multimodal frameworkfor collecting physiological and motion data of people understress. Physiological, motion and multimedia sensors fromthree commercial wearable devices and an IP camera areused to monitor people’s various reactions to stressors. Wehave also designed a stress inducing experiment whereparticipants will exercise cognitive and socio-evaluativetasks. For the signal checking and data collection duringthe experiments, we have developed a C# program whichmonitors incoming sensor data in real-time and later visual-izes the constructed database. Through the preliminary ex-periments, we were able to examine the effectiveness of theproposed stressors in inducing stress to the participants.During the stress induction period, the participants’ physio-logical signals not only increased but also showed variouscharacteristics in response to different kinds of stressors.

The proposed framework and the experimental setup havebeen approved by Yonsei University Institutional ReviewBoard. As for the future work, based on the data collectedfrom the experiments, we plan on building a stress esti-mation model. In particular, we expect to build not only apersonalized model but also the universal model where themodel can be applied to various types of people. As for im-proving our framework, we further plan on investigating theeffective location to wear each sensor. For example, EDAcan also be measured from a person’s ankle [14]. Eachsensing location will be evaluated in terms of accuracy, ef-fectiveness in estimating stress and comfortability in wear-ing the sensor in various contexts. Finally, the constructed

database and the software will be open-sourced to con-tribute to the mental health sensing community.

AcknowledgementsThis work was supported by Institute for Information &Communications Technology Promotion(IITP) grant fundedby the Korea government(MSIP) (R0124-16-0002, Emo-tional Intelligence Technology to Infer Human Emotion andCarry on Dialogue Accordingly).

REFERENCES1. Mustafa et al. Al’Absi. 1997. Cardiovascular and

neuroendocrine adjustment to public speaking andmental arithmetic stressors. Psychophysiology (1997).

2. Hugo D et al. Critchley. 2000. Neural activity relating togeneration and representation of galvanic skinconductance responses: a functional magneticresonance imaging study. Journal of Neuroscience(2000).

3. Chanwimalueang Theerasak et al. 2016a. Modellingstress in public speaking: Evolution of stress levelsduring conference presentations. In Acoustics, Speechand Signal Processing (ICASSP). IEEE.

4. Gao Hua et al. 2014. Detecting emotional stress fromfacial expressions for driving safety. In ImageProcessing (ICIP). IEEE.

5. Gjoreski Martin et al. 2016b. Continuous stressdetection using a wrist device: in laboratory and reallife. In ACM International Joint Conference onPervasive and Ubiquitous Computing: Adjunct. ACM.

6. Hovsepian Karen et al. 2015a. cStress: towards a goldstandard for continuous stress assessment in themobile environment. In Proceedings of the 2015 ACMinternational joint conference on pervasive andubiquitous computing. ACM.

7. Job Veronika et al. 2010. Ego depletion - Is it all in yourhead? Implicit theories about willpower affectself-regulation. Psychological science (2010).

8. Koo Helen et al. 2016c. Stresssense: skin conductivitymonitoring garment with a mobile app. In JointConference on Pervasive and Ubiquitous Computing:Adjunct. ACM.

9. Lyu Yongqiang et al. 2015b. Measuringphotoplethysmogram-based stress-induced vascularresponse index to assess cognitive load and stress. InACM Conference on Human Factors in ComputingSystems. ACM.

10. Mozos Oscar Martinez et al. 2017. Stress detectionusing wearable physiological and sociometric sensors.International Journal of Neural Systems (2017).

11. Plarre Kurt et al. 2011. Continuous inference ofpsychological stress from sensory measurementscollected in the natural environment. In InformationProcessing in Sensor Networks (IPSN). IEEE.

12. Segerstrom Suzanne C et al. 2004. Psychologicalstress and the human immune system: a meta-analyticstudy of 30 years of inquiry. Psychological bulletin(2004).

13. Wu Min et al. 2015c. Modeling perceived stress viaHRV and accelerometer sensor streams. InEngineering in Medicine and Biology Society (EMBC).IEEE.

14. Richard Ribón et al. Fletcher. 2011. Wearable sensorplatform and mobile application for use in cognitivebehavioral therapy for drug addiction and PTSD. InEngineering in Medicine and Biology Society, EMBC.IEEE.