Analysis of Spontaneous Emotional Expressivity in Adults with Traumatic Brain InjurySamuel JamesEdgemont High School

AcknowledgementsI would like to thank my teacher, Ms. Maria DeCandia, for all her help during the past three years while I was in the Science Scholars program. I would also like to thank my mentor, Dr. Preeti Raghavan, for providing me with the opportunity to work at the RUSK Rehabilitation Lab. I am also grateful for all her time and the guidance she has given me during my research. I would also like to thank Dr. Aluru Viswanath for his guidance during the course of the study, as well as, the other researchers at the RUSK Lab. Finally, and most importantly, I would like to thank my mother, Lovetta James, and my father, Joji James, for their extensive support.

AbstractA growing number of research studies have shown that people with Traumatic Brain Injury (TBI) have difficulties in identifying emotions. However, little research has been done to determine if this impairment has an impact on the emotive response of subjects with TBI. Do individuals with TBI have difficulty expressing their emotions? How different is their emotional expression compared to individuals without TBI? With advancements in Automatic Facial Action Coding (AFAC), the facial expressions of subjects with TBI can be analyzed quickly and thoroughly. In this exploratory study we analyzed the facial expressions of subjects with TBI as they watched disgust and amusement-eliciting film clips. Subjects were videotaped as they watched the clips. The subjects also completed a Post-Film Questionnaire (PFQ), which self-identified the emotions they experienced while watching the film clips. The software, Attention Tools FACET Module (InMotion, MA), was used to analyze the facial expressions of the subjects. We then compared the emotions that the subjects claimed to have experienced (PFQ) with the emotions expressed on their faces (FACET). The results suggest that subjects with TBI are impaired in expressing their emotions. Lack of emotional expressivity may contribute to dysfunctional inter-personal relationships after TBI.

Table of ContentsList of Tables..1Introduction.2Methods...3Results.9Discussion..13References..15

James 1

List of Tables

TABLE 1 Emotion-eliciting movie clips shown to participants 6

List of FiguresFIGURE 1FACET evidence for TBI Subject during Disgust and Amusement evoking video clips 11FIGURE 2PFQ evidence for TBI Subject during Disgust and Amusement evoking video clips11FIGURE 3FACET evidence for TBI Subject during Disgust and Amusement evoking video clips11FIGURE 4PFQ evidence for TBI Subject during Disgust and Amusement evoking video clips11FIGURE 5FACET evidence for TBI Subject during Disgust and Amusement evoking video clips11FIGURE 6PFQ evidence for TBI Subject during Disgust and Amusement evoking video clips11FIGURE 7FACET evidence for Control Subject during Disgust and Amusement evoking video clips12FIGURE 8PFQ evidence for TBI Subject during Disgust and Amusement evoking video clips12FIGURE 9FACET evidence for Control Subject during Disgust and Amusement evoking video clips12FIGURE 10PFQ evidence for Control Subject during Disgust and Amusement evoking video clips12FIGURE 11FACET evidence for Control Subject during Disgust and Amusement evoking video clips12FIGURE 12PFQ evidence for Control Subject during Disgust and Amusement evoking video clips12

IntroductionTraumatic Brain Injury (TBI) is the result of sudden forceful movement of the brain against the inner walls of the rigid skull. TBI can be caused by closed or open head injury from motor vehicle accidents, falls, and sports-injuries. While sports-related head injuries are commonly called concussions (Levine, 2010 & Anderson 2006), they are in effect mild traumatic brain injuries that can cause cumulative brain dysfunction.

It is known that TBI can affect a persons social and emotional behavior (Brown, 2011). More recent research has shown that TBI can cause deficits in facial emotion perception. When presented with facial models that exhibited various emotions, TBI subjects had difficulty identifying the particular emotion. They especially had trouble recognizing negative emotions such as disgust, fear and sadness (Cauldon,2000;Croker et al., 2005). However, little is known about whether a TBI also alters an individuals own emotive response. This is important because an inability to express ones feelings effectively may lead to unresolved frustration, anger and depression, which can have dire behavioral consequences, including suicide (Mckay et.al, 2009). The purpose of this study was to determine whether individuals with TBI suffer from deficits in spontaneous emotional expression when compared with healthy individuals.

MethodsParticipants

The study included six subjects; three post-TBI subjects, who complained of emotional dysfunction six months following their brain injury, and three healthy age-matched control subjects. They were equally divided in gender. Exclusion criteria for subjects included comorbid psychiatric disease (e.g. schizophrenia), active substance abuse, history of craniectomy, active infection, seizure activity beyond one week post-TBI, confounding medical complications (e.g. hypertension, diabetes, epilepsy or demyelinating disease, implanted neuromodulatory or electronic device) and pregnancy. Also, only those subjects with the capacity to understand the terms of the study and provide informed consent were enrolled.

Procedure

Subjects sat in a well lit room facing a computer monitor. They were informed that they were going to watch a series of eight film clips that would be 2-5 minutes long. However, they were not made aware of the content or the emotion evoking qualities of the clips. Before each clip, the subjects watched a set of instructions for 10 seconds informing them to relax and watch a + sign on the computer screen for the next minute. This technique was used to return subjects back to baseline prior to each new film clip. The clips were shown in a specific order to balance out the negative and positive emotions. This prevented the subjects from being overloaded with any single emotion.

There were two sets of film clips; each set included clips that elicited the following emotions - neutral, amusement, sadness, sexual amusement, fear and disgust. These clips were selected from a database of film clips validated to elicit specific emotions (Schaefer, A., Nils, F. et al. ,2010). Unlike the film clips used in previous studies, which only included facial expressions, the clips we selected have a background setting which provides subjects with contextual cues. This is relevant since a TBI subject may have difficulty identifying facial expressions, but they may still be able to identify emotions from contextual cues (Calder et al., 2000). Furthermore, the fil clips not only convey the spontaneity of emotions, but may also show head and facial movement which are also markers of emotions (Mcdonald, 2005). Another benefit of videos over static images is the addition of audio. Both individuals with TBI and without TBI have been shown to identify emotions better when there is a conjunction of both video and audio (Calder et al., 2000).

The subjects watched one film clip each that elicited neutral, amusement, sadness, sexual amusement and fear. They were told, We will now be showing you a short film clip. It is important to us that you watch the film clip carefully. Subjects then watched three clips depicting disgust. They were instructed to watch the first disgust film clip just as they had the previous clips. For the second disgust clip, subjects were told to suppress their emotional response. Watch the film clip carefully. If you have any feelings as you watch the film, please try your best not to let those feeling show. For the third disgust clip, subjects were told to amplify their emotional response. Watch the film clip carefully. If you have any feelings as you watch the film clip, try your best to let those feelings show. After watching each film clip, the subjects were asked to complete a questionnaire to self-rate the intensity of their feelings on scale of 0 (none) to 10 (extreme).

Table 1. Emotion-eliciting movie clips shown to participantsVideoSetTarget emotionMovie title & descriptionTime (min)

Set 1NeutralSticks accumulating on a black background (no audio).4.6

AmusementSomething about Mary man fights with a small dog.4.33

SadnessCity of Angels A woman dying in a mans arms.4.35

Sexual amusementWhen Harry Met Sally A woman loudly simulates sexual excitement in a crowded diner.3.75

FearThe Shining A boy plays alone in an empty hallway2.5

DisgustTrainspotting A man defecates then dives into a filthy toilet.2.38

Amputation noncommercial recording of arm amputation2.3

Bat Taste A group of young men eat vomit.2.96

Set 2NeutralSticks accumulating on a black background (no audio).4.6

AmusementBenny and Joon A man clowns around in a diner.3.33

SadnessThe Champ A young boy cries as his father lays dead.3.83

Sexual amusementA Fish Called Wanda A woman and man are caught while in a comedic tryst4.08

FearSilence of the Lambs A chase scene followed by a dead corpse.4.6

DisgustVampires Kiss A man eats a cockroach1.65

Black Swan A woman peels back a hangnail.1.87

Pink Flamingoes A woman eats dog feces.2.5

Note. Adapted from Uy, et. al. (2013)

IMOTION FACET Module

Motivated by advancements in facial action coding technology, this study is one of the first to collect facial video data on TBI subjects during spontaneous emotional expression. Analysis of the subjects facial expressions from the video was performed using The Facial Action Coding System (FACS) (G. Littlewort et al., 2011). The Facial Action Coding System is a database created by Paul Ekman in the 1970s, and was expanded in 2001. The original database was a collection of Ekmans own emotional expressions formed by varying facial muscle activity. Eckman used this data for his own study of emotion-related muscle response (Eckman & Friesen, 1976). To score a face according to the FACS, the rater examines different Action Units (AUs), as well as combinations of different AUs. Each AU corresponds to a different muscle movement. For example, if someone were to raise their inner eyebrow they would be exhibiting AU1. AUs can be broken down into varying emotions. For Example AUX1, AUX2, AUX4, may be prominent when someone is expressing fear (Eckman, Friesen and Hager, 2002).

IMOTIONs FACET Module is an automated facial expression recognition software that can analyze video in real time. Its predecessor, the Computer Expression Recognition Toolbox (CERT) has already been used in research. The FACET module is able to detect faces from the live feed and then code the faces using Paul Eckmans Facial Action Coding System (FACS). It can identify and measure the intensity of 19 Action Units (AU) {AUs 1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 18, 20, 23, 24, 25, 26, and 28}. The FACET Module differentiates nine different emotions, including the six basic emotions: Joy, Sadness, Fear, Anger, Disgust, and Surprise and three other advanced emotions Contempt, Frustration, and Confusion (Eckman & Friesen, 1976).

Face and Feature DetectionFACET employs different boosting algorithms to identify both the face and facial features. The facial features corresponding to the different AUs. The booting algorithms output the likelihood that a feature is present at a pixel in 2D Cartesian space.

Facial WarpingIn order to maximize facial detection performance the faces are cropped and warped to fit in a 96x96 frame. This process helps to normalize the face by adjusting for any pitch, yaw and size difference among the faces. The cropped face then undergoes multiple Gabor Filters for edge detection. The output is a single feature vector.

Action Unit RecognitionThe feature vector is input into linear Support Vector Machines (SVP) so as to estimate the intensity of each AU. The SVP was trained using multiple databases such as the Cohn Kascade Database which includes coded videos and static images. The SVP output is the distance between the cropped face and the separating hyperplane. This value correlates with the intensities of each emotion. In turn, the intensity score allows for a reliable and consistent way to measure the intensity of AUs, a task which is difficult to accomplish manually.

Analysis

Although the subjects watched the entire clip, specific attention was given to the Interval of Interest (IOI), which is 30 seconds of the clip that contains the most emotion eliciting content.

During the IOI, the Flatness and Inappropriateness of the subjects face was also measured. Flatness is a measure of how stoic a persons face is in response to a situation. For example, if someone were to see a close relative or friend pass away, one can expect them to show signs of remorse such as crying. Their face would be flat if they showed no emotion at all. Inappropriateness is a measure of how appropriate the reaction is in a situation. Laughing or smiling in a sad situation would be inappropriate.

Flatness was calculated by first finding the number of frames where the AU evidence for all the AUs was less than 0.2. This value was then divided by the total number of frames to produce a decimal value. Inappropriateness was calculated by first finding the number of frames where the targeted emotion was not the dominantly expressed emotion. This number was then divided by the total number of frames to get a decimal value. The values ranged from 0.001 to 1.

Results

Thus far, the FACET and PFQ data for the Amusement and Disgust (neutral phase) film clips have been analyzed for the TBI and Control subjects.

TBIFor this group, Amusement was the lowest evident emotion on facial expression analysis with FACET (Figures 2, 4, 6). However, it was the highest-rated emotion experienced as per the post-film questionnaires (PFQ, Figures 1, 3, 5). Anger was not at all evident in and disgust was minimal in both PFQ and FACET data. Confusion was evident in two of the three subjects PFQ scores and in all their FACET scores (Figure 1-6). Overall, the PFQ scores did not correspond with their FACET scores (Figures 1-6).

ControlThe target emotion on the film clip showed the most prominent facial expression in the Control group (Figures 8, 10, 12). The target emotion was also rated most highly on the PFQ (Figures 7, 9, 11). During the Disgust evoking clips, the Control group however also showed high confusion on both the FACET and PFQ (Figures 7-12).

Disgust and AmusementDuring the sequence of Disgust clips, the TBI group evidenced less disgust, and more anger and confusion than the Control group. Furthermore, there was more evidence of sadness and surprise for TBI subjects. During the Amusement clips, the TBI group showed more evidence of anger and confusion, and less evidence of surprise than the Control group. There was no difference in the expression of sadness for the TBI group, when watching the Disgust and the Amusement clip.

Figure 1: FACET evidence during Disgust and Amusement Figure 2: PFQ evidence during Disgust and Amusement video clips for TBI Subject 1. video clips for TBI Subject 1.

Figure 3: FACET evidence during Disgust and Amusement Figure 4: PFQ evidence during Disgust and Amusement video clips for TBI Subject 2. video clips for TBI Subject 2.

Figure 5: FACET evidence during Disgust and Amusement Figure 6 PFQ evidence during Disgust and Amusement video clips for TBI Subject 3. video clips for TBI Subject 3.

Figure 7: FACET evidence during Disgust and Amusement Figure 8: PFQ evidence during Disgust and Amusement video clips for Control Subject 1. video clips for Control Subject 1.

Figure 9: FACET evidence during Disgust and Amusement Figure 10: PFQ evidence during Disgust and Amusement video video clips for Control Subject 2. clips for Control Subject 2.

Figure 11: FACET evidence during Disgust and Amusement Figure 12: PFQ evidence during Disgust and Amusement video video clips for Control Subject 3. clips for Control Subject 3.

Discussion

In this study, we analyzed the facial expressions of subjects with TBI and age-matched Control subjects as they watched validated film clips that were intended to evoke target emotions. The subjects faces were videotaped as they watched the film clips. They were also asked to complete questionnaires after watching each clip where they rated their emotional experiences.

Social DysfunctionIt is known that TBI affects social relationships. Relatives have noted that subjects showed changes in self-control and social perception, creating distress in their relationships (Henry, 2006). They had difficulty understanding basic emotions, subtle inferences and intentions of those they interacted with (McDonald & Flanagan, 2004). Subjects also have an increased divorce rate following their injury (Brooks, 1986). In this study, TBI subjects had difficulty expressing an appropriate emotional response. They were confused when watching video clips that should have evoked either disgust or amusement. Their measure of disgust, when watching a disgust-evoking film clip, in expression and feeling was less than that of the Control groups. This was also the case for the amusement-evoking film clip. It was reflected in their responses, collected by the Post Film Questionnaires (PFQ), as well as identified from FACET. Studies have noted that TBI patients are challenged in identifying emotions in others (Caulder, 2000; Croker et al., 2005). This study identifies the added inability of TBI subjects to express appropriate emotional responses to external stimuli, in this case, emotion evoking film clips. The inability to understand emotions in others and within themselves can be a direct precursor to developing poor social relationships. It can develop feelings of frustration and anger. This can lead to sudden aggression such as verbal threats and even more serious physical harm (Gertler,2006). In conclusion, even though this was an exploratory study with a small number of subjects, the findings are relevant to the continued study of TBI patients in this area. Data collection for this study included a wider range of target emotion for TBI and Control subjects. This data continues to be examined. The FACET software was a useful tool in analyzing emotive response of TBI subjects.

References Brooks, N., L. Campsie, C. Symington, A. Beattie, and W. Mckinlay. "The Five Year Outcome of Severe Blunt Head Injury: A Relative's View."Journal of Neurology, Neurosurgery & Psychiatry 49.7 (1986): 764-70. Web. Brown, C. (2011). Emotion Recognition and Traumatic Brain Injury (Masters Thesis). Arizona State University, Arizona. Calder, A., Keane, J., Manes, F., Antoun, N., & Young, A. (2000). Impaired recognition and experience of disgust following brain injury. Nature Neuroscience, 3(11), 1077-1078. Croker, V., & Mcdonald, S. (2005). Recognition of emotion from facial expression following traumatic brain injury. Brain Injury, 19(10), 787-799. Ekman, P., & Friesen, W. (1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1(1), 56-75. Ekman, P., & Friesen, W. & Hager, J. (2002). Facial action coding system. Salt Lake City, Utah: A Human Face.Henry, Julie D., Louise H. Phillips, John R. Crawford, Magdalena Ietswaart, and Fiona Summers. "Theory of Mind following Traumatic Brain Injury: The Role of Emotion Recognition and Executive Dysfunction."Neuropsychologia 44.10 (2006): 1623-628. Web. Gertler, Paul. "Angry Young Men: Aggressive Behaviour after Traumatic Brain Injury." The Australian Psychological Society (2006): n. pag. Web. G. Littlewort , J. Whitehill , T. Wu , I. Fasel , M. Frank , J. Movellan and M. Bartlett The computer expression recognition toolbox (cert). Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp.298 -305 2011 J. Hamm, A. Pinkham, R. Gur, R. Verma, and C. Kohler, Dimensional Information-Theoretic Measurement of Facial Emotion Expressions in Schizophrenia. Schizophrenia Research and Treatment, vol. 2014 Mcdonald, S. (2005). Are You Crying or Laughing? Emotion Recognition Deficits After Severe Traumatic Brain Injury. Brain Impairment, 6(1), 56-67. McDonald, S., and S. Flanagan. "Social Perception Deficits after Traumatic Brain Injury: The Interaction between Emotion Recognition, Mentalising Ability and Social Communication."Neuropsychology 18 (2004): 572-79. Web. Uy, Catherine C., Iain A. Jeffrey, Matthew Wilson, Viswanath Aluru, Anita Madan, Ying Lu, and Preeti Raghavan. "Autonomic Mechanisms of Emotional Reactivity and Regulation." Psychology 04.08 (2013): 669-75. Web. James 14