Quantifying “awkward” in social communication of children with

autism spectrum disorder

Ruth B. Grossman

FACE LabEmerson College

Autism Spectrum Disorder• Current incidence 1/68• Ratio of 4 males to 1 female• Difficulties with

– Social communication and connectedness– Repetitive and/or stereotyped behaviors

• Wide range of profiles– Minimally verbal– Verbal and IQ levels within normal limit– All encompassed under ASD

Why HFA?• Fastest growing group of people with ASD• Intellectual/language capacity to succeed

• So close!

• And yet….– Socially awkward – Underemployed, unemployed, bullied

The social communication loop

?

Hi!

Studies on ExpressionStory Retelling

• Deficits in production of prosody (Shriberg et al. 2001, Grossman et al. 2010)

• Qualitative differences in facial expressions (Capps et al. 1993)

• Goals:– Investigate qualitative differences in faces and

voices in the same task– Subjective and objective measures for voices

(acoustic analysis and perceptual coding), subjective measures for faces

Hi!

Studies on ExpressionWhy Story Retelling?

• Standardized stories with four target emotions• Ability to determine accuracy and quality• No effect of word choice

• Coding for: emotion, intensity, naturalness

• 22 kids with HFA, 18 TD (mean age 11)• Matched on age, IQ, receptive vocab, reading

Hi!

“Safari Bob”

Hi!

Studies on ExpressionResults for kids with HFA

• Acoustic analysis of voices– Softer, higher voices– Larger variability in intensity and pitch

• Coding for faces and voices– More expressive and accurate prosody– More awkward facial expressions– More awkward prosody (trend: p = .06, η2 = .14)

Hi!

Studies on ExpressionConclusion

• Facial and vocal expressions of people with ASD are perceived as awkward– Even when they are accurate

• BUT:– Ratings done by trained research assistants– What do “naïve” observers see? (ecological

validity)

Hi!

Studies in Perception

• First impressions of competence, warmth, intelligence made quickly in TD (Zebrowitz et al. 2002, Willis & Todorov, 2005)

• TD adults (25-30 per group)• “Thin Slices” of 1 second

• Video+audio, video only, audio only• Binary rating “socially awkward”• Autism was never mentioned• No participants with exposure to ASD

Audio-Video Video-only Audio-only Stills0

10

20

30

40

50

60

70

Socially Awkward Ratings after 1 Second

Children with High-Functioning AutismTypically Developing Children

Fre

qu

en

cy

in

Pe

rce

nt

** ****

**

Studies in Perception

Studies on Perception

Conclusion• Impressions of social awkwardness are

formed very quickly• Photos tell the story

• Dysmorphology or body position?

Question: What are the social repercussions of perceived awkwardness?

Studies on Perception

• 24 2-4 second videos• Data Collection:

• Mechanical Turk (MTurk) • Online crowdsourcing website

• Data for 125 participants in 2 weeks• Excluded non-native English

speakers, outside the United States• N=98 (Mean age: 31.4, range19-64)

Studies on Perception• Each video with judgment questions.• How likely is it that the child would:

1. start a conversation with others

2. have a lot of friends

3. get along well with other people

4. spend a lot of time by themselves.• Also rated awkwardness

Very likelyNot likely

Studies on Perception• Each video with judgment questions.• How likely is it that the child would:

1. start a conversation with others

2. have a lot of friends

3. get along well with other people

4. spend a lot of time by themselves.• Also rated awkwardness

Very likelyNot likely

Studies on Perception• Each video with judgment questions.• How likely is it that the child would:

1. start a conversation with others

2. have a lot of friends

3. get along well with other people

4. spend a lot of time by themselves.• Also rated awkwardness

Very likelyNot likely

Studies on Perception

Studies on Perception

• Conclusions:• Children with HFA perceived as

• more awkward • less socially engaged

• TD individuals less likely to engage with children with ASD

Studies on Expression

• So what causes “awkward?”

• Can we use objective measures of facial expressions to quantify “awkward”?

Hi!

Motion Capture

Motion-captureHi!

Motion-captureHi!

Studies on ExpressionMotion Capture

• Mimicry of emotional expressions (Mindreading)

• Same participant group• 24 with HFA • 21 TD controls

Hi!

Studies on Expression• 32 reflective markers on the face

Studies on Expression• 32 reflective markers on the face • VICON cameras record marker movement

Studies on Expression• Marker movement data matched to face

model representing facial features

Studies on Expression• Marker data represents 3D movement of

facial features at 100 frames/second

Motion-captureHi!

• http://www.youtube.com/watch?v=b9kfAH8_W-8

Motion-captureHi!

Studies on Expression

• Markers grouped into 8 regions– Perceptual ratings

of models– Mo-cap of

participants• Data analysis for

– L-R symmetry– Up/down intensity

Hi!

Studies on Expression

• Models show:– No L-R asymmetry– Up/down intensity

differences:• More mouth than

eye movements for fear, sadness, particularly joy

Hi!

Studies on Expression• Compared to TD kids, HFA have:

• Left-right asymmetry• Less richness of mouth movements• Poor replication of modeled differences

in lower vs. upper face (joy)• People with HFA have difficulty

coordinating movement across face regions

• Is this what leads to perceived awkwardness?

Hi!

The social communication loop

?

Hi!

!

Hi!

These data were made possible by:

At the LDCNRhyannon BemisDavid BlackSteven BorawskiChristine ConnollyDanielle DeloshLisa EdelsonAlex FineRobert Joseph Meaghan KennedyJanice Lomibao Geri Owen Al Ozonoff Lara RubinsteinHelen Tager-Flusberg

At USCDaniel BoneTanaya GuhaAngeliki MetallinouShrikanth NarayananAnil RamakrishnaZhaojun Yang

At the Shriver CenterAnne HuntKathryn HastyWilliam McIlvaneTeresa MitchellAnna SchmidErin Steinhart

At Aarhus UniversityRiccardo FusaroliEthan Weed

At Emerson CollegeRebecca BakerSandy BuckGrace ConnollyNatalie FenskeMarlee GalperDarren HedleyGreg HurstDaniel KemplerAllison LaurendeauSarah LovellRhiannon LuysterJulia MertensNicole PitreAaron ShieldAnna Zembo

We thank the children and families who supported this research

This work was funded by NIH grants –NIH-NIDCD R21 DC010867-01 (Grossman, PI) –NIH-NIDCD R01 DC012774-01 (Grossman, PI)