Mental Simulations of Voice and Speech during Reading of QuotationsChristoph Scheepers Bo Yao Pascal Belin

Introduction

Direct versus indirect speechTwo quoting styles that exist in many languagesOften imply differences in perspective (cf. change in corresponding person and

tense features), e.g.– Mary said: “I am hungry”.– Mary said (that) she was hungry.

Important differences in usage– While the main function of indirect speech appears to be to provide a description of what

was said by another person, the use of direct speech typically also involves a demonstration (or depiction) of how it was said, e.g.• ..and she was like: “Oh my god! I’m so hungry!”

Direct speech as vivid demonstration– Tannen (1986; 1989)– Clark & Gerrig (1990)– Wade & Clark (1993)

• Speakers are more likely to use direct rather than indirect speech quotations when instructed to entertain, as opposed to just inform, a listener

Comprehension of Quotations

Implications for reading of quotations? Intuitive experience of an “inner voice”, especially during silent reading of direct

speech quotationsNo direct, objective confirmation of this intuition so far

Embodied cognitionLanguage processing is grounded in mental re-enactment (perceptual

simulation) of perceptual states and actions (Barsalou, 1999; 2008)Given that the use of direct speech is grounded in vivid demonstrations, direct

speech should trigger more enriched perceptual simulations (spontaneous auditory imagery) of, e.g., voice and/or manner of speaking than indirect speech

Present investigations– Study 1 (fMRI+eye-tracking): Voice-selective auditory cortex areas (cf. Belin et al., 2000)

become more active during silent reading of direct rather than indirect speech quotations– Study 2 (eye-tracking): The context preceding a quotation (implying a “fast-” or a “slow-

speaking” quoted speaker) has an influence on how fast one would read a direct-speech quotation, but not how fast one would read an indirect speech quotation

Study 1

MethodEvent-related fMRI combined with eye-tracking (silent reading of text)

Participants16 native English speakers aged 18 - 44 years, 10 female

Stimuli90 short stories containing direct or meaning-equivalent indirect speech

quotations (counterbalanced via Latin square) from fictitious protagonists, e.g.– PhD student Ella was summoned to her supervisor Jim’s office to give a report on her

current progress. Ella asked for an extension but Jim looked concerned. He said: “Hmm, we really need those data in by next month for that conference.” (direct speech)

– PhD student Ella was summoned to her supervisor Jim’s office to give a report on her current progress. Ella asked for an extension but Jim looked concerned. He said that they really needed those data in by next month for that conference. (indirect speech)

20 vocal (e.g. spoken vowels; laughing; coughing..) and 20 non-vocal (e.g. telephone ringing; dog barking..) sound-clips for a functional voice localizer session after the main reading session

Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

Study 1: Tasks

(1) Main reading session

Read stories silently for comprehension and answer yes/no-questions

(2) Voice localizer session

Listen to 20 vocal and 20 non-vocal sound-clips (8 sec each)

Eyes closed!Same as in Belin et al. (2000, Nature)

Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

Study 1: Results

Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

Auditory voice-localizer regions of interest (ROIs) in green

Within those regions, significantly higher activation (BOLD response) for direct speech than for indirect speech quotations– Measured from 1st fixation into the critical quotation

passage to first progressive saccade out of the critical quotation passage (eye-tracking)

Strongly right-lateralized pattern– Two main clusters of enhanced activity for direct (as

opposed to indirect) speech quotations were located in voice-selective areas along posterior and middle parts of the right STS

– Interestingly, similar right-lateralized activation patterns have previously been found in association with emotional prosody (e.g. Wiethoff et al., 2008; Wildgruber et al., 2006)

No differences in reading rate between direct vs. indirect speech quotations– Mean reading time per word: 204 ms vs. 203 ms– Suggests no difference in processing difficulty

between the two quotation styles

Study 1: Discussion

Evidence for enhanced auditory mental simulation/imagery of voice during silent reading of direct speech quotations (compared to meaning-equivalent indirect speech quotations)

Auditory cortex regions that are particularly ‘interested’ in human voices (cf. Belin et al., 2000) show more top-down activation during silent reading of direct speech quotations

Can auditory voice simulation effects be demonstrated behaviourally as well?...

Study 2

Stimuli24 short stories containing direct or meaning-equivalent indirect speech

quotations The context preceding those quotations implied either a fast or a slow-speaking

quoted protagonist:

– Fast context: Teenage pianist Bobby was to take tart in a local talent competition. He was extremely nervous before his performance. His mother encouraged him… but he was all shaking and said: “No! I can’t do it! This is the end of the journey because I won’t make it this time.” His mother … (direct speech)but he was all shaking and said that he couldn’t do it and that this was the end of the journey because he wouldn’t make it this time. His mother … (indirect speech)

– Slow context: At the Royal Infirmary, an old man was dying, and too weak to sit up. He wanted to say something, so his daughter placed a cushion under his head. Slowly, he looked around… and said: “I’m grateful you are all here. This is the end of the journey because I won’t make it this time.” Then he closed his eyes … (direct speech)and said that he was grateful for their coming and that this was the end of the journey because he wouldn’t make it this time. Then he closed his eyes … (indirect speech)

Yao & Scheepers (2011) Cognition

Study 2

MethodOral reading (20 participants):

– Read the stories aloud in a random order– Digital audio recordings throughout the session– Reading rate for critical quotation passages analysed in numbers of syllables

per second

Silent reading (48 participants):– Read the stories (plus fillers) silently in a random order– Answer yes/no comprehension question after each story– Eye-tracking (EyeLink 1000) throughout the session– Reading rate for the critical quotation passages analysed in terms of ~ first-

pass reading time (time from fixating the passage for the first time until another region is fixated)

Yao & Scheepers (2011) Cognition

Study 2: Hypotheses

Prior research (Alexander & Nygaard, 2008) has demonstrated that oral and silent reading rates are modulated by auditory imagery– People read faster when instructed to imagine familiar ‘fast speakers’ as

authors of text for reading

Here, we do not use instructions, but rather look at spontaneous reading rate adjustments as a function of quoting style (direct / indirect speech) and context (implying fast or slow quoted speaker)

Inspired by Alexander & Nygaard (2008), we expect reading rates to be modulated by context (fast or slow quoted speaker)

Informed by Study 1, we expect contextual modulation of reading rate to be stronger for direct than for indirect speech quotations

Yao & Scheepers (2011) Cognition

Study 2: Results

Reading aloud task: Oral reading rates (in syllables/second) for the critical quotation passages

Clear Context × Quoting Style interactionContext-fast > Context-slow in direct speechNo such context effect in indirect speech

With direct speech quotations, participants spontaneously “act out” the contextually implied speech rate of the quoted speaker

Silent reading task (eye-tracking): First-pass reading times (in ms) for the critical quotation passages

Clear Context × Quoting Style interactionContext-fast < Context-slow in direct speechNo such context effect in indirect speech

With direct speech quotations, silent readers adjust their reading rates to the contextually implied speech rate of the quoted speaker

direct indirectQuote

5.6

5.7

5.8

5.9

6

6.1

6.2

Oral reading (syllables/sec)

Context fastContext slow

direct indirectQuote

1600

1700

1800

1900

2000

2100

Silent reading (FPRT in ms)

Context fastContext slow

Yao & Scheepers (2011) Cognition

Conclusions

Converging evidence from fMRI and eye-tracking, suggesting that readers are more likely to mentally simulate (or spontaneously imagine) aspects of the quoted speaker’s voice (and manner of speaking) during silent reading of direct speech quotations

Consistent with “direct speech as vivid demonstration” (Clark & Gerrig, 1990)

Perceptual grounding (cf. Barsalou, 1999; 2008) of direct speech:– Since direct speech typically occurs in the context of vivid

demonstrations/re-enactments of actual speech acts, a direct quotation style is more likely to be taken as a cue to engage in enriched perceptual simulations of voice than an indirect quotation style (focus on content)• … and maybe not only of voice! (to be continued…)