doğu erdener
DESCRIPTION
Auditory-visual speech perception in young children: language and age-specific factors. Doğu Erdener. @. 6 December 2007. Development of Speech Perception. First 6 months phonetically-based, language-general Beyond 1 st year phonologically-based, language-specific - PowerPoint PPT PresentationTRANSCRIPT
Doğu ErdenerDoğu Erdener
Auditory-visual speech perception in Auditory-visual speech perception in young children: young children: language and age-language and age-
specific factorsspecific factors
@@
6 December 2007
Development of Speech Perception Development of Speech Perception
First 6 months phonetically-based, language-general
Beyond 1st year phonologically-based, language-specific Consonants 7-11 months (Best, 1995) Vowels 4-6 months (Kuhl et al., 1992) Tones 6-9 months (Burnham & Mattock, 2006)
A working heuristic (Burnham et al., 2002) Acoustic / Phonetic Stage (0-6 months) Phonological Stage (6-12 months) Semantic Stage (12-24 months) Orthographic Stage (6-8 years)
Amodal (e.g.,Schwartz, et al., 1997) Phonetic (e.g.,Burnham & Dodd, 2004) Phonological / Postcategorical (e.g.,Massaro, 1998) none/some/all of the above?
Pre-categorical
Auditory-Visual Speech Perception (AVSP): Integration Auditory-Visual Speech Perception (AVSP): Integration
in early infancy (till ~6 months) is phonetically based beyond first year of life phonological
Recall that Recall that auditory-only auditory-only speech perception …. speech perception ….
AVSP integration research: AVSP integration research: We need differential and ontogentic data!We need differential and ontogentic data!
AVSP: Cross-Language (Differential) StudiesAVSP: Cross-Language (Differential) Studies
Cross-language phonemic differences e.g., Werker et al. (1992) Visual influence from /th/ in the McGurk
stimulus [A]-/ba/ + [V]-/tha/ :• anglophones > francophones• more experienced francophones > less experienced francophones
Amount of visual speech influence across langauges: American > Japanese (Sekiyama & Tohkura, 1993) Japanese > Mandarin (Sekiyama, 1997)
Sekiyama studies Possible reasons: Cross-cultural studies: Less eye/face contact in some cultures +
Japanese Ss tend to look at the eye area whereas American Ss on the mouth area (Yuki et al., 2007).
Linguistic 1: Tonal (e.g. Cantonese) and pitch-accent (e.g., Japanese) information are not visually salient.
Linguistic 2: Fewer visually distinct consonant clusters and vowels in Japanese than in English.
Development of Auditory-Visual Speech PerceptionDevelopment of Auditory-Visual Speech Perception
Infancy Infants match auditory and visual speech information
(Burnham & Dodd, 2004, Rosenblum et al., 1997)
Childhood McGurk & MacDonald (1976) 3 to 8 years + adults Massaro et al. (1986) 4-6 years + adults
Adulthood AV integration adults > children (Sekiyama &
Burnham, in press; Massaro et al.,1986) Late adulthood older Ss > younger Ss on vısual
speech information when auditory sensitivity is diminished (Cienkowski & Carney, 2002).
Focus: Developmental Data from Infants and ChildrenFocus: Developmental Data from Infants and Children
Why is there an age-related increase?
Infancy and childhood are marked with several novel language-relevant and language-specific challenges, such as phonetic phonological semantic vocabulary reading
Japanese and Australian English SpeakersThree child (6-, 8-, and 11-year-olds) groups
and a control group of adults.Method
McGurk stimuli DV: Visual Speech Index (VSI) Score based
on the proportion of auditory-based responses Identification Task: /ba/, /da/, or /ga/?
Sekiyama & Burnham (in press): Sekiyama & Burnham (in press): an ontogenetic + differential study an ontogenetic + differential study
AV+ minus A
Augmentation
A minus AV-
Interference
Visual Speech
Index (VSI)
[AV+] minus [AV-]
Japanese
0
0.25
0.5
0.75
1
-4 4 12 Clear
0
0.25
0.5
0.75
1
1 2 3 4
0
0.25
0.5
0.75
1
-4 4 12 Clear
0
0.25
0.5
0.75
1
-4 4 12 Clear
SN Ratio
Australian English
0
0.25
0.5
0.75
1
-4 4 12 Clear
Prop. Correct Aud Rs
0
0.25
0.5
0.75
1
-4 4 12 Clear
Prop. Correct Aud Rs
0
0.25
0.5
0.75
1
-4 4 12 Clear
Prop. Correct Aud Rs
0
0.25
0.5
0.75
1
-4 4 12 Clear
SN Ratio
Prop. Correct Aud Rs
6-yos
8-yos
11-yos
Adults
Sekiyama and Burnham (in press) found: Developmental increase in visual speech influence with age…
• In English speakers but• Not in Japanese speakers
So…..
What factors modulate the development of auditory-visual What factors modulate the development of auditory-visual speech perception speech perception (measured by VSI), (measured by VSI),
especially especially between 6 and 8between 6 and 8 years of age? years of age?
Experiment 1: Factors InvestigatedExperiment 1: Factors Investigated
Language Specific Speech Perception (LSSP) Effect of phonological experience in native language: Native
Speech Perception minus Non-native Speech Perception (N-NN) (Burnham, 2003).
Reading Reading ability is positively related to LSSP in children
(Burnham, 2003). Evidence for reading-lipreading link:
(de Gelder & Vroomen, 1998; Cavé et al., 2007)
Articulation ability is positively related to visual speech influence in children (Desjardins et al., 1997), and in adults with cerebral palsy (Siva et al., 1995)
Experiment 1Experiment 1
AVSPLanguage-specific Speech Perception
(LSSP) Reading
Articulation
Regression
Correlation
Experiment 1: MethodExperiment 1: Method
Language Specific Speech Perception (LSSP) Test: (Native minus Non-native speech perception N-NN)18 N [voiced vs. voiceless] + 18 NN [voiced vs. prevoiced]
Reading Test WRAT-3 reading subtest: 15 letters + 42 words = 57 items
Articulation Test Queensland Articulation Test (QAT) Naming task: 64 pictures Aus English consonants: initial, medial, & final positions
Auditory-Visual Speech Perception (AVSP) TestAuditory-Visual Speech Perception (AVSP) Test McGurk paradigm: A & V combinations of /ba/, /da/, and /ga/ 48 McGurk stimuli: (24 AV, 12 AO, and 12 VO),
3 consonants x 4 talkers (2 Eng & 2 Jap) x 2 congruence types x 2 background noise (Clear & noisy [+4 dB]) versions
Visual Speech Index (VSI) AV (+) minus AV (-)
Experiment 1: MethodExperiment 1: Method
Participantsn=96; Australian English 5-, 6-, 7-, and 8-year-oldsn=48; Australian English adults
N=144
Procedure Individual testing (~60 minutes) Task orders counterbalanced Clear & Noise (+4dB SN Ratio) Each stimulus presented twice
Exp. 1 Results: AVSP TestExp. 1 Results: AVSP Test
0
0.25
0.5
0.75
1
5 years 6 years 7 years 8 years Adults
Age
Visual Speech Index Score (/1)
Native Speaker Clear VSIForeign Speaker Clear VSINative Speaker Noise VSIForeign Speaker Noise VSI
VSI scores: Linear increase with age (p <.01) + Quadratic between 5 and 6 years (p <.01).
VO scores: Linear increase with age (p <.01).
AO scores: Linear increase with age (p <.01).
0
0.25
0.5
0.75
1
5 years 6 years 7 years 8 years Adults
Age
Correct VO Score (/1) Native VO Scores
Non-native VO Scores
0
0.25
0.5
0.75
1
5-year-olds 6-year-olds 7-year-olds 8-year-olds Adults
Age
Proportion of Correct AO Response (/1)
Native Speaker ClearForeign Speaker ClearNative Speaker NoiseForeign Speaker Noise
Exp. 1 Results: LSSP, reading and articulation Exp. 1 Results: LSSP, reading and articulation
LSSP scores: No age-based differences (p >.01) but native > non-native (p< .01)
Reading scores: Increase with age (p< .01)
Articulation scores: Increase with age (p <.01) + adults > children (p <.01).
0
0.25
0.5
0.75
1
5 years 6 years 7 years 8 years Adults Mean Discrimination Index Score (/1)
Native DI Non-Native DI
Age
0
0.25
0.5
0.75
1
5 years 6 years 7 years 8 years Adults
Age
Reading Test Scores (/1) Reading Test Score (WRAT-3)
0
0.25
0.5
0.75
1
5 years 6 years 7 years 8 years Adults
Age
Articulation Test Scores (/1) Articulation Test Score
Children
Age + AO + VO + LSSP + Articulation + Reading VSI
Adults
Age + AO + VO + LSSP +Articulation + Reading VSI
Exp. 1 Results: Regression Analyses Exp. 1 Results: Regression Analyses (DV = VSI Scores)(DV = VSI Scores)
Exp. 1 Results: Correlation Coefficients Exp. 1 Results: Correlation Coefficients
VSI Age AO VO LSSP Artic. Read.
VSI -
Age .14 -
AO .07 .43** -
VO .33** .32** .26** -
LSSP .21* -.15 -.01 .05 -
Articulation .07 .69** .30** .25** -.15 -
Reading .13 .86** .46** .35** -.06 .68** -
* Sig. at á=.05, **Sig. at á= .01
Experiment 1: DiscussionExperiment 1: Discussion
AVSP and lipreading (VO) increase with age This increase is sharp at around 6 years of age, around the
onset of reading instruction. No age-based LSSP differences
native and non-native difference is largest at 6 years Reading instruction a new challenge new sources of
speech information used? Regression Analyses
Children: lipreading + LSSP VSI Adults : AO VSI
Correlation Analyses Articulation is correlated with AO, lipreading (VO), and reading.
Exp. 1 Discussion Exp. 1 Discussion Experiment 2 Experiment 2
Experiment 1: LSSP & Lipreading AVSP
Schooling and/or new language-specific challenges seem to increase visual speech influence.
Towards Experiment 2 (3- & 4-year-olds): Status of LSSP AVSP link in pre-orthographic
stages?? Are AVSP and basic Cognitive abilities related? Are AVSP and Vocabulary Knowledge related?
Experiment 2: Method Experiment 2: Method
Participants: N=48; 3-year-olds & 4-year-olds AVSP – material, stimuli and procedure
AVSP: AX discrimination task Three conditions AV(36), AO(12), VO(12) AX-VSI (/1): based on different trials
LSSP – material, stimuli and procedure 18 Native [pa-pha] & 18 Nonnative [ba-pa] trials. AX-based category change paradigm.
Vocabulary Test – material, stimuli, and procedure PPVT (Dunn & Dunn,1997) age-based word sets of 12 items on average 5-6 sets.
Cognitive Flexibility – material, stimuli and procedure FIST (Jacques & Zelazo, 2001) 15 items (shapes, sizes & colours)
Exp. 2 Results: AVSP TestExp. 2 Results: AVSP Test
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
3-year-olds 4-year-olds
VSI-AX Score (/1)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
3-year-olds 4-year-olds
Correct Response Rate (/1)
Auditory-Only
Visual-Only
VSI-AX scores: 4-year-olds > 3-year-olds (p <.01)
VO scores: 4-year-olds ≈ 3-year-olds (p =.07)
AO scores: 4-year-olds > 3-year-olds (p <.05)
Exp. 2 Results: LSSP, vocabulary and cognitive flexibilityExp. 2 Results: LSSP, vocabulary and cognitive flexibility
LSSP scores:
4-year-olds > 3-year-olds (p <.01)
No stimulus language effect or contrast/age interaction (p >.05)
Vocabulary scores: 4-year-olds > 3-year-olds (p <.01)
FIST scores: 4-year-olds > 3-year-olds (p <.01)
0
0.05
0.1
0.15
0.2
0.25
3-year-olds 4-year-olds
Discrimination Index Score (/1)
Native DINon-Native DI
0
10
20
30
40
50
60
70
80
3-year-olds 4-year-olds
Receptive Vocabulary Score
0
2
4
6
8
10
12
14
16
3-year-olds 4-year-oldsMean Executive Function Test Score (/15)
Rule Abstraction
Cognitive Flexibility
Exp. 2 Results: Regression Analyses Exp. 2 Results: Regression Analyses
VSI scores as dependent variable
Age + AO + VO + LSSP + Vocabulary + FIST VSI-AX
Exp. 2 Results: Correlation Coefficients Exp. 2 Results: Correlation Coefficients
VSI-AX Age AO VO LSSP Vocab. FIST
VSI-AX -
Age .45** -
AO .54** .42** -
VO .21 .21 .37** -
LSSP .30* .10 .21 .10 -
Vocab. .45** .66** .56** .27* .41** -
FIST .51** .36** .35** .36** .42** .54** -
* Sig. at á=.05, **Sig. at á= .01
Experiment 2: DiscussionExperiment 2: Discussion
AVSP, LSSP, vocabulary and FIST improve with age AVSP improves between 3 and 4 years. Lipreading p=.07 – same age-based trend, but small age
difference? Regression Analyses
AO & FIST VSI-AX similar to adults in Exp. 1 (AO VSI)
Why does AO predict VSI-AX? AVSP product of AO speech perception Three and 4 years: object-word pairing is a challenge, but not as
much as reading – so no need for extra info. AVSP is determined by inherent abilities?
e.g., AO speech perception and cognitive function.
Regression Analyses: Regression Analyses: LSSP, LSSP, aage, language tests regressed onto AVSPge, language tests regressed onto AVSP
5,6,7, 8 years LSSP + VO + AO + Articulation + Reading AVSP
AVSP is related to LSSP and lipreading
AdultsLSSP + VO + AO +Articulation + Reading AVSP
AVSP is related only to auditory speech perception
3, 4 years
LSSP + VO + AO+ Cognitive + Vocab. AVSP
AVSP is related only to auditory speech perception
General DiscussionGeneral Discussion
Visual speech influence develops with age AV: e.g., Massaro et al. (1986) VO: e.g., Desjardins et al. (1997)
(though p=.07 in Exp.2)
AVSP follows a phonetic phonological trend AVSP is amodal and phonetic
(Burnham & Dodd, 2004; Rosenblum et al., 1997) AVSP is phonological (Massaro, 1998) But more importantly, AVSP seems to occur at all
levels of language processing.
AVSP- Articulation Link Vocabulary and articulation challenges in early childhood Tests of AVSP, LSSP, and articulation for children (12-24
months), with and without speech disorder (Dodd et al.,in press). AVSP-Reading Link
Burnham (2003): reading LSSP LSSP AVSP So, integrative ability phonemic/phonological processing This may be the case for English but not for other languages?
Around school age, language-relevant challenges emerge Reading phoneme-grapheme mapping LSSP intensifies (orthographic stage: Burnham et al., 2002) Children use all speech information available
• Hyperarticulated speech styles (Lees & Burnham, 2005).e.g., teacherese (Håkansson, 1987).
• Cognitive skills activated + maturation
General DiscussionGeneral Discussion
Past Past Current Current Future Future
AVSP and reading link (Cave et al., 2007; de Gelder & Vroomen,
1998)
Burnham (2003)
Articulation-lipreading link (Desjardins et al.,1997)
AVSP (integrative module)
Auditory-only speech perception
Lipreading
Speech Production(articulation)
Reading
Some questions awaiting answerSome questions awaiting answer
Speech Production-Perception Link: the perception-production link: a developmental
perspective; e.g., testing children with speech disorders (Dodd et al., in press).
Visual Speech and Reading Link:Research on the link between study strong correlations between visual aspects of speech and reading This study correlations between reading and AO,
lipreading and articulation. Cavé et al. (2007); de Gelder & Vroomen (1998). Cross-language reading x AVSP studies.
Acknowledging…….Acknowledging…….
Prof. Denis Burnham
Prof. Barbara Dodd, Asc.Prof. Chris Davis, Prof.Kaoru Sekiyama
MARCS Auditory Laboratories & MARCS Baby Lab.
College of Arts, University of Western Sydney
Australian Postgraduate Award
Aprica Foundation, Osaka, Japan
Participating schools, teachers, parents and kids in Sydney