‘joining in’ spontaneous conversation and improvisational music-making sarah hawkins 1, richard...
TRANSCRIPT
‘Joining in’spontaneous conversation andimprovisational music-making
Sarah Hawkins1, Richard Ogden2, Ian Cross1
{sh110, ic108}@cam.ac.uk [email protected]
Language, Music and Interaction, Philological Society,QMUL, Nov. 2012
1 Centre for Music & ScienceUniversity of Cambridge
2 Dept. of Language and Linguistic Science,University of York
What processes underpin interaction in music-making and speech?
Spontaneous interaction:conversation and musical improvisation
• A controlled environment in which to elicit (relatively!) natural joint (inter)action in unrehearsed• talking• music-making• non-musical play
• Initial observations and hypotheses
2
with unfamiliar instruments / objects
Speaker–listener neural coupling underlies successful communication
Stephens, Silbert, and Hasson (2010)
fMRI as speaker tells a story; another P listens.• a speaker’s brain activity is spatially and temporally
coupled with the listener’s activity• but only when the listener understands the speaker• most correlated patterns in listener are delayed relative
to speaker’s; but some are anticipatory• greater anticipatory speaker–listener coupling
→ greater understanding (independent comp. measure)
3
Stephens, Silbert, and Hasson (2010) PNAS 107(32) 14425-14430Neural: Charles Schroeder group; Edward Large group e.g.
Schroeder et al. (2008) TICS 12(3), 106-113 Fujioka, Trainor, Large & Ross (2012) J. Neurosci. 32(5), 1791-1802
Sociophonetics: Garrod & Pickering (2004) TICS 8(1), 8-11
Hypothesis: if coupling of neural oscillations underpins successful
communication, then we should find:
similar processes in music and speech differing only by• demands of the medium• function of the particular interaction
our aim: find a set of comparable tasks & measuresin music-making and conversing
CA framework: alignment and disalignment
4
Alignment and disalignment in talk
Superordinate, multidimensional terms
We don’t expect binary classification every time
5
hasn’t he got nice eyes?
Alignment DisalignmentTiming YES! ( 200 ms ) YES!
Phonetic form YES! Yes
Syntactic form Yes, he has Yes
Word choiceYes, they’re really gorgeous!
They’re ok
In music, we can expect similar patterns (Turino)
• How did you get here?• What do you think of the room?
6
demographic questions; initial consent
10 non-musical play
10 musical play
10 conversation
• card houses• tallest tower: blocks• market stall: playdough
• xylophone, kalimba• drums, claves
detailed musical questions; final consent; £8
• 9/11; Princess Diana’s death…• important event you shared
≥ 5 conversationMinutes
no shakers!
at least 2
Who?
“Pilot”:5 dyads,• 3 musician pairs• 2 non-musician• various tasks
“Experiment 1”6 dyads,• 3 musician pairs• 3 non-musician• 2-3 prescribed
tasks
7
Dyads:• friends• same-sex• native
speakers of English
• 18-30• university
educated• both musicians,
or bothnon-musicians
Starting set-up: familiarisation
• card house
• tallest tower of blocks
• playdough market stall
8
• Sri Lankan drums (2 types, one not shown)
• claves• circular
xylophone• kalimba
(mbira)
• ≥ 10 years’ formal training
• currently actively engaged in music at least once a month
• ≤ 7 years’ formal training
• no active music-making in past 4 years
9
non-musiciansmusicians
Recording• 4 video cameras• overhead omni mike• stereo pair (music)• 2 close-talking mikes
10
Recording• 4 video cameras• overhead omni mike• stereo pair (music)• 2 close-talking mikes
11
Looking for co-ocurrences
• Focus: alignment and disalignment• rhythmically: entrainment and failure to entrain
• Body movement is well established as marking important events (beats) in both speech and music
• What happens when such beats carry across the two modalities?
• A single framework for labelling events• tracking beats in speech and music: currently, Cummins
12
Example(s) of what we found placeholder
• Alignment “magic”: E1_MF1 29:43-29:51• beat continuation across modalities, and between
participants• perfect coordination• unscripted (music not ‘counted in’)
• Disalignment: E1_MF1 26:31-26:47
13
Alignment: beat continuation across modalities, and between participants
• The criterion is (for alignment): there is speech before or after where
• the music changes, and the music 'works'.
• The questions are: to what extent is the speech and music beat coordinated?
• and how does this compare when there is breakdown or less successful interaction?
• what happens with body movements and eye gaze?
14
Tentative Hypotheses (to be completed)
15
• look at the effector:• hands when playing• faces when speaking• presumably faces when singing together
• look at times of uncertainty….
Next steps
• Quantify: proportion of positive instances of categories• Theory: which? is there only one?
• Why? (Causes) do people entrain willy nilly or element of prediction from one or other
• if we can’t tell bottom up from top down, and there’s not a clear listener vs clear talker, what are we dealing with – the holy spirit?
• Theory: top down and bottom expectancies mesh: me, Narmour, Pearce/Wiggins….we need to work actively to get this working for a general theory – and using sp and music as our test bed seems an exciting way forward. Form a working group???
16
17
Daniel Halford
Satinder Gill
Rein Ove Sikveland
David Greatrex
Thanks to!
Hannah LeachNewton Trust, U. Cambridge; BA/Leverhulme Foundation Small Grant
Some random slides from past presentations that may help orient you re
our way of thinking
18
So why do music and speech seem fairly different, but we feel they are the same?
1. music and speech typically differ in the relative balance accorded to conveying phatic vs referential meaning—but this is a very loose difference
• functions of both modalities dictate what is important, and where we should look for guiding principles
• languages will differ – as will musics (structure)
2. these different balances in large part dictate the greater predictability of rhythm in music than in speech
19
some things that music and speech share
• wide range of rates, affected by many factors• phrase-final lengthening• predictable tonal endings: cadence, nuclear tone• internal/local rate change: rubato, asides, emotion,
floor holding• deviation from rhythmicity indicates
• emotion • phrasing• in speech, the demands of the actual words used
• regular rhythms are constructed, in both speech and music• consistent with most other perceptual approaches
20
Summary of a video clip (not included)• the background speaker, L, maintains a beat of about 460 ms in
speaking; and appears to lead the transition into music:• though R talks more, and is talking very casually,
she seems to entrain to L’s speaking beat
• they start playing about on ‘the current beat’: c. 800-900 ms
• gradually increase tempo to c. 700 ms
• look at the effector:• hands when playing• faces when speaking• presumably faces when singing together
21