Prosody in Speech InteractionExpression of the Speaker and Appeal to the Listener

Klaus J. Kohler

IPDS, Kiel, Germany

8th Phonetic Conference of China, BeijingInternational Symposium on Phonetic Frontiers

18 April, 2008

1 The pradigm of systemic linguistics

• Following de Saussure, a language is modelled as being held together by a tight system of categorically distinct units that enter into syntagmatic structures.

• These units and structures are defined formally.

• If communicative function is considered at all, it is brought in post hoc and limited to linguistic function, e.g. differentiation of words, of sentence mode, various types of focus, information structure.

• The units are regarded as discretely different in their physical manifestation. – categorical perception

• Phenomena that cannot easily be accommodated in such a formal systemic framework, and depend on substantive identification in the flow of speech rather than on abstract paradigmatic differentiation are relegated to paralinguistics.

• If linguists do not leave paralinguistics to the social sciences but tackle it themselves they tend to apply their systemic framework

– categorical linguistic form

– no network of communicative functions

– no or insufficient contextualization in data acquisition, e.g. isolated sentences

2 The paradigm of Conversation Analysis

• naturally occurring data in talk-in-interaction

• phonetic and interactional analysis in parallel

• all details relevant, especially fine phonetic detail

• behaviouristic approach – focus on singularity of spontaneous data– generalizing to other data not without problems– systematic experimental data excluded– categories of functional – phonetic relation

emerge from the data° no intuitions° no testing of hypotheses

3 Combining aspects of both paradigms

• following the CA paradigm paralinguistic research must

– rely on data from spontaneous dialogue interaction

– have a functional orientation

– take fine phonetic detail into account from auditory and instrumental observation

A paradigm of speech communication

• based on these observations

– and relying on the native speaker’s intuition

– hypotheses are formulated

– and tested on data of large spontaneous corpora

• based on the results

– experimental data are collected in systematically controlled acquisition procedures ° situational and linguistic contextualization° dialogue simulation

– involving the speaker and the listener

– for generalizations to categories of communicative functions

• repetition of the cycle from data via hypotheses to categories and back again until satisfactory paralinguistic modelling has been achieved

• data-triggered deductive approach

• needs to start from initial postulates for speech communication

• Bühler’s Organon Model

factual world

sender receiver

representation

expression appealsymbol

signalsymptom

SIGN

Appell

4 Developing a functional framework for speech communication

• linguistics is typically concerned with the function of representation, i.e. propositional meaning

• it makes a distinction between given and new– this is information structure in the factual world

• it ignores the argumentation structure into which the speaker projects information – this introduces the functions of concluding/

opening an argument

4.1 Expression of the Sender

– early and medial synchronization of f0 peak contours

“He used to be slim.”

• the photo provides the given fact that “he used to be slim”

– but the speaker decides on its argumentative weight

– either a final conlusion “that’s it”

– or an opening argument “I see it like this”

– by focussing on low or high pitch in the accent

• contrast and expressive evaluation can now come in– introducing a new perspective in an opening

argument may be neutral: matter-of-fact observation“he’s slim in this old photo”contrastive: surprise observation ° accepting the contrast

“in this old photo he’s slim, but he no longer is”° opposing expressive evaluation of the contrast

“my word, he has put on a bit of weight”– medial – late peak, peak height, phonation

Expression superimposed on Representation

“He used to be slim.”

(a) Experimental Analysis of Argumentation Structure in German

• perceptual test paradigm, 11th ICPhS Tallinn 1987– shift of complete f0 contours through otherwise

constant utterances– f0 peak contour shift from early via medial to

late synchronization with the articulation of an accented syllable

– discrimination of pitch changes – identification of meaning changes– "Sie hat ja gelogen." "She's been lying."

70Hz

140Hz

85Hz

90Hz

110Hz

– functional link of peak synchronization shown by contextualization for Sie hat ja gelogen. °early

¶ Wer einmal lügt, dem glaubt man nicht. ¶ "Once a liar, always a liar."

°medial¶ Jetzt versteh ich das erst.¶ Now I understand.

°late¶ Oh!

– Identification as +- matching in context

Identification in context “now I understand”

Sie hat ja gelogen. "She's been lying."

early > medial

• pragmatic functions of peak contour synchronization

– early - finality° knowing° summarizing° coming to the end of an argument° resignation

– medial - openness° observing° realising° starting a new argument

– late - unexpectedness° observing, realising in contrast to one‘s

expectation° surprise° disbelief

• f0 peak synchronization strong cue for the three functional categories

• but there are other acoustic cues– shape of the f0 contour – durations of the pre-accent and accent syllables– energy levels in pre-accent and accent syllables

• these property values are usually coupled in the natural productions of the three categories

• so peak shift paradigm introduces an artifact• decoupling must be analysed systematically to

evaluate the contribution of each property

Coupling of F0 and energy time courses“sie hat ja gelogen”

‘medial'

'late’

F0

Energy

F0

Energy

• the characteristics of the 3 peak categories are– early – falling pitch into the accented vowel to

low level, decreasing prominence– medial – rising pitch into the accented vowel

to high level before fall, increasing-decreasing prominence

– late – rising pitch from low level late in the accented vowel to high before fall, late increasing prominence

• peak categorization went into development of KIM

The Kiel Intonation Model– which became the basis for prosodic labelling

of the Corpus of Read and Spontaneous Speech of German

– which in turn made corpus analyses possible

(b) Experimental Analysis of Argumentation Structure in English

• Same perceptual and functional differentiation of pitch peak contours in English– "She's been lying." exactly parallel– formally tested by Kleber

° MA dissertation, Kiel 2005 under my supervision

° SP2006 Dresden

"Form and Function of Falling Pitch Contours in English"

AS = accented syllable PAS = postaccented syllable

111 Hz

114 Hz

140 Hz96 Hz

76 Hz

AS PAS t

F0

She's gone to Malaga.

• basis for modelling peak categories in corpus analysis

• excerpts from the Film "The Queen" with Helen Mirren

• 'early' 'medial' 'late'

• 'early' and 'late' in dialogue in this film – "The sheer joy of being partial."– "Yes. ('late') Of course one forgets that as sovereign,

you are not entitled to vote." ('late')– "No." ('early' conceding)

"No." 'early' – accepting, conceding 'late' (2x) – emphatically rejecting

(c) Aspects of Argumentation Structure in Mandarin Chinese

• I first reported on the form and function of pitch peak contours at an international level in 1986 at AT&T Bell Labs, Murray Hill

• there was a young Chinese doctoral student in the audience, Chilin Shih– who had no knowledge of German – but who perceived the same psychophonetic

category changes – and equated them with Mandarin tones

° early peak – the low tone 3° medial peak – the falling tone 4° late peak – a combination of tones 2 + 4

• 3 conclusions deducible from these observations– aspects of pitch perception have a psychophonetic

basis independently of the language and its comprehension

– Mandarin uses similar pitch categorizations at the lexical level that are systematized at the phrase level in German and English: low vs. high fall

– so, how do Chinese speakers realise the types of argumentation structure associated with f0 peaks?

• It is to be expected that the functions of finality and openness in argumentation are basic in speech communication in any language.– they should therefore also be manifested in

adapted form in a tone language like Mandarin– hao with low tone 3

° "OK, I am forced to agree"° "OK, I am happy to agree"

– xing with rising tone 2 ° the same distinction in argumentation

Chinese hao "OK" (upper) und xing "OK" (lower), resigned (left), happy (right); Yi Xu, UCL

Chinese hao "OK" (upper) und xing "OK" (lower), resigned (left), happy (right);

Aoju Chen, MPI Psycholinguistics, Nijmegen

• In the 'resigned' as against the 'happy' context– hao

° either no rise or a much lower one ° intensity lower and descending more quickly° the syllable is shorter

– xing in the two contexts is similarly differentiated° lower vs. higher pitch rise° lower vs. higher and faster vs. more slowly

descending intensity° shorter vs. longer duration.

– both speakers differentiate the two contexts across the different word tones in the same way by lowering vs. raising pitch and prominence, superimposed on the lexical tones

– this essentially parallels signalling of the functional categories in non-tonal languages

• The conclusions to be drawn from these few data are– the substance – form – function relationship of

argumentation in Chinese needs detailed analysis– collection of a contextualized database– including all 4 tones of Mandarin Chinese

4.2 Appeal to the Receiver

• question, request, command are receiver-directed, – typical appeal functions– formally coded – lexically, morphologically,

syntactically, prosodically– thus of great interest to the linguist under the

perspective of propositional representation– resulting in simple associations, in German/English

° of question-word questions with falling pitch° of polarity questions with rising pitch

4.2.1 Signals to the reveiver in questions

• Rising vs falling in word-order questions– original rising; female speaker ANS, g091a013

“würde Ihnen das passen?” “would that suit you?”° final rise: openness as to ‘yes-no’° final fall: speaker expects ‘yes’° final late fall: speaker contrasts

¶ expected ‘yes’ ¶ with potential ‘no‘, which would be

inappropriate ¶ irritation

rising vs falling in word-order questionsoriginal rising; ANS, g091a013

“würde Ihnen das passen?”

4.2.2 Surprise signals to the receiver

• ‘focus’ – with peak contours– signals sender and fact orientation

• ‘focus’ to express receiver orientation – high rising valley contours– category of surprise

• expression superimposed on appeal

“He’s in Rome?”

1. matter-of-fact echo question

2. disbelieving echo question

3. increased disbelief

• Valleys may be synchronized early or late with the accented-vowel onset

• early and late valleys are functionally differentiated

– ‘Factual Surprise’, e.g. in an echo question as a request for repetition and confirmation in case of bad transmission

– ‘Affective Surprise’, expressing disbelief

5 The empirical basis of the paradigm

• I have presented a large number of audio examples to illustrate various categories of a network of functions in a paradigm of speech communication for paralinguistic research.

• They may strike some of you as anecdotal and therefore unscientific.

• These “anecdotal” examples are, however, based on two decades of empirical investigation into the production and perception of speech interaction, with particular focus on prosody.

• This research has combined corpus analysis with experimental data acquisition in the cyclic way I proposed at the beginning of this talk and which is further referenced in the published paper.

• So, my proposals have an empirical foundation which has not only proved to be reliable between data sets and experiments but also valid as a basis for explaining aspects of speech communication.

• Since I am a firm believer in Bertrand Russel’s important distinction between ‘knowledge by acquaintance’ and ‘knowledge by description’ I also prefer to introduce an audience to complex structures by the demonstration of examples that appeal to their intuitive knowledge and let description follow.

• The main message of my talk is that the many frontiers linguistics has set up need to be transcended.

I hope it's come across

Thank you for your attention!