Department of Linguistics and PhilologySpråkteknologiprogrammet(Language Technology Programme)Master’s thesis in Computational Linguistics

May 11, 2006

Visual Correlates of Disfluencyin Swedish

Mojgan Seraji

Supervisor:Adj. Professor Bertil Lyberg, Linköping University

Abstract

Speech is usually accompanied by different gestures such as head movements and move-ments of the eyebrows. These gestures seem to have a significant meaning in human-humanand human-machine communication. Such information is a key component for the progressin synthesizing naturally looking talking head. This thesis examines the movements of thehead and eyebrows to disfluency and how these movements or gestures are related to speechsignals in spontaneous speech within a communicative situation. The utilized method in thisexperiment is “Wavesurfer” for speech and movement analysis. To analyze the correlationof disfluency and head movements, this experiment was done under natural circumstances. Irecorded several minutes of communication of two subjects; one man and one woman, andlet them talk about an optional topic. Although strength and direction of the movements ofthe head and eyebrows vary widely from one speaker to another the results turned out thatboth speakers had a correlation of disfluency to the movements of the eyebrows. Disfluencyhad also an indirect influence to the head movements.

Contents

Abstract ii

Contents iii

List of Figures v

List of Tables vii

Preface viii

1 Introduction 11.1 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Background 32.1 Human Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Cognitive Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2.1 Broca’s Area . . . . . . . . . . . . . . . . . . . . . . . . . 42.2.2 M1-Mouth Area . . . . . . . . . . . . . . . . . . . . . . . 42.2.3 Wernicke’s Area . . . . . . . . . . . . . . . . . . . . . . . 52.2.4 Auditory Cortex . . . . . . . . . . . . . . . . . . . . . . . 52.2.5 Visual Cortex . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.3 Spontaneous Speech . . . . . . . . . . . . . . . . . . . . . . . . . 52.4 Conversational Interaction . . . . . . . . . . . . . . . . . . . . . . 52.5 Disfluency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.5.1 Tongue Tips and Slips . . . . . . . . . . . . . . . . . . . . 72.6 Research Tools in Speech Science . . . . . . . . . . . . . . . . . . 7

2.6.1 The Narrowband Spectrogram . . . . . . . . . . . . . . . . 72.6.2 The Wideband Spectrogram . . . . . . . . . . . . . . . . . 7

2.7 Language and Machines . . . . . . . . . . . . . . . . . . . . . . . 82.7.1 Talking Heads . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Experimental Design 113.1 Recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3.1 Wavesurfer . . . . . . . . . . . . . . . . . . . . . . . . . . 123.4 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

iii

4 Results of the Evaluation 154.1 Unfilled Pauses (UPs) . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.1.1 The Correlation of Unfilled Pauses to the Head Movements . 184.1.2 The Correlation of Unfilled Pauses to the Movements of the

Eyebrows . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2 Filled Pauses (FPs) . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.2.1 The Correlation of Filled Pauses to the Head Movements . . 204.2.2 The Correlation of Filled Pauses to the Movements of the

Eyebrows . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.3 Prolongations (PRs) . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.3.1 The Correlation of Prolongations to the Head Movements . 234.3.2 The Correlation of Prolongations to the Movements of the

Eyebrows . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.4 Truncations (TRs) . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.4.1 The Correlation of Truncations to the Head Movements . . . 274.4.2 The Correlation of Truncations to the Movements of the Eye-

brows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.5 Conclusions and Future Development . . . . . . . . . . . . . . . . 30

Bibliography 32

A Appendix 1 33A.1 Subject 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

A.1.1 file 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33A.1.2 file 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33A.1.3 file 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34A.1.4 file 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34A.1.5 file 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

A.2 Subject 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35A.2.1 file 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35A.2.2 file 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35A.2.3 file 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36A.2.4 file 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36A.2.5 file 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36A.2.6 file 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37A.2.7 file 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

A.3 Subject 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37A.3.1 file 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37A.3.2 file 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38A.3.3 file 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38A.3.4 file 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38A.3.5 file 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

B Appendix 2 40

iv

List of Figures

2.1 Functions of the brain, adopted from the final project of the "Biotechnol-ogy and Its Social Impact (MOL427/WWS462)" . . . . . . . . . . . . 4

2.2 Narrowband spectrogram . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Wideband spectrogram . . . . . . . . . . . . . . . . . . . . . . . . . . 82.4 Synface, adopted from the SYNFACE project at kth . . . . . . . . . . . 9

3.1 Spectacles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Wavesurfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.1 Unfilled pause and head movements in three dimensions, the scale of they-, z- and x-axis is counted in mm . . . . . . . . . . . . . . . . . . . . 16

4.2 The movements of the eyebrows before an unfilld pause . . . . . . . . . 174.3 Unfilled pause after the movements of the eyebrows . . . . . . . . . . . 194.4 Filled pause and head movements in three dimensions . . . . . . . . . . 214.5 Filled pause and the movements of the eyebrows . . . . . . . . . . . . 224.6 Prolongation before head movements . . . . . . . . . . . . . . . . . . . 244.7 Head movements after a prolongation . . . . . . . . . . . . . . . . . . 254.8 Prolongation and the movements of the eyebrows . . . . . . . . . . . . 264.9 Truncation and head movements in three dimensions . . . . . . . . . . 284.10 The movements of the eyebrows before a truncation . . . . . . . . . . . 294.11 Truncation after the movements of the eyebrows . . . . . . . . . . . . . 30

B.1 Unfilled pause and head movements in three dimensions, when one ofthe subjects uttered “... på arbetstid UP Å det var ju inte ...” . . . . . . . 41

B.2 Unfilled pause and head movements in three dimensions, when one ofthe subjects uttered “... mitt öra ju. (UP) Och sen så vaknade jag ...” . . 42

B.3 Unfilled pause and head movements in three dimensions, when one ofthe subjects uttered “... på hela kudden (UP) Och då ...” . . . . . . . . . 43

B.4 Unfilled pause and head movements in three dimensions, when one of thesubjects uttered “... så fick jag då komma till överläkaren UP Schiratski ...” 44

B.5 Unfilled pause and head movements in three dimensions, when one ofthe subjects uttered “... fem doktorandpoäng (UP) Och då tänkte jag ...” 45

B.6 Unfilled pause and head movements in three dimensions, when one ofthe subjects uttered “... tju ett nu (UP) eh ...” . . . . . . . . . . . . . . . 46

B.7 Unfilled pause and head movements in three dimensions, when one ofthe subjects uttered “... vad man vill så (UP) så får vi se vad det blir ...” . 47

B.8 Unfilled pause and the movements of the eyebrows, when one of thesubjects uttered “... högst upp på det här UP huset och där ...” . . . . . . 48

v

B.9 Filled pause and head movements in three dimensions, when one of thesubjects uttered “... Och eh (FP) och så en dag ...” . . . . . . . . . . . . 49

B.10 Filled pause and the movements of the eyebrows, when one of the sub-jects uttered “... Eh (FP) går på högskolan ...” . . . . . . . . . . . . . . 50

B.11 Prolongations and head movements in three dimensions, when one of thesubjects uttered “... Det e om (PR) ”marackesh” ...” . . . . . . . . . . . 51

B.12 Prolongations and the movements of the eyebrows, when one of the sub-jects uttered “... när hon började i (PR) Malmö så ...” . . . . . . . . . . 52

B.13 Truncations and head movements in three dimensions, when one of thesubjects uttered “... den ligger lili TR lite utanför stan inte långt ...” . . . 53

B.14 Truncations and head movements in three dimensions, when one of thesubjects uttered “... nej hal TR halvtid det jobbar hon bara ...” . . . . . . 54

B.15 Truncations and the movements of the eyebrows, when one of the sub-jects uttered “... den ligger lili TR lite utanför stan inte långt ...” . . . . 55

B.16 Truncations and the movements of the eyebrows, when one of the sub-jects uttered “... ah nä(TR)när hon kommer hem ...” . . . . . . . . . . . 56

B.17 Truncations and the movements of the eyebrows, when one of the sub-jects uttered “... är ju som kyckli(TR)kyckling i sig smakar ju heller ...”

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

vi

List of Tables

4.1 Utterances containing unfilled pauses . . . . . . . . . . . . . . . . . . 154.2 Utterances containing filled pauses . . . . . . . . . . . . . . . . . . . . 204.3 Utterances containing prolongations . . . . . . . . . . . . . . . . . . . 234.4 Utterances containing truncations . . . . . . . . . . . . . . . . . . . . . 27

vii

Preface

First and foremost, I would sincerely like to thank my supervisor Professor BertilLyberg, for his guidance, encouragement and support during this study. He has beenan endless source of inspiration and his valuable suggestions and knowledge in thisarea have been of great help in my thesis. Also I want to express my special gratitudeto his doctoral students Mustapha Skhiri and Sonia Sangari who helped me a lotin an exciting way with the interviews and the recording parts of this pilot study.

viii

1 Introduction

When we talk despite all ages, cultures, backgrounds we have, we usually move thehead, show various facial expressions and use body language. These gestures aidthe understanding of the communication and they also convey a lot of additionalinformation, i.e. visual information about the speaker such as the speaker’s mood.These gestures, in other words, nonverbal communications are mostly related to thesubject we are talking about and the situation at hand. For example, some movementssuch as body language facilitate turn-taking in a conversation. Hence, gestures playan important role in language production and comprehensions. (Kelly, 2001)

Many facial expressions and head movements are also linked to the semantic andprosodic structure of a text. For example, a stress on a word is often accompaniedby a nod of the head or rising the voice at the end of a phrase can be linked to a riseof the head, possibly combined with rising eyebrows. (Hans Peter Graf, 2002) Manyresearches have been done in this field to make animated talking agents (or talkinghead) to look more naturally in their nonverbal communication.

1.1 Aim

The aim of this study is to better understand the correlation between movementsof the head and eyebrows and disfluency. A secondary aim is to investigate howthese movements are related to speech signals in spontaneous speech within a com-municative situation. In this pilot project only two subjects have participated in theinterviews. I have also used other data collections of one additional subject who hadtaken part in another study related to head movements. Therefore, upcoming state-ments in this paper is based on these three subjects (two men, one woman) I haveobserved. All data analysis in this research can be used in construction of animatedtalking agents.

1.2 Outline of the Thesis

This thesis covers following chapters: chapter 1 presents a brief introduction tothe concept of nonverbal communications and also introduces the purpose of thisstudy. Chapter 2 presents the relevant background information which discuss humanspeech, cognitive functions, spontaneous speech, conversational interaction, disflu-ency, research tools in speech science, language and machines. In chapter 3, thedesign of this experiment follows and it introduces the interview selection, the mea-surements and how I utilized the apparatus “Wavesurfer”. Last chapter; chapter 4,introduces the results of the evaluation which consists of two main studies, i.e. the

1

correlation of disfluency to the head movements and the correlation of disfluency tothe movements of the eyebrows.

2

2 Background

2.1 Human Speech

The fundamental view of human speech is based on the concept of natural sounds.Early human used to imitate of natural sounds they heard around themselves. Forexample: the CAWCAW sound, which have been using in the early man’s history torefer to an object flew by. Even in all modern languages there are some words whichhave almost the same pronunciations as naturally occurring sounds. Some example inEnglish can be cuckoo, splash, bang, boom, ding-dong, rattle, buzz, hiss, screech andforms such as bow-wow and the view of this type has been called as the “bow-wowtheory” or the “yo-heave-ho theory” that refer to a person who is under a physicaleffort especially when that physical effort involve several people. (Yule, 1985) Thereare also other original sounds coming from natural cries of emotion, such as pain,anger, joy and expressive noises which people make of emotional reaction, such asWow, Ugh and Oops contain sounds which are not used in their language but arecounted as source sounds or natural sounds.

Human accompany much of their speech with physical gestures such as pointingand raising of the arm, bend at the elbow and so on. All these have to do with howhumans use language to interact with each other, socially or emotionally; how theyshow friendliness, or hostility, or annoyance, pain or pleasure. (Yule, 1985)

There is an idea regarding the origins of the sounds of language that there is alink between physical gesture and the production of oral sounds. It seems completelyreasonable that physical gestures, involving the whole body, can indicate a lot ofemotional situations and intentions. Physical gestures, using body, hands, face andhead, are methods or processes of nonverbal communication which still are usedby modern human with developed linguistic skills. Hence, it is said that physicalgestures were developed as a method for communications. On the other hand, thegestures involving mouth, the movement of tongue, lips and so on were recognizedas oral gestures. The movement of the tongue (oral gesture) in a “goodbye” messageand the waving of the hand or arm (physical gesture) give both a similar message.(Yule, 1985) In deed, we can use mime or specific gestures to transmit an informationin order to a variety of communicative purposes.

2.2 Cognitive Functions

Speech is, without doubt, one of our most important abilities. We use various fea-tures of language to produce and understand linguistic messages. The ability of us-ing language is located in two regions of the human brain that are known to playimportant roles in speech production and processing: Broca’s area and Wernicke’s

3

Figure 2.1: Functions of the brain, adopted from the final project of the "Biotechnology andIts Social Impact (MOL427/WWS462)"

area. These areas were identified through the study of brain damage patients in nine-teenth century. Patients with damage to Broca’s area exhibit an inability to producegrammatical sentences, although they can use single words properly. Patients withdamage to Wernicke’s area, in contrast, produce well-formed but meaningless sen-tences. (Geschwind, 1965) These areas are localized only in the left hemisphere ofthe brain. Brain activity is involved in hearing a word, understanding it, then sayingit, and all these processes follow a definite pattern. When a word is heard and com-prehended via Wernicke’s area, this signal is transferred to Broca’s area to do somepreparations for producing it. Finally, a signal is sent to the motor area to physicallyarticulate the word.

2.2.1 Broca’s Area

Broca’s area described in 1861 by Pierre Paul Broca and is responsible for deliveringa list of words and parts of words to produce meanings, in other words, it is respon-sible for semantic processing. This area is not exactly a speech area, but is associatedwith the process of articulation of speech. It controls not only spoken, but also writtenand signed language production. Patients with lesion of this area can understand lan-guage and conversation, even complex concepts, but cannot talk coherently. (KeithA. Johnson, 2000)

2.2.2 M1-Mouth Area

This is the area of the brain is linked to control the physical movements of the mouthand articulators in producing speech. This part is a part of motor cortex, and controlsthe muscles of the face and mouth. The rest of motor cortex controls other parts ofthe body’s movement. It is located near Broca’s area and works in speech tasks alongwith Broca’s area. (Keith A. Johnson, 2000)

4

2.2.3 Wernicke’s Area

Wernicke’s area is an area of semantic processing. It is associated with memory func-tions, hearing function and object identification, as well as language comprehension,in written or spoken language. Wernicke’s area works with Broca’s area, Wernicke’sarea takes care of incoming speech and Broca’s area controls outgoing speech. (KeithA. Johnson, 2000)

2.2.4 Auditory Cortex

This area of the brain manages recognizing and receiving sound. When people speakor read words aloud, there is evidence that they listen to themselves as they are speak-ing in order to make sure they are speaking correctly. (Keith A. Johnson, 2000)

2.2.5 Visual Cortex

This area is also known as the striate cortex and it has responsible for vision. Thevisual areas of the brain are the first parts of the brain which are activated in readingand object naming. Above this visual area there is another region which is associatedwith object naming and word reading, and is thought of as supplementary to theprimary visual cortex. (Keith A. Johnson, 2000)

2.3 Spontaneous Speech

Spontaneous or unprepared speech is the most natural form of an interactive com-munication. This phenomena unlike written text, is online i.e. when speakers saysomething which is needed to be repaired, there is a little opportunity to take it back.In contrast, written text provides always the chance to be revised and rephrased andthat is one of the major difference between spontaneous speech and written text.

In spontaneous speech speakers use unfilled pauses (or silent), filled pauses (suchas err, eh, uh, uhm), truncated words, restarts, mispronunciations, “editing terms” likeoops, sorry, no, I mean and so on. These are referred to as disfluency also dysfluency,nonfluency, disturbance and discontinuity.

2.4 Conversational Interaction

Conversation is an activity where two or more people speak by turns. Althoughspeaking turns are not pre-allocated in conversation and it is clear that turn distri-bution is systematic. But typically, only one person speaks at a time and the othersare expected to keep silence between speaking turns. (Despite the fact that it is nottrue in some cultures.) If more than one speaker tries to talk at the same time, oneof them usually stops. Therefore participants wait until one speaker shows that he orshe has finished by signaling a completion point, for example, by pausing at the endof a phrase or a sentence. But if the speaker wants to keep his or her own turn shouldavoid having pause at the end of sentences. Speaker should make sentences run on byusing connectors like and, and then, so, but or by using hesitation markers or filledpauses such as er, em, uh, ah.

5

If other participants want to talk, they usually indicate it by different ways bothverbal and nonverbal behaviours such as making short sounds usually repeated whilethe speaker is talking, shifting their body or using facial expressions to signal thatthey have something to say. These signals are defined as turn-taking signals. In thissituation the participants can show their character as “rudeness” (the speaker cutanother speaker) or “shyness” (the speaker keeps waiting for an opportunity to takea turn and it seems to not occur). (Yule, 1985)

Many attempts to make naturally looking talking heads have focused on turn-taking signals to process natural language behaviour in conversational interaction.

2.5 Disfluency

Several approaches have been studied in disfluency or hesitation phenomena in nor-mal speech, for example Freud discussed disfluency from a psychological point ofview as something that opens our inner character or individuality. Within stutteringresearch, speech therapists, psychologists and speech pathologists have stated thedifference between pathological speech, like stuttering (or stammering), and normaldisfluencies of human languages. Many researches have also been done in disfluencyfrom a gender perspective, connected to gestures and body language.

When we talk about some topics, we have always a number of choices of wordsand the ways of expressing them. The more formal, structured and discipline, thefewer the options. That is our choice about how we talk and how fluent we are. Eklund(2004) noted that disfluency production is largely dependent on psychological andindividual factors. He has also noted that disfluency to some degree is under speakercontrol. Some disfluencies such as silent or unfilled pauses and filled pauses (uh, ah,er, and um) are based on the hypothesis of that these pauses or breaks in speech flowprovide the time for the production of speech to search for the next word or phrase.

Eklund (2004) has observed the disfluency phenomena in the following fields:Unfilled Pauses (UPs) Silent parts in fluent speech, an example would be “I want

a ..... go to school”. Unfilled pauses are not always disfluencies, sometimes we usepauses to mark sentence boundaries (Deese, 1978) and all silences shorter than 250ms, are not counted as unfilled pauses, they are more like silences of longer duration.(Goldman-Eisler, 1968)

Filled Pauses (FPs) Also called “vocalized pauses”, such as “eh, em, er, um, ah,..”.

Prolongations (PRs) Some phones or segments which are longer than in normalfluent speech, like “I’m comiiiiiing”.

Truncations (TRs) Cut-off words or syllables, in other words, interrupted wordsor syllables, e.g. “li..li..little”.

Mispronunciations (MPs) Words with the wrong pronunciation and it is almostsimilar to the term “slip-of-the-tongue” (see next section), such as “black bloxes”(for “black boxes”).

Repairs (REPs) This term is a kind of self-corrections, sometimes includingsubstitutions (I want to drink a cup of coffee tea), repetitions (can I can I have a cupof tea), insertion (I want to have a cup of tea hot tea).

Explicit Editing Terms (EETs) Words or phrases like “sorry”, “you know”,“opps” and so on.

6

2.5.1 Tongue Tips and Slips

As language users, we all experience occasional difficulty in getting the brain andspeech production to work together. The tip-of-the-tongue phenomenon is when weknow the word but it just won’t come out on the tongue. Researches have shown thatspeakers generally have a correct phonological outline of the word, they know theinitial sound of the word and even the number of syllables in the word. It indicatesthat our “word-storage” can be partially organized on the phonological informationand some words in that “word-storage” can be easily retrieved than other words. Forexample, speakers produced secant, sextet and sexton, when they were asked to namea type of instrument; sextant.

A similar type of speech error is described as slip-of-the-tongue, also is known asa Spoonerism, after the Rev. William A. Spooner, an Oxford dean, who was famousfor his tongue-slips.(Yule, 1985) Spoonerism results when a sound is carried overfrom one word to the next, in other words, an intended word is replaced by anotherword such as the thine sing (for “the sign thing”), black bloxes (for “black boxes”) ornoman numeral (for “roman numeral”). These examples give us a clue to the normalworking of the human brain functions.

2.6 Research Tools in Speech Science

One of the research tools which provides an analysis of the acoustic speech signalis spectrogram. Moreover, spectrogram (or sonogram) is a visual description of anacoustic signal and give an analysis of different components of speech. In additionto spectrogram, pitch contour and waveform of the acoustic signal are other ways tomake sound waves visible. There are two different kinds of spectrograms, the nar-rowband spectrogram and the wideband spectrogram and both show different things.

2.6.1 The Narrowband Spectrogram

A narrowband spectrogram displays horizontal bands (consider Figure 2.2) whichrepresent the harmonics of the glottal source. The darker bands represent the har-monics that are closest to peaks of resonance in the vocal tract. The lighter bandsrepresent harmonics whose frequencies are far away from the resonance peaks. Thebandwidth of the filter used to generate narrowband spectrograms is usually some-where between 30 and 50 Hz. Narrowband spectrograms are traditionally being usedfor measuring the fundamental frequency and intonation. (Gloria J. Borden, 1980)

2.6.2 The Wideband Spectrogram

Wideband spectrograms are broad bands of energy that depict the formants; whichare the actual peak of resonance at a particular moment. They are dark bands on awideband spectrogram which correspond to a vocal tract resonance. Different vocaltract shapes will produce different formant patterns. Formants are conclusive for un-derstanding of vowels. In contrast, formants in consonants are less clear and havelower intensity.

Wideband spectrograms are used in spectrogram reading because they give usmore information about what is going on in the vocal tract. Speech scientists are more

7

Figure 2.2: Narrowband spectrogram

Figure 2.3: Wideband spectrogram

interested in the changing resonance than the harmonic of the glottal source. There-fore, wideband spectrogram is most appropriate alternative to measure formants com-pared to the narrowband spectrogram. Wideband spectrogram in Figure 2.3 showsthe horizontal bands of energy that represent the formants are composed of individ-ual vertical lines. The blank spaces of the wideband spectrogram indicate silence,including pauses and some sort of silent gaps that are generated by voiceless stopclosures. The bandwidth of a filter used to generate wideband spectrograms is be-tween 300 and 500 Hz. Researches who want to measure the duration or the time ofacoustic occurrences use wideband spectrograms. (Gloria J. Borden, 1980)

2.7 Language and Machines

In 1738, Jacques de Vaucanson produced amazing mechanical duck which could per-form the act of drinking water and eating grain. The containing was digested and thenexcreted via a mysterious chemical process in its stomach. (Yule, 1985) This is oneexample of human ambitions that have always tried to create a model of natural or-

8

Figure 2.4: Synface, adopted from the SYNFACE project at kth

ganisms, and natural articulation of speech sounds have always been one interestingaspect to be modeled by machines. Many devices of talking machines were modeledfrom the function of human vocal tract to produce phrases and sentences, pronun-ciation of individual words, intonation and pausing as well as syntactic rules for theformation of natural language sentences. However, according George Yule, the devel-opment of synthetic speech, would only produce a model of speech articulation not amodel of “speaking”. Having something to say is an attribute of the human’s mentalprocesses and attempting to model that attribute is, the modeling of intelligence.

2.7.1 Talking Heads

Audio-visual speech synthesis, is based on automatic generation of voice and facialanimation. The visual speech synthesis uses 3D (three dimensional) polygon mod-els which are parametrically articulated and deformed. Researchers main interest inhead and facial movements is to synthesis talking heads which look as naturally aspossible. Munhall (2004) found in his research that head movements contribute peo-ple to understand an uttered message more accurately. Cassel (2000) also points outin his article that a user of an animated talking agent with nonverbal communicationunderstand better and is more cooperative than an animated talking agent with nononverbal communication.

One of the main application of audio-visual speech synthesis is within the sphereof hearing impaired, in fact, people who have hearing difficulties to communicateby telephone. These people usually use lip-reading during conversations. However,it dose not work over the telephone, but this is done in the SYNFACE project whichuses a computer program. A computer that is attached to user’s telephone will showan artificial face. When the user makes a phone call she or he is able to listen and atthe same time watch the artificial face to lip read it and follow all information.

Figure 2.4 shows a SYNFACE on a computer screen. Users can choose differentfaces by clicking on the buttons under the face and there is also another control underthe face to change the volume of the sound. On the right, there is a phonebook anda keypad for dialing. The user can also attach a face to a telephone number in his orher phonebook in order to choose the same face every time the user dial that number.

9

As means to make communications easier, SYNFACE can also be used on laptopcomputers and mobile phones.

10

3 Experimental Design

To analyze the movements of head and eyebrows in a conversation, this experimenthas been done under natural circumstances. Two native Swedish speakers have partic-ipated in the recording of an interview. In order to make the dialogue as spontaneousas possible, the subjects have been allowed to talk about an optional topic. To get awider analysis of spoken dialogue, I have also used other data collections belongingto one additional subject who had taken part in another study related to head move-ments. Several minutes of the interview of each subject were listened and varioustypes of disfluency were localized, measured and analyzed. In total, 22 minutes ofrecorded material were analyzed.

3.1 Recording

The participants interviewed one by one in a sound-isolated studio. They wore spe-cial spectacles with five hemispherical markers on it, for recording of the head move-ments. These hemispherical markers have a diameter of 4 mm and is made of a re-flecting material. Two markers were also attached to the subject’s eyebrows (oneon each eyebrow) for the purpose of recording the movements of the eyebrows. Thespectacles lack the upper part so that the movements of the eyebrows can be recordedby the system (figure 3.1).

Moreover, the studio is equipped by a microphone which registers the sound andfour IR cameras (cameras with infrared light) for capturing the movements of the

Figure 3.1: Spectacles

11

markers with a frequency of 60 Hz. In other words, the cameras flash infrared lights1

to the markers and then these lights will be reflected into the cameras to registerthe movements. IR cameras can register the light in three dimensions (y-, z-, and x-axis). Each camera catch the light from a certain angle and that can be adjusted bymoving the camera. All movements data are sent via four special video processors toa Mac computer for storing the data and post preparation. All equipments are fromthe manufacture “Qualisys” and both the software which is used in Mac computer andthe system are called “MacReflex”. The recording time of the movements is limited.The movement data is in a special format; TSV (Tab Separated Values) format, inorder that, these formats can be easily read by Wavesurfer.

The sound is recorded and sent to a PC for storing and post preparation. Thesound is recorded in two channels, a sound signal and a sync signal.

The sync signals are used for synchronizing the sound and the movements, thesesignals can be filtered out by Wavesurfer. The accuracy of the measurement of themovements is about 0.1 mm in three dimensions.

3.2 Measurements

The co-ordinates of the markers are measured in order to get the values of the vertical-(y), the inclined- (z) and the horizontal (x) movements of the head and also the ver-tical (y) movements of the eyebrows. The vertical directions (y) exhibit the motionsof “up or down”, the inclined directions (z) exhibit the motions of “front or back”and the horizontal directions (x) display the motions of “left or right” (ConsiderFigure 3.2.) The co-ordinates of the markers at the eyebrows are measured only invertical movements which is “up or down” (y-axis) and then are transformed to a co-ordinate system. The acoustic signals which are synchronized with the movements ofthe markers are used as means to see the correlation of disfluency and the movementsof the head and eyebrows.

3.3 Apparatus

3.3.1 Wavesurfer

Wavesurfer is a signal processing program which is utilized as a tool for analysisof speech signals. Wavesurfer is enormously useful for doing the tasks in speechresearch and education, like speech analysis and speech transcription.

In Wavesurfer both acoustic signals and movement signals which have alreadybeen synchronized can be studied in different windows at the same time. The soundis represented by spectrogram, waveform and pitch contour and the movements arerepresented by three windows (each for one type of movements i.e. one for verticaldirection or y-axis, one for inclined direction or z-axis and finally one for horizontaldirection or x-axis) with curves. Rising of a curve indicates movement “up”, “front”

1Infrared is the region in the electromagnetic spectrum that falls between radio waves and visiblelight. Infrared can’t be seen, but it can be felt as heat given a strong energy source. A camera equippedwith infrared-sensitive film can take pictures of “warm” objects in low-light environments. Most peopleare familiar with infrared as implemented in remote control devices.

12

Figure 3.2: Wavesurfer

or “right” and falling of a curve indicates movement “down”, “back” or “left”. Fig-ure 3.2 represents an unfilled pause and the head movements.

3.4 Observations

In this pilot study I have looked at four types of disfluency in relation to move-ments of the head and eyebrows, i.e. unfilled pauses, filled pauses, prolongations andtruncations. I listened to all sound files in order to trace disfluency and simultane-ously studied the related TSV files in three directions (y-, z- and x-axis) for headmovements and in one direction (y-axis) for the movements of the eyebrows in dif-ferent windows in Wavesurfer. The mentioned types of disfluency were localized andmarked in Wavesurfer and were then measured to see how long that type of disfluencylasted. Finally, the correlation between disfluency and the movements of the head andeyebrows were analyzed. I had access to 10 TSV files related to the movements ofthe eyebrows of one subject for the observation of the motions of the eyebrows and

13

22 TSV files of the head movements of three subjects for the observation of the headmovements. The result of this analysis will be presented in the following chapter.

14

4 Results of the Evaluation

The speech was transcribed orthographically word-by-word and then after listeningto the all conversations, the correlation of disfluency (according to the disfluencycategory described in the following) to the movements of the head and eyebrowswere analyzed. The scales of y-, z- and x-axis of the head movements are counted inmm and the scale of y-axis of the movements of the eyebrows is counted in a tenthof mm.

4.1 Unfilled Pauses (UPs)

An unfilled pause refers to a silent part of fluent speech, when a speaker turn silentfor shorter or longer periods of time. Sometimes these silences can barely perceivedbut they can be also very long indeed. (Eklund, 2004) Unfilled pauses are the mostproblematic type of the disfluency category because all unfilled pauses are not alwaysdisfluencies, sometimes we use pauses to mark sentence boundaries (Deese, 1978)and all silences shorter than 250 ms, are not counted as unfilled pauses, they aremore like silences of longer duration. (Goldman-Eisler, 1968)

Table 4.1 displays some examples of unfilled pauses which occurred to the sub-jects during the interview:

Table 4.1: Utterances containing unfilled pauses

Subject Utterance DurationSubject 1 ... högst upp på det här UP huset och ... 0.551 ms

... at the top of this UP house and ...Subject 1 ... jag ska prata lite tydligare UP om mat ... 1.647 ms

... I will talk more obvious UP about food ...Subject 2 ... dom har UP beställt tid ... 1.26 ms

... they have UP made an appointment ...Subject 2 ... så fick jag då komma till överläkaren UP Schiratski ... 1.022 ms

... then I got to come to the doctor UP Schiratski ...

However, unfilled pauses were the most common type of the disfluency categoryin this pilot study and I found 121 unfilled pauses compared to 71 filled pauses,45 prolongations and 17 truncations. The quite common location for unfilled pauseswas right before the head of the phrase, in other words, this hesitation phenomenaoccurred mostly before the important items in a phrase. Unfilled pauses also occurredinside words, an example in this case would be:

• “... studentUPlägenhet ...”, “... studentUPflat ...”.

15

Figure 4.1: Unfilled pause and head movements in three dimensions, the scale of the y-, z-and x-axis is counted in mm

Another types of unfilled pauses were, when they occurred between grammati-cally complete forms, for example:

• “... så fick jag då komma till överläkaren, UP Schiratski ...”, “... then I came tothe doctor, UP Schiratski ...”.

The last types of unfilled pauses that I discovered in this experiment were UPsthat speakers employed in order to make a plan for pursuing their meaning, like:

• “... hon bor hemma UP och vet inte vad hon vill göra ...”, “... she lives at home(or she lives with us) UP and she does not know what to do ...”.

16

Figure 4.2: The movements of the eyebrows before an unfilld pause

17

4.1.1 The Correlation of Unfilled Pauses to the HeadMovements

The localized areas of unfilled pauses were studied as means to determine whetherthere were any head movements related to the unfilled pauses. In this experiment Ifound out that there was no significant relation between them. However, the speakersheld back and froze their gestures when they got disfluent and then head movementsoccurred later on. In other words, first there was an unfilled pause while the speakersheld still and then the head movements occurred.

Figure 4.1 displays the head movements and an unfilled pause for one of thespeakers uttering the sentence “... Karlsson och Granström var UP assistenter ochhade labbarna ...”, “ ... Karlsson and Granström were UP assistants and took care ofthe labs ...”. As is shown in this figure, first an unfilled pause takes place while thespeaker freezes his gestures and then strong head movements occur. (Head move-ments in y- and z-axis)

4.1.2 The Correlation of Unfilled Pauses to the Movements ofthe Eyebrows

The analysis of the correlation of unfilled pauses to the movements of the eyebrowsshowed that there were strong movements before unfilled pauses occurred, i.e. firstthe movements of the eyebrows happened and then the motions went down to unfilledpauses.

The movements of the eyebrows in the relation with unfilled pauses can introducethe case of focal accent, i.e. the speaker may signal a focal accent in a sentence. Inthis case, unfilled pauses can be completely dependent on the semantic content.

Figure 4.2 illustrates the movements of the eyebrows when one of the subjectsuttered the sentence “... hon ville ha en annan lägenhet som UP nu har hon fått ...”,“... she wanted to have another apartment as UP she now has got ...”. In this figure,first strong motions of the eyebrows appear, which can signal a focal accent, and thenthe movements are followed by an unfilled pause, which is displayed in Figure 4.3.

18

Figure 4.3: Unfilled pause after the movements of the eyebrows

19

Table 4.2: Utterances containing filled pauses

Subject Utterance DurationSubject 1 ... hon eh FP hade tur för när ... 0.239 ms

... she eh FP was lucky because when ...Subject 1 ... emigration och eh FP vad heter den kursen eller ... 0.571 ms

... emigration and eh FP what is called that course or ...Subject 2 ... det var lite annorlunda på den tiden eh FP och jag ... 1.943 ms

... it was a little different at that time eh FP and I ...Subject 2 ... så ska vi beställa tid till eh FP överläkaren ... 1.258 ms

... so we are going to make an appointment with eh FP the doctor ...

4.2 Filled Pauses (FPs)

Filled pauses or vocalized pauses describe “eh, em, er, ah, um”. Filled pauses werethe second most common type of disfluency category. (71 filled pauses were foundin this experiment.)

Table 4.2 shows some utterances which include filled pauses that subjects pro-duced during the interview.

Mostly, filled pauses signal that there are many options available to the speakerbut no commitment has yet been made by the speaker and sooner or later he or shewill choose one of the available alternatives.

4.2.1 The Correlation of Filled Pauses to the Head Movements

The filled pauses were analyzed to find out if there was any relation between thesepauses and head movements. I discovered no crucial relation between filled pausesand head movements. On the other hand, filled pauses were employed as a sort ofplace-holders for the next word or to continue on, to the next sentence. Somehow,filled pauses signaled that the speakers had many options available of which theyelected one to continue their speech. Speakers employed these hesitation sounds toindicate uncertainty or to maintain control of the conversation while thinking of whatto say next. Consequently, filled pauses did not add any new information to the con-versations.

Figure 4.4 displays the head movements in three dimensions and a filled pausewhen one of the subjects uttered the sentence “... så att eh FP men jag kunde ...”, “...so eh FP but I could ...”. There is clearly no relation between filled pauses and headmovements.

4.2.2 The Correlation of Filled Pauses to the Movements of theEyebrows

In this experiment I found no significant relation between filled pauses and the move-ments of the eyebrows.

In Figure 4.5, the movements of the eyebrows and a filled pause are displayed forone of the subjects uttering “... språket är ju mera eh FP det är ju inte just ...”, “... thelanguage is more like eh FP it is not just ...”.

20

Figure 4.4: Filled pause and head movements in three dimensions

21

Figure 4.5: Filled pause and the movements of the eyebrows

22

4.3 Prolongations (PRs)

Prolongations describe phones or syllables which are longer than in normal fluentspeech. Prolongations were the third most common type of disfluency category. (45prolongations discovered in this study)

Table 4.3 presents some examples of prolongations which occurred to the sub-jects during the interview:

Table 4.3: Utterances containing prolongations

Subject Utterance DurationSubject 1 ... köper man dem iii PR bitar ... 0.385 ms

... you can buy them iiin PR pieces ...Subject 1 ... utan det äää PR sexton grader ... 508 ms

... but also it iiis PR sixteen degree ...Subject 2 ... det var fullt iii PR väntrummet ... 0.400 ms

... it was full iiin PR the waiting room ...Subject 2 ... Schiratski som dååå PR tittade på det här ... 0.633 ms

... Schiratski who theeen PR looked at this ...

Prolongations can be seen as another way of hesitating without being silent. Inthis case, speakers hesitated by drawling phones to continue their speech withoutbeing silent.

4.3.1 The Correlation of Prolongations to the Head Movements

The analysis of the correlation of prolongations to the head movements showed thatthere was no relation between them. Instead, the speakers held back and froze theirmotions when they got disfluent and then a head movement took place. In otherwords, first a prolongation occurred while the speakers held off, without any actingand then head movements appeared.

Figure 4.6 illustrates the head movements and a prolongation for one of thespeakers uttering the sentence “... 40 poäng i alltsååå PR språkdelen i fonetik ...”,“... 40 points in that is to saaay PR language field in phonetics ...”. This figure showsno correlation of prolongations to the head movements.

As is shown in Figure 4.7, head movements appear after a prolongation, which isdisplayed in the previous figure (Figure 4.6.)

4.3.2 The Correlation of Prolongations to the Movements ofthe Eyebrows

I found no crucial relation between prolongations and the movements of the eye-brows.

Figure 4.8 displays the movements of the eyebrows and a prolongation for oneof the subjects uttering “... hon läser ju dååå PR det blir mycket ...”, “... she studiestheeen PR it becomes too much ...”.

23

Figure 4.6: Prolongation before head movements

24

Figure 4.7: Head movements after a prolongation

25

Figure 4.8: Prolongation and the movements of the eyebrows

26

4.4 Truncations (TRs)

Truncations are also called cut-off words or syllables, in other words, they are notfully executed or finished but they will be completed later, after an unfilled pause.Some truncations occur due to interlocutor interruptions but in this pilot study I haveobserved the truncations which speakers produced by themselves without any inter-ruptions from the interlocutor. Here, the total number of truncations was 17.

Table 4.4 introduces some utterances which include truncations that subjects pro-duced during the interview:

Table 4.4: Utterances containing truncations

Subject Utterance DurationSubject 1 ... den ligger liliTRlite utanför stan inte långt ... 1.668 ms

... it is located a liliTRlittle outside of the town not far ...Subject 1 ... det blir mycket att hon joTRjobbar mot ... 0.887 ms

... it becomes too much that she woTRworks against ...Subject 2 ... Ericsson som då haTRhade hållt på med telekommunikation ... 0.335 ms

... Ericsson which at that time waTRwas working with telecommunications ...Subject 2 ... jag fick ju ggTRgå alltså jag fick faktiskt gå på den här kursen ... 0.875 ms

... I got to ggTRgo as a matter of fact I got to take this couse ...

4.4.1 The Correlation of Truncations to the Head Movements

To find out whether there were any head movements related to the truncations, thelocalized truncation areas were studied and I discovered no specific relation betweenthem. The correlation of truncations to the head movements was quite similar to thecorrelation of unfilled pauses and prolongations to the head movements, the speakersfroze and held back their gestures when they got disfluent and then at the end ofthe truncations, head movements occurred. In other words, at the beginning of thetruncations speakers held still and at the end of the truncations and even after that,head movements occurred.

Figure 4.9 displays the head movements and a truncation for one of the speakersuttering the sentence “... den ligger liliTRlite utanför stan inte långt ...”, “... it islocated a liliTRlittle outside of the town not far ...”. As is shown in this figure, atthe beginning of the truncation speaker freezes her gestures but at the end of thetruncation quite strong head movements appear.

4.4.2 The Correlation of Truncations to the Movements of theEyebrows

Contrary to the results concerning the correlation between truncations and headmovements, there were strong eyebrow movements before truncations occurred, i.e.first, the movements of the eyebrows appeared and then the movements went downwith the truncation. These movements in relation to truncations can present the caseof focal accent, i.e. the speaker may signal a focal accent in a sentence. In this case,truncations, like unfilled pauses, can be totally dependent on the semantic content.

27

Figure 4.9: Truncation and head movements in three dimensions

28

Figure 4.10: The movements of the eyebrows before a truncation

29

Figure 4.11: Truncation after the movements of the eyebrows

Figure 4.10 illustrates strong movements of the eyebrows when one of the sub-jects uttered the sentence “... nej hal TR halvtid det jobbar hon bara ...”, “... no halTR half time she just works ...”. First, strong motions of the eyebrows occur whichmay signal a focal accent and then the movements end up with a truncation, which isdisplayed in the next figure (figure 4.11 ).

4.5 Conclusions and Future Development

To sum up, I have analyzed the correlation of some types of disfluency such as un-filled pauses, filled pauses, prolongations and truncations to the movements of thehead and eyebrow in spontaneous speech.

Unfilled pauses, prolongations and truncations did not seem to have a crucialconnection to the head movements. On the other hand, the speakers froze their ges-tures when they got disfluent and head movements occurred later on. In the case oftruncations, the head movements started at the end of the truncations and even after

30

that.Filled pauses were not either directly related to the head movements, but they

played a role as place-holders for the next word or to continue on, to the next sen-tence. Filled pauses were also employed to maintain control of a conversation whilethinking of what to say next.

Filled pauses and prolongations did not have any relation to the movements ofthe eyebrows.

However, unfilled pauses and truncations had indeed a significant meaning inthe relation with the movements of the eyebrows. There were strong movementsof the eyebrows before unfilled pauses or truncations occurred. The movements ofthe eyebrows in relation with unfilled pauses and truncations can be linked to theconcept of focal accent, i.e. the speaker may signal a focal accent in the conversationand in this case, unfilled pauses and truncations can be completely dependent on thesemantic content.

The experiment described here is done in a limited domain. The next step ofresearch in this field would be to employ several more subjects to perceive if it ispossible to generalize these results in spontaneous speech, especially concerning thecorrelation of unfilled pauses and truncations to the movements of the eyebrows.Another interesting step of research in this field can be also to retrieve other visualinformation, for instance eye movement: is there any simultaneous correlation ofdisfluency to eye movements?

31

Bibliography

Cassel, Justine. Verbal communication: Using approximate sound propagation todesign an inter-agents communication language, 2000.

Deese, James. Thought into speech. american scientist, 1978.

Eklund, Robert. Disfluency in Swedish human-human and human-machine travelbooking dialogues. Institute of Technology Linköping University, 2004.

Geschwind, N. Disconnexion syndromes in animals and man, brain, 1965.

Gloria J. Borden Katherine S. Harris, Lawrence J. Raphael. SPEECH SCIENCEPRIMER. Library of Congress Cataloging-in-Publication Data, 1980.

Goldman-Eisler, Frieda. Psycholinguistics: Experiments in spontaneous speech,1968.

Hans Peter Graf, Eric Cosatto Volker Storm Fu Jie Huang.Visual prosody: Facial movements accompanying speech.http://www.cstr.ed.ac.uk/downloads/publications/2002/paper.vtts.pdf, 2002.

Keith A. Johnson M.D., J. Alex Becker Ph.D. Atlas: The whole brain.http://www.med.harvard.edu/AANLIB/home.html, 2000.

Kelly, Spencer D. Does gesture play a special role in the brain’s processing of lan-guage?, 2001.

Munhall, Kevin G. Visual prosody and speech intelligibility, 2004.

Yule, George. THE STUDY OF LANGUAGE. Press Syndicate of the University ofCambridge, 1985.

32

A Appendix 1

When interviewer has taken part in the conversation, it has marked with an I. Filerelated to the subject number 3 is belong to another study related to the head move-ments which I have also employed in this thesis in order to obtain a wider analysis ofspontaneous speech.

A.1 Subject 1

A.1.1 file 1

Så *skratt, I: SÅ VI PRATAR OM FRITIDEN. Hum (UP) ju (PR) men det här väl sådå att min fritid (UP) går mycket då till den stugan em (FP) om (UP) jaha vad sa jagberätta mera? Jag kan berätta om mina barn. I: Ja. Eh (FP) dem är två stycken och eh(FP) min äldsta är ju (PR) tjugofy(TR)tjugofy ah hon fyllde tjugo fyra åring (UP)bor i malmö, (UP) I: OKEJ. Em (FP) går på högskolan där och har just fått lägenhet.I: AH BRA. Ah ja jätte bra. Hon eh (FP) hade tur för när hon började i (PR) Malmöså eh (FP) så hade dem byggt om ett stort sjukhus till student(UP)lägenheten eh(FP) men så hade hon fått den lägenhet dä(TR)där högst upp på det här (UP) husetoch där (UP) läckte det in vatten. I: JASÅ. Så att där har hon kört på nu när (UP)hon vill ha en annan lägenhet. Som (UP) nu har hon fått nytt i stan, den här (UP)lägenheten den ligger lili(TR)lite utanför stan inte långt eller utanför stan den (UP)låg inte mitt i centrum. I: VAD LÄSER HON? Men (PR) eh (FP) nu då så fick honen ny lägenhet det är ju med att (UP) det läckte lite då (UP) hon körde mycket på det.*skratt. I: LÄSER HON ALLTSÅ PLUGGAR HON? Ah precis. I: VAD PLUGGARHON? På eh eh (FP) emigration emigration och eh (FP) vad heter den kursen ellerprogram? eh (FP) den heter eh (FP) sve det var språk och emigration. I: SPRÅKOCH EMIGRATION? DET HÄR HAR JAG INTE HÖRT FÖRUT MEN DET ÄRLIKSOM SPÄNNANDE. Ah precis så att eh (FP) hon läser ju då eh (FP) det blirmycket att hon jo(TR)jobbar mot...

A.1.2 file 2

Eh det är det eh (FP) språket är ju mera eh (FP) det är ju inte just att du läser en-gelska eller så utan det är ju eh (FP) kommunikativ språk har dem hum. I: JA, DETHÄR LÅTER SPÄNNANDE. Hum, hon trivs jätte bra, nu som hon har läst till (PR)filkand då men hon ska fortsätta till magister. I: DU HAR JU ETT BARN TILL. Ahjag har ett barn till. Hon bor hemma (UP) och vet inte vad hon vill göra *skratt. Honär tju hon fyller tju ett nu. (UP) Em (FP) och hon lä(TR)läser eh (FP) på halvfarti norrköping (UP) på (PR) eh (FP) I: VAD LÄSER HON? religionsvetenskap. I:

33

KUL. och sen har hon fått en anställning på posten som (UP) brevbärare. I: I NOR-RKÖPING? Nej här i stan. Ja sen jobbar (PR) borta vid ”gekholmen”. (UP) *skratt.I: MEN HON BOR HEMMA. Hon bor hemma, hum, hon tycker att det är jätte brajätte bra. I: VISST. Sa du visst I: JA VISST VARFÖR INTE? *skratt. Ah ja så att jagvet inte men nu har jag sagt åt att hon måste tag i livet. (UP) Efter sommarn (UP) detgår inte å gå omkring å (PR) liksom inte vet vad man vill så (UP) så får vi se vad detblir med det ah. I: DET ÄR SVÅRT NU FÖRTIDEN FÖR UNGDOMAR. Ja det ärdet. (UP) hon haft jätte tur vet du som har (UP) hon jobbat eh (FP) jag tror att honjobbar mer än 50 procent på posten. I: VA BRA. Hum och få just att få en anställ-ning utan å annars tar dem i bara in där (UP) när det behövs så som extra men nuhar hon fått en anställning, så att det e det e jätte bra. I: H ELTIDSANSTÄLLNINGKANSKE. Nej hal(TR)halvtid det jobbar hon bara um (FP) men det är ju ah ...

A.1.3 file 3

Anställning i (UP) posten eh (FP) I: HON TRIVS PÅ JOBBET? Ja det är klart det ärbra. I: CYKLAR HON MYCKET? Cyklar mycket? Hum det gör hon mycket ramlarmycket omkull varje dag *skratt. Drar hon upp sina knäna. Antingen cyklar omkulleller annars har hon ramlat genom trapp. I: JAG HAR SETT ALLTSÅ DEM DETFINNS VÄL SMÅ BILAR LIKSOM SOM DE KÖR KANSKE DET E SMIDIGAREÅ HA EN SÅN I STÄLLET FÖR CYKEL. Ja precis. Hon (PR) jobbar ju mera (PR)i (PR) (UP) vad heter det i (PR) ”gekholmen” å (UP) ”berga” heter det (UP) så demycket såna hus då (PR) springer dem upp igenom trappan då får dem bara cykel.I: JA VISST. (UP) Okej (UP) ska vi prata om något annat. Ja (PR) I: OM JOBBET.Jag ska prata lite tydligare (UP). Om mat (UP) *skratt. Vi ska prata om mat. Jaha.*skratt. Eh (FP) I: ÄR DU, ÄTER DU KÖTT? Ja jag e kött, jag äter kött, däremotmin dotter då min äldsta dotter hon e vegetarian. Mycket svårt å jag trodde att detskulle gå över när hon var stor men det har inte gått över. Så hon äter aldrig kött.I: DET E SVÅRT ATT LAGA MAT, HITTA PÅ NÅGOT. Precis det e ju mycket(PR) vi köper som (UP) färdig korn (UP) såna du vet svampar ser ju ut som köttfärsegentligen som man köper fryst men det är svamp å så köper man dem i (PR) bitardet är också korn ...

A.1.4 file 4

Det finns i (PR) bland det vegitariska man måste krydda jätte mycket I: ANNARSSMAKAR DET INGENTING. Nej nej precis men som kött om eh (FP) köttfärs såstill exempel med korn då (UP) så är det (UP) helt okej (UP), men man få ta e lök åsvamp å så I: ÄTER NI SAMMA MAT MED HENNE? Ah nä(TR)när hon kommerhem så gör vi faktiskt det. I: ALLTSÅ NI, INGEN KLAGAR PÅ? *skratt Nej nejdet e jag tycker att det smakar ungefär som som kött. Ja, det gör det och just (UP)dem kött bi eller de bitarna är ju som kyckli(TR)kyckling i sig smakar ju heller inte såmycket om man tar bara kycki(TR)kyckling om man inte krydda. I: MEN HON VETATT HON MÅSTE VARA FÖRSIKTIG LIKSOM ATT TA PROTEIN BEHÖVERHON. Ah precis men det e något lite si å så med det. Jah det är det är sånt som manhar tjatat om (UP) mycket. Det finns många som ja det finns mer folk som vegetariskaoch sen finns det mera alternativ alltså mer eh att välja på. Ju det finns jätte det ärjätte stor skillnad hon började när (UP) det var väl mest att det var mode när honvar 15, 16 år. Hum, men nu är det egentligen eh oj nu kommer ju massa folk å tittar

34

massa student. I: GYMNASISTER FRÅN NYKÖPING DE HAR STUDIEBESÖK,DE BARA SER LIKSOM Å BERTIL KOMMER ATT BERÄTTA OM LABBEN.Okej. Ja. Ha. *skratt. I: JAG VET INTE HUR LÄNGE SKA VI PRATA ...

A.1.5 file 5

Vi ska fortsätta å prata ett tag lite. I: JA PRECIS VI SKULLE VARA HÄR OCHPRATA EN LITE STUND TILL. En lite stund var det faktia. *skratt. (UP) Vad skavi prata om igen? Ju vi kan prata om att jag ska till ”marocko” . Det ska jag göraom 14 dar. I: VA SPÄNNANDE. Så jag åker o(TR)om två vecker idag. Hum, eh(FP) I: JAG HAR ALDRIG VARIT DÄR MEN JAG HAR SETT PÅ TV. Ikväll detär i i packat och klart. Det e om (PR) ”marackesh” (UP) å det ska du åka. I: HURLÄNGE SKA DU STANNA DÄR? En vecka. I: ÄR DET ”MAROCKO” SOM DUSKA TILL ELLER VILKEN STAD? Ti ”agadir” ja (UP) så vi trodde att det skullebli sol och mycket värme (UP) så och vi tittar på karta varje dag min kompis (UP)vi är 4 styck eller vi är 8 stycken som åker. I: Å VA HÄRLIGT. Ah (UP) Och detär inte sol utan det är (PR) sexton grader och kallt och regn har det värt 14 darnu (UP). I: MEN DET KAN MAN INTE, DU SKA ÅKA OM TVÅ VECKER. Japrecis. Ah, men vi säger d(TR)det får bli det kolturella den här gången (UP). Vi borprecis vid havet I: OJOJOJ. Fast det e väl inte värmt men (PR) vi har som tempreradbas(TR)bassäng. I: MAN KAN HITTA PÅ. Ja. I: EN VECKA ÄR INTE SÅ LÅNG.Nej, precis. Nej. Egentligen så skulle man väl åka längre men. I: OM TVÅ VECKERBLIR DET ALLTS, VI NÄRMAR OSS VÅREN. Ja. I: 21 MARS ÄR DET, FÖRSTAVÅREN. Ja vår dag jämning ja. I: DET NÄRMAR SIG BÄTTRE SÅ DET HOPPAS.Vi får hoppas på det bästa. I: JAG HÅLLER TUMMARNA. Ja. *skratt.

A.2 Subject 2

A.2.1 file 1

Okej då kör vi igång här ifrån Ericsson igen som då ha(TR)hade hållt på medtelekommunikation. Jo, jag fick ju gg(TR)gå alltså jag fick faktiskt gå på den härkursen på arbetstid. (UP) Å det var ju inte, ja det var väl en eller två gånger i veckansom jag stack på eftermiddagen Och det var Gunnar Fant som höll kursen (UP) i hu-vudsak och Karlsson och Granström var (UP) assistenter och hade labbarna. (UP) Såatt eh (FP) och eh (FP) så hade han ju lite gästföreläsare också då Gunnar Fant. Menpå den tiden fanns ju inte syntes och igenkänning, utan (UP) det handlade om (UP)hur tillämpar man fyrpolteori eller hur allt sitt (UP) matematiska arsenal på att räknapå käften. (UP) Det var liksom det som (UP) kursen. Sen så var det då perceptionoch en av han som (UP) pratade om perception det var Å-G Möller och han var påfysiologen på Karolinska. (UP) Och det är rätt kul med den här killen, han han börjasom (UP) fonetiker (UP) och då tyckte han att han kunde för lite...

A.2.2 file 2

..så tyckte han kunde för lite om det medicinska, så nu är han medicinare (UP), ja*skratt*. Och sen så (PR) höll han på me med med och följde (UP) talet från (UP)det perifera hörselorganet alltså öronen och så upp till hjärnan. Och så mä(TR)mätte

35

signalen me(TR)med elektroder på vägen alltså små kapilär elektroder. Dom hadedom på katter. Man lyfte på locket till hjärnan och så körde ma in och så följde man(UP) hela vägen upp alltså. Och det var (UP) otroligt spännande tyckte jag, han var julika bra svänga upp ån eh (FP) ekvationer som aldrig. jag var otroligt imponerad avden där karln alltså. Så det var lite annorlunda på den tiden. eh, (FP) och jag kommerihåg att att jag missa eller vad säger jag jag missa tentamen i det här, eller jag kördeinte, jag missa den. Därför att (UP) jag fick problem med mitt öra.

A.2.3 file 3

..öra. Och eh eh (FP) och så en dag, men sen när jag kom till Stockholm då försöktejag ringa, ja det var ju omöjligt att få nån tid för att kolla mitt öra ju. (UP) Och senså vaknade jag en natt och då var det blod på hela kudden. (UP) Och då eh (FP) dåringde jag (UP) sjukvårdsupplysningen och fick ju naturligtvis ett annat nummer till(PR) eh (FP) ja. Och då kommer jag fram till en tant då som sa: (UP) jaha, har niförsökt att sätta bomull i öronen? *skratt* Det blödde ju ut. (UP) Så iallafall så fickjag en tid att komma direkt på morgonen alltså klockan åtta. Och eh (FP) den härtanten, eller vad säger jag den här sköterskan då som när jag kom dit där så var detfullt i (PR) väntrummet på öronkliniken där ju så sa hon: (UP) Dom människornadär, dom har (UP) beställt tid, du har inte beställt tid du får vänta hela dan *skratt*och så fick jag, så fick jag.

A.2.4 file 4

Det roligaste var då, då tittade den här läkaren: va fan har hänt med det här örat ärdet eh (FP) är det eh (FP), ja vi får nog sätta in pencillin, ja det rann ju inte blod dåva. Vi får nog sätta in pencillin för det kanske är någon infektion och så vidare. Ochvi ska nog, eh (FP) vi får sätta in pencillin så får vi se hur det blir och sen så ska vibeställa tid till eh (FP) överläkaren. (UP) Och så fick jag då komma till överläkaren,(UP) Schiratski, som då (PR) titta på det här: såna här öraon kan man inte hålla påmed och gå med. (UP) Det här måste vi operera. Han var säkert, han höll på medsin doktorsavhandling *skratt* och eh (FP) då eh (FP) så att eh (FP) men jag kundeju själv bestämma ändå va. men det var detta som gjorde att jag kunde inte gå upp itentamen i talöverföring. Så jag jag fick upp på (UP) omtentan, så vi var två styckensom tenterade talöverföring.

A.2.5 file 5

...på det här. Och det är ett stort steg när man jobbar på industrin faktiskt. (UP) Så dåeh (FP) gick jag och fundera och sånt eh (FP) om man kunde gå andra kurser och detvisade sig att man kunde, läste man (UP) 40 poäng i alltså (PR) eh (FP) språkdelen ifonetik så fick man fem doktorandpoäng. (UP) Och då tänkte jag att då kan jag börjadär va. Och eh (FP) och då var det så va eh (FP) efter operationen, jag fick ju opereraett öra då jo. ja det är ju också en speciell seans. Han ville jag skulle operera, det härmåste man operera och så vidare och rekonstruera och greja va. Och sp kom ju då(PR) dan innan dom opererar så kollar dom ju (PR) eh (FP) att man kan överlevaoperationen. Lyssnar på hjärtat och allt sånt där och tittar dom i örat och då säger dåden här underläkaren: varför ska vi operera det här, det ser ju fint konserverat ut (UP)och då säger överläkaren: jag känner fallet sen tidigare.

36

A.2.6 file 6

Och det var ju faktiskt så , det är så snyggt gjort att (UP) vanliga eh (FP) läkare trorju inte att det är opererat. Alltså han har ju ändå (UP) bytt eh (FP) hörselben ochhela skiten alltså. Ah (FP) ah (FP) så det ja eh (FP) och det var ju (UP) precis i denskarv då man börja göra detta, tidigare så gjorde man bara en radikal håla. Så att detinget skulle kunna bli infektioner, alltså ing. alltså ingen trumhinna. Bara låt liksomdet läka ihop som ett hål in va. (UP) Men här ser det ut som med en riktig trumhinnaoch så var där en stigbygelplattan var kvar och så var det eh (FP) (UP) hammarenvar borta. Städet fanns kvar å hängde och dingla. Så tog han och satte städet (UP)direkt mot trumhinnan och så mot stigbygelplattan och så fixerade det här. (UP) i allafall och då när jag var konvalecent då tänkte jag ja jag mådde ju jättebra jag varjusjukskriven en hel månad, då gick jag på fonetik (UP) i Stockholm.

A.2.7 file 7

...Som man gjorde på den tiden. Utan att anmäla sig utan att anmäla, sa han då att(UP) sa han: ja dom håller på (PR) tentera idag, (UP) första kursen. jaha sa ja, när ärdet omtentamen? ja, det var vekcan efter. Ja då anmäler jag mig till den, sa jag. (UP)Och så gick jag upp och skrev omtentamen, alltså den gången (UP) eh (FP) ja. Ochså började jag. Så jag läste fu(TR)full hastighet i fonetik (UP) vid sidan om jobbetså att säga. (UP) Men på den tiden kunde man läsa på kvällstid. (UP) Dom hade allakurser på, alltså många ämnen hade dom parallellt hela kursen på kvällstid. (UP) Ochdom hade då både halvfart och helfart. (UP) I: VAR DU DOKTORAND DÅ? Nejnej, jag var tvungen att snickra ihop min filkand ju. Annars så annars så skulle jag blidoktorand på fonetik då hade man vart tvungen ta dispans, börja läst nåt annat. Men(PR) utan jag hade min (PR) ryska va.

A.3 Subject 3

A.3.1 file 1

I: DU KAN JU.. Nu ska vi prata på *skratt*, jag kan berätta om (PR) eh (FP) när viåkte till Österrike här nu senast i (PR) eh (FP) vecka sex. En resa som jag i vanligafall ordnar lite senare under våren. eh (FP) Vi hyr in en buss och så åker vi ner ettgäng, förhoppningsvis upp till 30-40 personer, kompisar, släktingar såna saker. Nueh (FP) valde vi att lägga det lite tidigare på året för att det har varit lite dåligt medsnö sådär sent i april. Så att för att få lite bättre snö nere i dalen la vi det tidigare. Viblev inte riktigt så många som vi räknat med det här året, utan vi fick åka tåg iställetfrån Linköping. I: GAMLA BAKANTA ELLER? Ja jamensan, nej men det är väldigtmycket kompisar, vi försöker samla ihop, det är jag och en kompis som ordnar dethär eh (FP). Eftersom vi var så få så bokade vi ingen egen buss utan vi körde tåg dåner till Malmö. Det är väl dina hemtrakter va skulle jag gissa. I: HELSINGBORG. jai närheten *skratt*. Nej men Och i Malmö så bytte vi då till och hoppade på en busssom gick ner reguljärt istället ner till (UP) Sel Am See, där vi bor.

37

A.3.2 file 2

Ja (PR) nu var vi bara tolv så då var det inte så. Men (UP) i vanliga fall hyr vi in helaja, det stämmer. eh (FP) Så kom vi ner och det var en (PR) solig och vacker dag. enfantastisk fin dag, vi kom ner på eftermiddagen. Och då visade det sig att det var baraden andra fina dagen man haft på flera veckor egentligen. Så (UP) det kom lagom tillvi kom ner (UP). Och sen åkte vi skidor dan efter då och det va precis lika fint väderbåde den dan och även (UP) dom samtliga kommande dagar faktiskt som följde.(UP) så att vi varierade oss så mycket som möjligt i dom olika (UP) skidområdenasom finns där nere. det finns ju ganska mycket i Sell Am See trakten. Har du vart därsjälv? I: NEJ ALDRIG. Aldrig. Har du åkt skidor da? (UP) I: JA DET HAR GJORT.mm I: MEN DET ÄR MÅNGA ÅR SEDAN. Nej men då får du testa det här, för detär en riktig höjdare. Svenska fjällen är inte helt fel. Nej men här nere så finns det jumånga olika områden, det finns Sell Am See, du har (UP) Kitsbühl, det ligger barabara (UP) ett par mil därifrån, ja kanske bara en mil. Det är ju en klassisk..

A.3.3 file 3

Saalbach där har till exempel har varit VM-91 är, en jättetrevlig ort att var i och åkaskidor. Saalbach hinterglenn. MarieAlm är ett stort område som också ligger inom(PR) eh (FP) väldigt nära avstånd från detta. Och så finns det en glaciär som heter(UP) Kithssteinhohl som ligger ovanför den lilla byn Kaprun som alltid är snösäkert.Väldigt trevligt område. Så att eh (FP) vi varierade oss ganska så bra där nere. Och(PR) höll oss till dom här olika områdena, å trivdes bra med det. Badderstein liggerockså där i härheten. Och där var vi också. I: FINNS DET NÅN BRA BAR? va sadu? I: FINNS DET NÅN BRA BAR OCKSÅ? Nån bra bar. ja, det brukar många*skratt* det brukar va många som efterfrågar det, nej men vi vi fokuserar faktisktpå skidåkningen. Det är skidåkningne som är det viktigaste i det hela och (UP) vihoppas att eh (FP) dom som följer med är riktiga skidentusiaster också. Att dom eh(FP) delar det stora intresset som vi har för den biten av det hela också. Se skadardet aldrig med en öl heller efter en (PR) eh (FP) hel dags skidåkning. Utan det blirganska bra. Men det är också, det finns ju också väldigt mycket olika goda sorterockså.

A.3.4 file 4

Det var vecka sex. Det var en (PR) fantastisk vecka. eh (FP) Härlig skidåkning. å(PR) I:LÅNGT FÖRE SPORTLOVET. Ja, ett par veckor före sportlovet blev det. Vihade faktiskt en kille med (UP) som (PR) ska åka vasaloppet här nu i helgen somkommer. Så att han hade me sig längdskidorna också då. Gav sig upp på 3000 meter.Vid ett ett höghöjdsspår som ligger där uppe. Och föresatte sig att han skulle åka 50varv i den här slingan som gick, det blev väl 8 till slut men å andra sidan var slinganbetydligt längre än vad han trodde så att han klarade av en hel del mil däruppe. så vifår se här nu hur det går för honom nästa vecka. Å sen hann ja va hemma en (PR)fyra dar, sen åkte jag iväg igen. (UP) Till alperna till samma ställe faktiskt. En nyresa till (PR) eh (FP) precis samma hotell och allting sånt också. Med ett gäng nyamänniskor. (UP) Och det här var en rätt så anorlunda upplevelse den vekan. Det varungefär lika bra väder, vi åkte till ungefär samma områden men för första gången på

38

väldigt många år där nere så fick jag uppleva en riktigt allvarlig skada. Inte jag självutan en...

A.3.5 file 5

Vi hade väl (UP) eh (FP) tyckt att det var väldigt väldigt bra, förutsättningar förskidåkning den här dan så vi föresatte oss att vi skulle köra (UP) stenhårt. Och intevänta eller pausa på nån. Utan det var jag och den här killen som åkte upp och ner ien och samma backe som var väldigt bra just för tillfället och (PR) eh (FP) vi hadeväl precis hoppat av liften och gett oss av neråt. Å jag var före, han låg straxt eftermig. När jag kom ner till (PR) eh (FP) liftstationen å skulle åka upp så vände jagmig om bara för att kolla att han var med, men det var han inte utan 200 meter uppi backen var det två stycken åkare och skidor spridda överallt som låg runt där. I:HADE DOM KÖRT PÅ HAN? Dom hade krockat i en eh mycket kraftig kollision.För att (UP) dom åkte fort båda två (UP). Så jag åkte upp och kollade hur det var,han Per som han heter han låg och hade väldigt ont i knät (UP) visade det sig. Ochhan ville inte flytta på det och den andra killen var orörlig. Jag var orolig för att hanvar medvetslös först men han (PR) var (UP) vid medvetande men hade, visade detsig, brutit le. lårbenet.

39

B Appendix 2

40

Figure B.1: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... på arbetstid UP Å det var ju inte ...”

41

Figure B.2: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... mitt öra ju. (UP) Och sen så vaknade jag ...”

42

Figure B.3: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... på hela kudden (UP) Och då ...”

43

Figure B.4: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... så fick jag då komma till överläkaren UP Schiratski ...”

44

Figure B.5: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... fem doktorandpoäng (UP) Och då tänkte jag ...”

45

Figure B.6: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... tju ett nu (UP) eh ...”

46

Figure B.7: Unfilled pause and head movements in three dimensions, when one of the sub-jects uttered “... vad man vill så (UP) så får vi se vad det blir ...”

47

Figure B.8: Unfilled pause and the movements of the eyebrows, when one of the subjectsuttered “... högst upp på det här UP huset och där ...”

48

Figure B.9: Filled pause and head movements in three dimensions, when one of the subjectsuttered “... Och eh (FP) och så en dag ...”

49

Figure B.10: Filled pause and the movements of the eyebrows, when one of the subjectsuttered “... Eh (FP) går på högskolan ...”

50

Figure B.11: Prolongations and head movements in three dimensions, when one of the sub-jects uttered “... Det e om (PR) ”marackesh” ...”

51

Figure B.12: Prolongations and the movements of the eyebrows, when one of the subjectsuttered “... när hon började i (PR) Malmö så ...”

52

Figure B.13: Truncations and head movements in three dimensions, when one of the subjectsuttered “... den ligger lili TR lite utanför stan inte långt ...”

53

Figure B.14: Truncations and head movements in three dimensions, when one of the subjectsuttered “... nej hal TR halvtid det jobbar hon bara ...”

54

Figure B.15: Truncations and the movements of the eyebrows, when one of the subjectsuttered “... den ligger lili TR lite utanför stan inte långt ...”

55

Figure B.16: Truncations and the movements of the eyebrows, when one of the subjectsuttered “... ah nä(TR)när hon kommer hem ...”

56

Figure B.17: Truncations and the movements of the eyebrows, when one of the subjectsuttered “... är ju som kyckli(TR)kyckling i sig smakar ju heller ...”

57

AbstractContentsList of FiguresList of TablesPreface1 Introduction1.1 Aim1.2 Outline of the Thesis

2 Background2.1 Human Speech2.2 Cognitive Functions2.2.1 Broca's Area2.2.2 M1-Mouth Area2.2.3 Wernicke's Area2.2.4 Auditory Cortex2.2.5 Visual Cortex

2.3 Spontaneous Speech2.4 Conversational Interaction2.5 Disfluency2.5.1 Tongue Tips and Slips

2.6 Research Tools in Speech Science2.6.1 The Narrowband Spectrogram2.6.2 The Wideband Spectrogram

2.7 Language and Machines2.7.1 Talking Heads

3 Experimental Design3.1 Recording3.2 Measurements3.3 Apparatus3.3.1 Wavesurfer

3.4 Observations

4 Results of the Evaluation4.1 Unfilled Pauses (UPs)4.1.1 The Correlation of Unfilled Pauses to the Head Movements4.1.2 The Correlation of Unfilled Pauses to the Movements of the Eyebrows

4.2 Filled Pauses (FPs)4.2.1 The Correlation of Filled Pauses to the Head Movements4.2.2 The Correlation of Filled Pauses to the Movements of the Eyebrows

4.3 Prolongations (PRs)4.3.1 The Correlation of Prolongations to the Head Movements4.3.2 The Correlation of Prolongations to the Movements of the Eyebrows

4.4 Truncations (TRs)4.4.1 The Correlation of Truncations to the Head Movements4.4.2 The Correlation of Truncations to the Movements of the Eyebrows

4.5 Conclusions and Future Development

BibliographyA Appendix 1A.1 Subject 1A.1.1 file 1A.1.2 file 2A.1.3 file 3A.1.4 file 4A.1.5 file 5

A.2 Subject 2A.2.1 file 1A.2.2 file 2A.2.3 file 3A.2.4 file 4A.2.5 file 5A.2.6 file 6A.2.7 file 7

A.3 Subject 3A.3.1 file 1A.3.2 file 2A.3.3 file 3A.3.4 file 4A.3.5 file 5

B Appendix 2