module u1: speech in the interface 6: human communication
DESCRIPTION
Module u1: Speech in the Interface 6: Human Communication. Jacques Terken. Motivation. provide knowledge about basic concepts of human communication relevant to the design of speech interfaces explore use of knowledge about human communication as model for man-machine interaction. Contents. - PowerPoint PPT PresentationTRANSCRIPT
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 1
Module u1: Speech in the Interface
6: Human Communication
Jacques Terken
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 2
Motivation
provide knowledge about basic concepts of human communication relevant to the design of speech interfaces
explore use of knowledge about human communication as model for man-machine interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 3
Contents
basic scheme for communication the structure of messages the structure of communication “how to do things with words”1: communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
1 J.L. Austin: “How to do things with words”, 1962
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 4
basic scheme
Context– Situational context situated communication: time, place, objects– Social context – Cultural context– Dialogue context
contextsender channel receiver
feedback
context
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 5
intentions and effects
Sender: intention to convey a message
message needs to be encoded Receiver: decoding effect
Ideal communication: intention = effect
Distortions underway: intention effect’
Furthermore: effect is more than intention (sender also communicates unintentional information)
complex messages require sequence of steps
communication as structured action / collaboration
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 6
Contents
basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 7
formal aspects (1)
Simple case: messages containing a single dialogue act (inform, request, confirmation etc)
Sentence/utterance– put the red block before the yellow block next to the ball– I herewith pronounce you man and wife– What time does the next flight to Boston leave?
Sentences made up of word groups Word groups made up of words Words made up of syllables and phonemes (~40) or
graphemes (~25) / @, e, a, r, … / ; { e, e, a, r, … }
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 8
formal aspects (2)
rules determine well-formedness of representations at different levels:– pragmatics: sentence meaning in context (speech act,
dialogue act) contextual meanings
*the king of France is bald (but may be okay in a story)– semantics: sentence meanings possible meanings
*the 50 year old bachelor surprised his wife– syntax: sentence structure possible sentences– morphology: word structure possible words– Dutch: loopje; *loopetje; kommetje; *kompje– Phonology: sound structure possible sound sequences
Dutch: herfst; Polish: przewalski
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 9
formal aspects (3)
mapping between levels not fully specified (not one - to - one):
ambiguity Mapping sounds onto words:
/Aiscri:m/ {I scream} or {Ice cream}
Mapping words onto sentence structure:
Syntactic ambiguities
i saw the girl with the binoculars
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 10
put the red block before the yellow block next to the ball
(1) put the red block
before the yellow block [ that is ] next to the ball
(2) put the red block before the yellow block,
next to the ball
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 11
Contents
basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 12
levels of analysis
topic … topic … topic
transaction
exchange exchange exchange
turn turn turn
move move move
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 13
transaction
(part of) conversation dealing with a single topic consists of one or more exchanges
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 14
exchange
smallest interactive unit consists minimally of two turns produced by two different
speakers involves the negotiation of a single piece of information examples:
request - inform - confirm
presentation - acceptance
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 15
adjacency pairs:
question answer
request accept / turn down
invitation accept / decline
greeting greeting
apologysmoother
Interactive pressure: opening move induces obligation to answer with closing move (conversational obligation)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 16
nesting of adjacency pairs:s: question X what time is the next train to amsterdam
a: question Y amsterdam central?s: answer Y yes
a: answer X in seven minutes at platform 5
mixed initiative vs. system initiative
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 17
turn
Turn: everything the speaker says before the next speaker takes over
Rules for turn-taking behaviour
1. Speaker selects next speaker
2. Next speaker selects self
Window of opportunity for turn switching (minimizes overlap and interruption; interruptions only permissible in particular circumstances
Turn consists of one or more moves
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 18
temporal aspects of turn-taking
dialog not as a sequence of well-formed non-overlapping turns:– time window for turn taking– overlap, interruptions
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 19
move/dialog act
no fixed inventory, but dependent on application Common types: Primary dialogue acts
– request for action – inform– question
Game theory conceives of communication as a sequence of movesExpresses the assumption that communication is governed by a set of rules
Dialogue control acts– acknowledge– check– confirm– greeting– thanks
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 20
Indirect communication
No one-to-one mapping between form and function: indirect communication (politeness forms)
Questions:– Can you tell me what time it is (yes/no question) – I would like to know what time it is (statement)– Do you have a watch (yes/no question with indirect
communication )
Direct communication:– What time is it (wh-question)– Please tell me the time (request)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 21
spoken dialog systems
Identify dialog act performed by user in order to decide what dialog act to produce as an an answer
Machine learning techniques in order to learn best continuation in context x (and to learn best path)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 22
Contents
basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 23
spoken communication as joint action/cooperation Speech is volatile Mechanisms to cope with volatile nature of speech co-operation
receiver must be in receiving mode, i.e., be open to message
sender must adjust to channel and receiver characteristics
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 24
Goal achievement is usually accomplished through a number of steps involving both participants: dialogue
Monitoring of transmission success by sender on the basis of conscious and unconscious feedback from receiver
Immediate action in case of problems either by user or sender
– Role of eye gaze in staying in touch, monitoring and turn taking
spoken communication as joint action/cooperation (2)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 25
speech generation
two simultaneous processes:– conceptualisation (“what to say”: content)– formulation (“how to say it”: converting content into
utterance)
simultaneous, but formulation lagging behind
( spontaneous speech phenomena)
– restarts, self-corrections, hesitations (silence, ehm), disfluencies
– grammar of spoken language
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 26
message formulation: issues
continuity, coherence: what we say usually links to what was said before:
context provides search space for interpretation and allows for efficient expression
– pronouns, anaphora – prosody– bridging: establishing a relation between expressions in
consecutive utterancescan you take the picnic supplies out of the van? the beer is probably warm, so please put it in the fridge.
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 27
grice’s cooperativity principle
human communication (supposed to be) governed by grice’s cooperativity principle, consisting of the following maxims
(guidelines for communication): quantity: make your contribution as informative as is required
I want a flight; I want a flight from A to B because …. quality: do not say what you believe is false or for which you
lack adequate evidence relation: be relevant manner: be perspicuous, brief and orderly; avoid obscurity
and ambiguity
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 28
speech comprehension
Stages in comprehension:– Perception: perceiving sounds and mapping sounds onto
words– Interpretation– Evaluation– From a processing point of view perception and
interpretation are closely interlinked
keywords: on-line processing, hypothesis testing and integrating
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 29
backchannelling
“utterances” and other communicative signals guiding the progress of the communication, i.e., serving a control function– nodding– yes, hm-hm– “madrid?”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 30
error recovery
on-line property allows fast and efficient recovery:– problems noticed immediately– cause of problem usually easily identified
“what did you say?”
“did you say …”
“what do you mean”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 31
Grounding
agreeing about current state of the dialogue
Important role of feedback Shared knowledge
+ knowledge that speaker knows is available to hearer
+ world knowledge (“living memory”)
+ knowledge concerning context of communication and what has been established by previous part of discourse
common ground
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 32
Contents
basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 33
human communication no neat succession of turns: overlap in time: provision for speech recognition needed
Indirect communication: message function and form no 1:1 relation
interpretation dependent on domain knowledge: requires domain modelling: successful for restricted domains
interpretation dependent on context: requires discourse model
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 34
supporting the interaction
Prompting: system initiative Zooming: increase amount of feedforward in system prompt
– How may I help you– Which connection– Are you interested in connections or prices– Please respond with yes or no
Cooperative replies– User: “Do you have flights between 7:15 and 7:45?”– S: “No, but there is a flight at 7:55”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 35
robustness issue
Lack of robustness creates need for verifcation
Explicit verification:– Did you say “amsterdam”?
Implicit verification– U: “I want to travel from .. to rotterdam”– S: “when do you want to travel to rotterdam?”
– Cf: S: “when do you want to travel to amsterdam?”
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 36
Verification is costly when you got it right Feedback (positive feedback or implicit verification) is
misleading when you good it wrong
Use verfication when uncertain Use feedback when confident
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 37
confidence-based verification
High confidence: just accept or implicit verification Medium confidence: explicit verification Low confidence: explicit request for repetition
Precision/recall issue for correctness and confidence measure:– Confidence measure may not be right: either too high or too
low
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 38
Contents
basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 39
communication as joint action (cooperation)– taking into consideration listener needs– listening for intention
rule-governed behaviour at many different levels– structure of messages– structure of conversations (e.g. turn-taking behaviour)
grounding feedback (back-channelling)
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 40
Contents
basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer
interaction
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 41
cooperative assistant
Characteristics of cooperative assistant– Interprets messages in terms of intentions not in terms of
form
(“c.a. is not a politician or lawyer”)– Cooperative replies
(not blunt “no” but offers alternatives, e.g. through constraint relaxation)
– Context-sensitive replies and help
No global help manual– Proactive, triggered by implicit cues
Does not await explicit request for help but anticipates
SAI User-System Interaction U1, Speech in the interface: 6. Human communication 42
discussion
Is this a good model for human computer interaction? Considerations:
– Schneiderman c.s.: user should stay in control
– Maes/Sidner: • computers should be able to understand our intentions
so that we don’t have to specify all the actions explicitly ourselves
• Computers should monitor our actions so that they can learn how to anticipate our needs and preferences