to beat or not to beat? beat gestures in direction giving chris brandhorst & mariët theune...
TRANSCRIPT
To Beat or Not To Beat?
Beat Gestures in Direction Giving
Chris Brandhorst & Mariët TheuneUniversity of Twente
2Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Overview
• Beats and other gesture types• Research context: The Virtual Guide• A small direction giving corpus• How to recognize beats? The Beat Filter• When are beats used? Concept categories• A (very) simple beat usage model• Conclusions and future work
3Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Beats and other gestures
Gesture types• Deictic: pointing at an object’s location• Iconic: representing the shape of a concrete object• Metaphoric: depicting an abstract object using metaphor• Beat: indicating discourse structure; emphasis
McNeill (1992) “Hand and Mind” p.93: beats made up 44.7% of used gestures in a cartoon narration corpus.
4Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Context: The Virtual Guide
• An embodied direction giving agent in a 3D environment
5Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Video
(link naar filmpje hier)
Try out the Virtual Guide “live” here:
http://wwwhome.ewi.utwente.nl/~hofs/dialogue/
6Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Gesture generation
• Keyword-based• Turns (“left”, “right”, etc.): fixed pointing
gesture in turn direction• Objects (“the coffee counter”, etc.):
– pointing gesture to absolute 3D object location (gesture is computed dynamically)
– pointing gesture to relative object location, from viewpoint along the route (fixed gesture; like Turns)
– iconic gesture reflecting object shape (fixed gesture from “gestionary”)
7Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
When to use beat gestures?
• In the BEAT system (Cassell et al., 2001) gestures are used to mark new information and to contrast items
• Beats are given low priority: they are only used when no other gesture type is available
• In the Virtual Guide, in almost all cases a pointing or iconic gesture is available – so, no “need” for beats?
• No: human direction givers do often use beat gestures, as shown in our small video corpus.
8Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Direction giving video corpus
Used for this study: • 15 short video clips (± 45 sec. each)• 4 different Dutch speakers, 3-4 clips each• 2 different destinations in our building• 2 versions of each clip (except one): with a listener
present or to the camera• 133 gestures;
124 annotated
(others not
clearly visible)
9Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
How to recognize beats?
Beat characteristics: • “A simple flick of the hand”• Short and quick• Only 2 gesture phases: preparation and retraction (no
stroke)• No “tensed stasis”• Formless hand shape
Formal coding, based on shape only: the Beat Filter (McNeill, 1992) “filters out” beats from other gestures.
10Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
The Beat Filter
1. Does the gesture have other than 2 movement phases?2. How often does tensed stasis or finger movement
appear?3. If the first movement is in non-center space, is any other
movement in center space?4. If there are exactly 2 movement phases, are they in
different spaces?
• Add 1 point for each “yes” answer to Questions 1, 3, 4 to the number given in answer to Question 2.
• The lower the score, the more likely the gesture is a beat.
11Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Using the Beat Filter (1)
• 109 gestures were scored with the Beat Filter*• 95 of those were annotated by two annotators (the other
14 were used as test items)• Annotator agreement on the Beat Filter questions was
very low:– Question 1: K = 0.43– Question 2: K = 0.31– Question 3: K = 0.18– Question 4: answer is dependent on Q1, so computing reliability
makes no sense
*15 gestures were considered to be “obvious” (other gesture types than beats) and not “filtered” by annotator A …!
12Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Using the Beat Filter (2)
Agreement on total Beat Filter scores: • 44.2% same score (but possibly on different grounds!)• 36.8% difference of 1• 16.8% difference of 2• 2.1% difference of 3
In the end, only the scores of annotator A were used.
13Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Annotating gesture types
• Gesture types based on global shape information: resemblance to mentioned object, finger pointing, directional component, etc. (in combination with speech)
Annotator agreement:• Agreed on 83.3% of gesture types (102 of 124), K=0.73
– Of these, 33.3% are beats (34 of 102)
• Disagreed on 17.7% of gesture types (22 of 124)– Most confused were point and iconic (45.5%)– Next most confused were beat and point (13.6%)
14Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Gesture types and beat score
• Beats do have lower Beat Filter scores• Many pointing gestures have low scores too
*NF (Not Filtered) gesture types were not entirely obvious after all…!
0 1 2 3 4 5+ NF* # %
beat 3 30 1 34 27.4
‘multi-beat’ 4 4 3.2
iconic 4 2 2 2 10 8.1
point 14 7 12 8 3 10 54 43.5
not agreed 1 3 5 7 2 1 3 22 17.7
# 4 47 17 25 12 4 15 124 100
15Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
When are beat gestures used?
Some direction giving concept categories were defined:• Directions (up, down, left, right, …)• (Other) Spatial information (through, in, on, at, across, …)• Duration & Timing (all the way, continue, immediately, …)• Landmarks
– Nouns (windows, a square, the hallway, …)– Pronouns (that, the same, this, they, it, …)
• Points in Time or Space (now, then, here, there, …)• Hesitations (uh, uhm, I would say, something like that,
maybe, …)
16Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Concepts / gestures overview
Concept # Beat (%) Other (%) Total (%)
Spatial info 45 6 (13.3%) 3 (6.7%) 9 (20.0%)
Hesitations 95 9 (9.5%) 4 (4.2%) 13 (13.7%)
Duration & timing 37 3 (8.1%) 1 (2.7%) 4 (10.8%)
Landmarks 176 7 (4.0%) 30 (17.0%) 37 (21.0%)
pronoun 49 3 (6.1%) 6 (12.2%) 9 (18.3%)
noun 127 4 (3.1%) 25 (19.7%) 29 (22.8%)
Points in time/space 102 4 (3.9%) 12 (11.8%) 16 (15.7%)
Directions 84 1 (1.2%) 39 (46.4%) 40 (47.6%)
Total 539 30 (5.6%) 89 (16.5%) 119 (22.1%)
Other n/a 4 (n/a) 10 (n/a) 14 (n/a)
17Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Concept Categories
Relative frequency of concept categories:• Landmarks are most frequently mentioned• Directions are only in fourth place
32,7
18,9 17,615,6
8,3 6,9
0
5
10
15
20
25
30
35
Concept category
%
Landmarks
Points in Time or Space
Hesitations
Directions
Spatial Information
Duration & Timing
18Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Concepts and gestures
• Not fitting into these categories: 4 beats, 10 “other gestures”
0
20
40
60
80
100
SI H DT L PTS D
Concept Category
%Beat
Other gesture
No gesture
SI = Spatial Information; H = Hesitations; DT = Duration & Timing; L = Landmarks; PTS = Points in Time or Space; D = Directions
19Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Landmarks: pronoun or noun
• Landmarks as pronouns: fewer gestures, relatively more beats
3,1 6,1
19,712,2
77,281,7
0102030405060708090
Noun Pronoun
Landmarks
%Beat
Other gesture
No gesture
20Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
A simple beat usage model
The probability that a beat gesture B is generated to accompany an utterance u (and modelling speaker s):
P(B|u) = P(B|Cu) x ms
where• Cu is the concept category of u• P(B|Cu) is the probability of B accompanying Cu based
on corpus data• ms is an optional multiplier for speaker s (weight factor)
21Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Toward a better model
Other factors than just corpus frequency should be taken into account. For example:
• First or second time the same directions are given?• Listener present or not?• Context: influence of preceding and following concepts /
gestures• Etc.
And of course, more (and more reliable!) corpus data are needed.
22Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Conclusions
How can we recognize beats?• Applying Beat Filter to recognize beat gestures may not
give reliable results• “Impressionistic” gesture type annotation was more reliable• Add “directionality” and “hand shape” to Beat Filter?
When are beats used?• “Other” gestures don’t always take precedence over beats• Beats mark spatial information, hesitations, duration and
timing more often than other gestures do
23Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
Future work
• More data• More reliable annotation• Investigate when beats or other gestures are used given
a concept category• Better / more general concept categories?
→ Implement in the Virtual Guide
24Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009
The End
Questions?