engaging spect-actors with multimodal digital puppetry [nordichi 2010]
TRANSCRIPT
Engaging Spect-actors with Multimodal Digital Puppetry
Céline Coutrix (presenter),
Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin,
Gabriela Richard, Lorenza Parisi, Stefano Roveda, Prayag Narula
Introduction
• (Digital) Puppetry
1
Introduction
• Field study
- Describe an example of how a computer can be used as a medium for engagement
- Understand why would this be a promising field for HCI applications
2
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
3
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
4
The system: Euclide
5
pupeteer audience, in front of puppet
The system: Euclide
Euclide system
6
The system: Euclide
• Has been successfully used for engagement in the Science museum in Naples, Italy
• Why? How could it be improved? What can we learn for future puppetry systems?
7
The system: Euclide
• Puppeteer
• Audience
8
The system: Puppeteer
pupeteer
puppet on distant screen
9
The system: Puppeteer
• Puppeteer
- Voice
- Face
- Body
· Global movement
· Local movement
- Scenery
10
The system: Puppeteer
• Voice
11
The system: Puppeteer
• Face
12
The system: Puppeteer
• Face
13
The system: Puppeteer
• Face
14
The system: Puppeteer
• Face
15
The system: Puppeteer
• Face
16
The system: Puppeteer
• Face
17
The system: Puppeteer
• Face
18
The system: Puppeteer
• Face
19
The system: Puppeteer
• Body: Global movement
20
The system: Puppeteer
• Body: Global movement
21
The system: Puppeteer
• Body: Global movement
22
The system: Puppeteer
• Body: Local movement
23
The system: Puppeteer
• Body: Local movement
24
The system: Puppeteer
• Body: Local movement
25
The system: Puppeteer
• Body
26
The system: Puppeteer
• Scenery
27
transparency
The system: Puppeteer
• Scenery
28
orientation
The system: Puppeteer
• Scenery
29
Zoom+Position
The system: Euclide
• Puppeteer
• Audience
30
The system: Euclide
audience, in front of puppet
distant pupeteer
31
The system: Euclide
• Compromise between
- Museum’s requirements for cheap, robust, generic and flexible hardware
- Puppeteers’ requirements for large number of functionalities
- Usability
· e.g. mouth control moved from index to thumb to avoid tiredness
32
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
33
Data Collection
• Video+audio recordings
- Over several months
- Four viewpoints
- Three different pupeteers
• Interview of head pupeteer
34
Data Analysis • Constant comparison analysis
• Top-down and Bottom-up approach
- Findings originates from both the data and our research questions
• Coding scheme built through three steps: open, axial, selective coding
• Clips selected together, then divided between coders
• Discussion and revision of coding until agreements > 0.6 35
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
36
Findings: Multimodal Puppetry
• Most used features are controlled through more accessible devices, due to design
37
Findings: Multimodal Puppetry
• Mostly animation of face,
• second comes body,
• last comes scenery
face body scenery
38
Findings: Multimodal Puppetry
• Important multimodal use
Simultaneous use of modalities
Single modality
39
Findings: Multimodal Puppetry
1. Speech ⇒ synchronization’s problem
2. Speech+lips
3. Lips
40
Findings: Multimodal Puppetry
• Lot of very short speeches and silences in order to maintain engagement
• Puppet active even when silent
Speech durations Silence durations
41
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
42
Findings: Emergence of Performative Structure
• Engaging the Audience Throughout the Sessions: Evidences
- People in the audience:
· Don’t talk to each other 99% of the time
· Don’t pay attention to the outside 98% of the time
· Talk to the puppet 65% of the time
· Pleasure and Arousal annotated as positive all the time
· Number increase, then decrease
43
Findings: Emergence of Performative Structure
• Rather short, but:
- Sessions interrupted by teachers
- Group of children continue interacting from the next puppet station
0 2 4 6 8
10
< 1 min 1 min < < 2 min 2 min < < 9 min > 9 min
Number and length of sessions (mins)
44
Findings: Emergence of Performative Structure
• Structure of sessions
- Approach: enter the interaction area; observe
- Testing: Try to interact to find out which actions have an effect
- Playing: Interact in an aware, active, involved way; climax
- Ending: attention diverted; leave
45
Findings: Emergence of Performative Structure
• Structure of sessions
- Approach, Testing, Playing, Ending
- Long playing phases, with scenery animations
46
Findings: Emergence of Performative Structure
47
Findings: Emergence of Performative Structure
• the audience starts interacting happily but calm,
• then gets excited,
• finishes the interaction happily and calm again
48
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
49
Findings: Construction of Engagement
• The Puppet’s Multimodal Resources: Effects (face, body and scenery animations), like natural gestures, in relation to speech
- Pure special effects
- Non-verbal turns
- Verbal accompaniements
- Virtual gesturing
50
Findings: Construction of Engagement
• Pure special effects
- Non verbal ressources are predominant modality for interacting with the audience
- Used to attract visitors that cannot be reached by speech (babies, far away visitors)
51
Findings: Construction of Engagement
• Non-verbal turns: Contribution in an interaction that includes speech, e.g.
- wearing accessory like glasses on special request from the audience and subsequently commenting on it
- Explaining a fact by displaying a picture of it
52
Findings: Construction of Engagement
• Verbal accompaniements: Non-verbal actions are accompagnied by speech, e.g.
- Acrobatic movement with onomatopeic sound
→ Communication relies on the nonverbal resource, which dictates its structure and meaning
53
Findings: Construction of Engagement
• Verbal accompaniements: Example
54
Puppet: Ho una molla I’ve got a spring Mi serve per saltare I need it to jump ⎡ Dong:: dong:: ⎤
Dong dong ⎣((Starts jumping))⎦
Findings: Construction of Engagement
• Virtual gesturing: priority is given to speech; visual features are used in synch with speech, following its structure and duration, e.g.
- Head movement, like pointing gesture
55
Findings: Construction of Engagement
• Virtual gesturing: Example
56
Visitor: Dove sei? Where are you?
Puppet: Sono⎡::((pause))⎤ I’m
⎣((Looking down))⎦ ⎡qui dentro,⎤
inside here ⎣((Looking around))⎦ non mi vedi? can’t you see me
Findings: Construction of Engagement
• Multimodal interaction for puppeteer
is essential
- Helps avoiding the low status of a conversational agent, e.g. verbal abuse
57
Findings: Construction of Engagement
• Response to verbal abuse: Example
58
Child: (Scemo) Dumb
Puppet: Vabbeh adesso basta. All right; let’s stop this
Child: ((turns back, surprised)) Puppet: Ogni volta che scemo, ogni volta che scemo.
All the times dumb, all the times dumb. E tu invece come sei? And what about you instead, what are you?
Child: ((goes away))
Findings: Construction of Engagement
• Talk = Main ressource,
Multimodal animation = instrument
• Combination attracts visitors effectively
59
Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
60
Conclusion
• Why is digital puppetry a promising application field for HCI?
- On the contrary to a fully computer-driven system
· Engagement stems from combination of multimodal ressources by puppeteer
· Users from the audience inspire the narrative by the puppeteer and feel like they have an active role
61
Conclusion
• Why is digital puppetry a promising application field for HCI?
- Performative Structures for Brief Interactions ⇒ repetitive element of an entertainment interface could be further taken into account
- Multimodal Use of the System by the Puppeteer ⇒ possibility to introduce advanced interface techniques
· Easing lip/speech sync,
· Mapping expressive hand gestures to puppet expressions,
· Etc.
62
Engaging Spect-actors with Multimodal Digital Puppetry