breath as an embodied connection for performer-system collaborative interaction

1

Breath as an Embodied Connection for Performer-System

Collaborative Interaction

Greg Corness Thecla Schiphorst

Simon Fraser University

2

3

Overview Background + Literature Review

Initial Inspiration Key Interaction paradigms

The Ariel System Methodology

Studio Session (Performers improvising with Ariel) Interview technique

Procedures for Data Analysis The Three Primary Results Conclusion + Contribution

“The holy grail for me as a director is to produce a dramatic relationship between performer and

media, to grant media real agency, casting them in a role on par with the live performers” [Saltz, D.

“Live Media: Interactive Technology and Theatre”]

4

5

Background – Literature ReviewPerformer- System Interaction: the interaction

between a Performer and a Computer SystemPerformer-Performer Interaction: the interaction

between Two Human Performers

6

Background (Performer-System)Traditional ‘Sense- Respond’ paradigm tends to

focuses on after-the-event interaction.

Rowe, R. (1999). The Aesthetics of Interactive Music Systems.

Winkler, Todd. (1995). Making Motion Musical: Gesture Mapping Strategies for Interactive Computer Music.

7

Background (Performer-System)Traditional Sense- Respond paradigm tends to

focuses on After-the-Event interaction.Multi-modal base interaction considers the

system’s ability to track and anticipate the performer by sensing more channels of information.

Camurri, Antonio et. al. EyesWeb: Toward Gesture and Affect Recognition in

Interactive Dance and Music Systems.

8

Background (Performer-System)Traditional Sense- Respond paradigm tends to

focuses on After-the-Event interactionMulti-modal base interaction considers the

system’s ability to track and anticipate the performer by sensing more channels of information

The performer’s ability to anticipate the system is rarely discussed.

9

Background (Performer-Performer)Performance practiceResearch in neuro-scienceSocial Cues in Embodied Conversational Agent

10

Background (Performer-Performer)Performance practice

Agency, Trust, Collaboration, Negotiation Intuition related to Intention

Research in neuro-scienceSocial Cues in Embodied Conversational Agent

Lockford, L., & Pelias, R. J. (2004). Bodily Poeticizing in Theatrical Improvisation: a Typology of Performative Knowledge.

Bogart, A., & Landau, T. (2005). The Viewpoints Book: a Practical Guide to Viewpoints and Composition

11

Background (Performer-Performer)Performance practiceResearch in neuro-science

Mirror neurons Cognitive latency

Social Cues in Embodied Conversational Agent

Gallese, V. et. al. (2007). Intentional Attunement: mirror neurons and the neural underpinnings of interpersonal relations.

Wagner, D. M. (2003). The Illusion of Conscious Will

12

Background (Performer-Performer)Performance practiceResearch in neuro-scienceSocial Cues in Embodied Conversational Agent

Generating facial expressions Adding Breath Sounds

Foster, M. E. (2007). Enhancing Human-Computer Interaction with Embodied Conversational Agents

Whalen, D. H. (1995). The Effect of Breath Sounds on the Perception of Synthetic Speech

How can the interaction between a movement based performer and an autonomous generative

music system use breath to leverage the performer’s sense of intuition?

13

14

Click icon to add picture

Research Design

-Performers invited to improvise with the system over several sessions.-System designed to act as an autonomous agent generating music in the improvisational sessions. -System provided two basic conditions 1) presenting a preparation breath before it acts, 2) act without any breath.

15

The Ariel SystemDesigned to be

responsive

Designed to be Autonomous

Incorporates a model of embodied/social cues for projecting intention.

16

The Ariel SystemDesigned to be

responsive

Designed to be Autonomous

Incorporates a model of embodied/social cues for projecting intention.

Responds to performer’s actions through a Computer Vision Tracking Responding to motion Responding to shapes

17

Shape tracking Motion tracking

18

The Ariel SystemDesigned to be

responsive

Designed to be Autonomous

Incorporates a model of embodied/social cues for projecting intention.

Responds to performer’s actions through a Computer Vision Tracking Responding to motion Responding to shapes

Combine external and internal influences on the system’s response using weighted random processes

19

20

The Ariel SystemDesigned to be

responsive

Designed to be Autonomous

Incorporates a model of embodied/social cues for projecting intention.

Responds to performer’s actions through a Computer Vision Tracking Responding to motion Responding to shapes

Combine external and internal influences on the system’s response using weighted random processes

Incorporated a simulated breathing behaviour

21

Basis for modeling relationship between the Preparation Breath and following gesture

BreathBite Tail

Phrase

Loudness

22

The Ariel SystemThe system was designed and tested over three pilot

studies.

“It feels like ... It feels musical, as in…what feels good...it feels like it makes sense” (with breath)

“… it feels understandable ‘you know why there was silence” (with breath)

“… it felt like more background to me this time…like …it was more like a track of music playing..” (without breath)

“…(it) felt more chaotic” (without breath)

23

Studio SessionsSystem switched between presenting/not

presenting breath to provided comparison experiences for performers.

Structured improvisations guided performers to avoiding habitual approaches in interacting with a computer system.

Interviewing technique facilitating thick description of experience

24

Interview TechniqueBased on Interview of Explicitation

(Petitmengin)Re-living in an “inner way”Guiding performer’s attention of the experiencePutting into words

Petitmengin, C. (2006). Describing one’s subjective experience in the second Person:

an Interview Method for the Science of Consciousness.

25

Interview TechniqueRe-living in an “inner way”

Metaphor of video to encourage reflection Focus on selected Sensory Triggers found by

performer Sensitive to Body and verbal cues of Re-living

Guiding performer’s attention of the experience

Putting into words

26

Interview TechniqueRe-living in an “inner way”Guiding performer’s attention of the

experience Stabilizing Attention Move from representation to experience using how

questions Directing attention to dimensions of experience

Putting into words

27

Interview TechniqueRe-living in an “inner way”

Guiding performer’s attention of the experience

Putting into words Open use of language and gestures

28

Data AnalysisSeries of procedures

Close reading Open coding (developing themes) Construction of models

synchronic (independent of time) diachronic (time dependent)

Construction of a General model

Petitmengin, C. (2006). Describing one’s subjective experience in the second Person: an Interview Method for the Science of Consciousness.

29

Affordance from the Simulated Breath

Related observations of experience

1 Perceiving System Intention Conscious perception of a signalEmbodied understanding of gesture quality

2 Coordinating with quality of action

Increased ability to connect with new and complex material

3 Understanding and connecting with others.

Attention to presence and social expectations

30

Close Reading - (Perceiving Intention)The simulated breath is perceived as a cue The volume of the breath affects the

performer’s perception of the system’s intention.

31

Perception of the breath as a cue included: A conscious acknowledgment of the breath as a signal An embodied understanding of the upcoming gesture

“I find it was easier to expect. Although it (the music) was still really fast and sporadic at least you get a sense of ‘ it’s coming…. it’s coming… it’s coming” (system with breath).

Close Reading - (Perceiving Intention)

32

The volume of the breath affected the perception of the system’s intention as unnatural or demanding.

“When the breath is loud it sounds like an action being taken, but when it is soft it sounds like a physical cue as if the flute is in the room and getting ready to act.”.  

“ … so it was no-longer a command or anything like that, it was like a cue… so it cues you that the music is starting but it doesn’t impede in a way…it doesn’t command you to follow…”  

Close Reading - (Perceiving Intention)

33

Affordance from the Simulated Breath

Related observations of experience

1 Perceiving System Intention Conscious perception of a signalEmbodied understanding of gesture quality

2 Coordinating with quality of action

Increased ability to connect with new and complex material

3 Understanding and connecting with others.

Attention to presence and social expectations

34

Open Coding – (Quality of Action)Separating Participant’s CommentsPerformer-Performer communication

Belief/ AssumptionsDescription of experience

Approaches to Improvisation

Belief/AssumptionsDescription of experience

Experience with the Ariel System

Belief/ AssumptionsDescription of experience

35

36

“we were starting to just take silences

“it felt wrong to make big motions on complete silence

“breath was the safest time to move because you know the music is going to start

“there was more freedom with no breath…”

37

Open Coding - (Quality of Action)

38

Model of Expected Reciprocation“… it felt like more background to me this time…like …it was more like a track of music playing..” ( no Breath)

“I was surprised at how responsive…it felt like there was somebody…like give and take” (full Breath)

39

Model of Expected Reciprocation“the flute breaths and we say go ahead and sometimes it doesn’t and we are like ‘you let us down’ (with breath)

“I found I was responding to multiple things at once so picking up elements of what M and S and the music were doing. I also felt I could pay attention to effort and energy rather then precise movement” (with breath)

40

Micro and Macro-actions Macro-actions refers to large actions undertaken

with conscious intentionality . micro-actions refer to small actions performed

with little awareness or conscious intention Addresses the the affect of performer’s

attention Perceiving the system as one of the partners Social and artistic expectation

41

Stages in Performers’ Dynamic knowledge

Pre-improvisation Own expectations, training affecting interpretation of rules, tasks and environment

Start of Improvisation Sense presence, attention, in the room. Assigning of roles of elements still affected by previous experience

Reflection Perceive actions and states of self and othersPerceiving connection/meaning with actions

Construct New Knowledge

Develop variations of significant gesturesDevelop new connection with performersShift attention to perceived relevant sensations

Test Act in the moment based on new knowledge

42

Pre-Knowledge/

Start of Improvisati

on

Reflection

New Knowledg

e

Test

43

Stages in Performers’ Dynamic knowledge

Reflection Perceive actions and states of self and othersPerceiving connection/meaning with actions

Construct New Knowledge

Develop variations of significant gesturesDevelop new connection with performersShift attention to perceived relevant sensations

“…you can anticipate what is coming next-even though it wasn’t patterned.” (with breath)  “My body know the length of the gesture by learning the breathing of the system”

44

Affordance from the Simulated Breath

Related observations of experience

1 Perceiving System Intention Conscious perception of a signalEmbodied understanding of gesture quality

2 Coordinating with quality of action

Increased ability to connect with new and complex material

3 Understanding and connecting with others.

Attention to presence and social expectations

45

46

47

48

49

Conclusion

How can the interaction between a movement based performer and an autonomous generative

music system use breath to leverage the performer’s sense of intuition?

50

51

Conclusion (Contributions)Designed Implemented and tested an

interactive autonomous system which uses breath as a social cue to leverage performer’s intuition.

Developed a model of interaction between Performers and autonomous computer system that addresses social cues for intuition and anticipation.

Developed a Methodology for accessing, collecting and evaluating performer experience data in performance

52

Conclusion (Contributions)Designed, Implemented and tested an interactive

autonomous system which uses breath as a social cue Demonstrated the use/ possibilities of social cues as a

parameter in Performer-System interaction Demonstrated Breath as a mechanism for the

embodied knowing of the system’s intentions New Knowledge of interaction between Performers

and autonomous computer systemDeveloped a Methodology for accessing, collecting

and evaluating performer experience data in performance

53

Conclusion (Contributions) Implemented a model of social cues in interactive

performanceDeveloped a model of interaction between

Performers and autonomous computer system Illustrated the importance of social cues in the

experience of interaction Identified three primary affordances of a simulated

breath in relation to the performer’s experience interacting with a system

Developed a Methodology for accessing, collecting and evaluating performer experience data in performance

54

Conclusion (Contributions) Implemented a model of social cues in

interactive performanceNew Knowledge of interaction between

Performers and autonomous computer systemDeveloped a Methodology for accessing,

collecting and evaluating performer experience data introduces a new approach to studying interactive

performance paradigms. Addressing identified issues in collecting thick

description data on performer’s experience

Conclusion (Limitations)

Current model for breathing behaviour quite course

Model acknowledges only a single social cueProviding time for getting to know the system

55

56

Conclusion (Future Work) Improving the breathing model Incorporation intuition into the Ariel system Investigating other body intention cues such as

vision and kinaesthetic Explore the effect of attention and association

Thank you

57