a brush with the real world: the future of inertial motion...
TRANSCRIPT
A Brush with the Real World: the future of inertial motion
capture in live performance
John Haag
Drama (Performance Studies)
Creative Industries Faculty
Queensland University of Technology
September 2009
This thesis has been submitted in fulfilment of the requirements for the award
of Masters of Arts (Research) in the Creative Industries Faculty at Queensland
University of Technology, 2009.
ii
Acknowledgements
This study has been made possible through the collaboration of cutting edge
equipment supplier VR Solutions who have been very generous in allowing
me significant time using their motion capture equipment. Michael Bosworth,
as director, was open to experimentation and was most helpful in facilitating
sessions at their studio. Jon Newell, lead developer at visualisation company
VR Space, provided many hours of his own time in ironing out problems with
the suit, guiding me through the complexities of the software and working with
me on numerous motion capture sessions particularly in the earlier stages.
During this time I also received timely assistance from QANTM graduates,
Kim Allen and Vana Prayitno, who helped with the initial modeling. Rosetta
Cook provided insight into the world of dance and, with Joe Lau and Bridget
Fiske, contributed many hours in the studio patiently improvising dance moves
as I tested the real-time environments. A Brush with the Real World was
performed as an improvisational installation at Judith Wright Centre of
Contemporary Art in Brisbane. It was a difficult undertaking and we were
provided with generous assistance from the administrative and technical staff,
along with the small dedicated team comprising Rosetta Cook, Miriam
Hanmer and Adrian Green. Academic research did not come naturally but my
supervisor, Paul Makeham, has been tremendously helpful in provided clarity
in designing the written component of my research as well as assistance in
structuring the research as a whole.
iii
Statement of Original Authorship
The work contained in this thesis has not been
previously submitted to meet requirements for an
award at this or any other higher education institution.
To the best of my knowledge and belief, the thesis
contains no material previously published or written by
another person except where due reference is made.
Signature
Date / /
iv
Table of Contents
Acknowledgements
Statement of Original Authorship
Abstract
Keywords
Note on the Accompanying Media
1 Preface
2 Contextual Review
2.1 Introduction
2.2 Motion Capture
2.2.1 Issues of terminology
2.2.2 From performance to screen
2.2.3 The actor
2.3 Performativity
2.3.1 Introduction
2.3.2 Spectacle
2.3.3 Reality
2.3.4 Presence
2.3.5 Liveness
2.3.6 Spontaneity
ii
iii
viii
ix
x
1
7
7
9
9
12
14
17
17
28
22
26
30
33
v
2.4 Motion capture in the arts
2.4.1 Motion capture in visual media
2.4.2 Digital Performance
2.5 Conclusion
3 Methodology
3.1 Research question
3.2 The nature of research
3.3 Postmodernism
3.4 Performative research
3.5 Practice-led research
3.6 Investigative framework
3.7 Methods
3.7.1 Case study
3.7.2 Creative practice
3.7.3 Industry input
3.7.4 Applied methods
3.8 The working environment
4 Description of practice
4.1 Overview
4.2 Case 1: A Brush with the Real World
4.3 Case 2: Chasing Shadows
4.4 Case 3: Private Eyes
4.5 Case 4: Motionics
4.6 Log of practice
35
35
37
42
45
45
46
47
52
54
56
59
59
64
67
70
72
74
74
76
80
82
84
87
vi
5 Evaluation
5.1 Inertial motion capture
5.2 The Case study
5.2.1 Case 1: A Brush with the Real World
5.2.2 Case 2: Private Eyes
5.2.3 Case 3:Motionics
5.2.4 Case 4: Chasing Shadows
5.3 Observations of the performer
5.4 General findings
5.5 Implications for live performance
5.6 Summary
5.7 The future of motion capture
6 Conclusion
7 References
Diagrams
Figure 1: VRS Studio (2005)
Figure 2: The dancer’s movements are captured and applied
Figure 3: Using rear projected animation in public windows
Figure 4: Large-scale shadow puppetry using animation
Figure 5: Using animated effects triggered by movement
Figure 6: Projected background moves with actor’s movements
Figure 7: Projection within sets onstage
Figure 8: Interactive animated shadow
Figure 9: Shadow projection on the tent wall in circus
Figure 10: Improvisational screen character
Figure 11: A Brush with the Real World
Figure 12: The layout for A Brush with the Real World
Figure 13: Chasing Shadows
Figure 14: Private Eyes
91
91
95
96
100
101
104
105
107
109
113
113
120
124
1
13
68
68
68
69
69
69
70
70
76
77
80
82
vii
Figure 15: Motionics 1
Figure 16: Motionics 2
Figure 17: Motionics 3
Figure 18: Motionics 4
Figure 19: Trialing the paintbrush
Figure 20: Early relations constraint for activating the colour orange
Figure 21: Reactive background
Figure 22: Eyes follow the performer
Figure 23: Moving through a forest of eyes
Figure 24: Triggering effects
Figure 25: Using video
Figure 26: The paint colour changes universally
Charts
Chart 1: Attributes of inertial suit
Chart 2: Limitations and problems
84
84
84
85
96
99
101
102
102
102
102
103
93
93
viii
Abstract
The history of art is a series of successful transgressions.
(Sontag cited in Kramer, 1999: 9)
3D Motion capture is a medium that plots motion, typically human motion,
converting it into a form that can be represented digitally. It is a fast evolving
field and recent inertial technology may provide new artistic possibilities for its
use in live performance. Although not often used in this context, motion
capture has a combination of attributes that can provide unique forms of
collaboration with performance arts. The inertial motion capture suit used for
this study has orientation sensors placed at strategic points on the body to
map body motion. Its portability, real-time performance, ease of use, and its
immunity from line-of-sight problems inherent in optical systems suggest it
would work well as a live performance technology. Many animation techniques
can be used in real-time. This research examines a broad cross-section of
these techniques using four practice-led cases to assess the suitability of
inertial motion capture to live performance. Although each case explores
different visual possibilities, all make use of the performativity of the medium,
using either an improvisational format or interactivity among stage, audience
and screen that would be difficult to emulate any other way.
A real-time environment is not capable of reproducing the depth and
sophistication of animation people have come to expect through media. These
environments take many hours to render. In time the combination of what can
be produced in real-time and the tools available in a 3D environment will no
doubt create their own tree of aesthetic directions in live performance. The
case study looks at the potential of interactivity that this technology offers.
The written thesis comprises 100% of this Masters by Research (35,000
words).
ix
Keywords
Motion capture, performance animation, performance capture, real-time,
inertial, live, liveness, spectacle, virtual reality, paralanguage, improvisation
x
A note on the accompanying media
The documentation presented here includes a DVD and a CD. This is
intended to support and further explicate the written data, and may be viewed
at any time prior to, during, or after reading the written thesis, at the reader’s
discretion. It is recommended, however, that the written thesis be completed
prior to viewing the DVD.
A Brush with the Real World installation (DVD): The DVD contains edited
video of the technical proof of concept performance installation, A Brush with
the Real World. The installation was held each evening from 16-18th July 2009
at the Judith Wright Centre of Contemporary Art, Fortitude Valley,
Queensland. The performances were improvisational. All video was taken on
the 18th July. (There were some software problems on the night and several
interactive features were disabled which prevented full use of the painting
environment and participation by the dancer.)
Research studio trials (CD): Over the period of the research many animation
features were trialed in real-time to gauge their suitability as tools for creating
visuals in performance. The CD contains short clips showing some of these
trials. It should be noted that most of these clips are not intended for
performance but for testing real-time performance of software features.
1
Figure 1: VRS Studio (2005)
1 Preface
The theoretical activity of humans, like the practical, is
not the independent knowledge of a fixed object, but a
product of ever-changing reality
(Horkheimer, 1982: 28)
When I began this research my
intention was to explore the use of
projected animations for public
events and performances. I had
accumulated a series of projection
concepts and was keen to develop
the best of them. Fortunately,
during my early enquiries Jon
Newell, software engineer at VRS
in South Brisbane, told me about
their newly acquired motion
capture suit. This was an opportunity to explore how real-time animation
could work in live performance situations, and it fitted in well with several of
the concepts I planned to explore. The motion capture suit was an inertial
based system, which appeared to have features compatible with working in a
live environment.
As a performer and deviser, I had a long interest in mediatized and
convergent artforms, producing stage acts dating back to the early eighties
that mixed visual media, such as video and animation, with sound and
lighting effects to enhance live performance. It was the “Bjelke-Petersen” era.
Politics in Queensland was strongly polarised and there was a strong political
undercurrent in much of the arts scene at the time. The motivation for using a
hybrid visual approach was less to do with art than working out ways to
maximise the appeal of a message.
2
My early career had diverse artistic influences, originally studying fine arts
and later drama focusing on group-devised and effects-based theatre. I also
learned fine art screen-printing for a while from Charles Bannon and became
involved in politically based street theatre with Pitjatjantjarra Land Rights
Movement under Ruby Hammond. I taught for several years, followed by
itinerant work in northern South Australia and Western Australia, before
heading up the Birdsville track on my motorbike and meandering towards
Brisbane. I worked as illustrator for Queensland National Parks for a while
but grew increasingly interested in theatre. Much of the early involvement in
performance was with political and community theatre, working on
campuses, at rallies, in cabaret venues, ZZZ venues, the Community Arts
Centre and other fringe venues, as well as facilitating unemployed and
community group projects. The work at cabaret venues took an increasingly
mediatised focus, melding into a form that came to be known at the time as
mixed media, in which visual techniques, sound and lighting effects were
incorporated into live performance to create a performance montage. So, as
much a reflection of social change, the arts, innovation, even science
appeared more as a vast wilderness open to exploration than a cluster of
genres separated by fences. Mixing genres and exploring new venues along
with experimentation in theme and form at the time was unencumbered by
the weight of tradition.
The visual theatre, mostly devised in collaboration with musician Janelle
Skinner, was an attempt at placing impact ahead of tradition in performance.
It was influenced by video clips, and freely mixed movement, synthesizers,
animation, black light, video and a bizarre array of themes, from performing
cockroaches to a baby Jesus ventriloquist doll; all aimed at visualizing a
message rather than tugging the bellropes of an artform. This visual form
seemed to have more impact than some of the didacticism in political theatre
and could be used in rock venues as comfortably as theatre cabarets. During
this time I experimented with the use of animation in live performance,
collaborating with Terry Kavanagh on several simple stop-frame animations
using cutouts on layers of glass. This, along with Terry’s individual work, was
3
used in several music-based performances. This sowed the seeds of what
now has become an important part of my work.
After a ten month community theatre tour through the Queensland Arts
Council, I worked in an array of positions including artist in residence for the
Multicultural Festival for 1984 and 1985 as well as designer, director, and
tour manager with productions through the late eighties. Commedia dell'Arte
and improvisation were strong influences around this time and touring
became a big part of the lifestyle. General audience tours, schools tours,
interstate tours, overseas tours, residencies, festivals and workshops filled
the nineties. My illustration work also continued with regular contracts for
illustration and graphic design work including a regular graphic work for the
Queensland Arts Council. More recently, I took on two stints as an events
artist through the Premier’s Department, three productions for Out of the Box
Children’s Festival and design for two children’s productions. From the mid
nineties more of the design and illustration work gravitated to computer and
so in 2002 I studied multimedia and later received a scholarship to study
animation at QANTM. In the last couple of years, my work has diversified to
incorporate 2D and 3D animation. This has included several projects that
used motion capture. The performance work has also continued with regular
stage and roving at festivals and conventions.
This research is a culmination of a long and ongoing learning process
through practice as well as an opportunity to explore an emerging field that
makes it possible to use live performance, animation and illustration in
combination. The research uses 3D motion capture as a tool in live
performance. For a variety of reasons, both technical and economic, the use
of real-time 3D motion capture in live performance and at live events has only
recently been feasible and, while not yet cost effective, it will soon reach a
stage where its use at public events is not limited by cost or technical
difficulty.
The four cases underpinning this research collectively shed light on ways of
creating real-time narratives or “instant movies” as a projected visual form:
4
movies that are presented as they are performed, in which actors and screen
depictions can both be part of the performative dynamic. The task of creating
complete movie performance would be a large collaborative project and
beyond the scope of this research. By segmenting the practice into the four
cases, each with its own techniques and outcomes, many of the technical
boundaries and creative possibilities can be tested in a more resource
efficient way that, when viewed collectively, still provide insight into the
logistics of creating big picture projects.
Animation based on motion capture has been used with increasing frequency
in the past few years in live dance, creating a visual landscape or
synaesthetic enhancement to the performance. One case, Motionics,
investigates the range of dance visuals that can be generated in real-time
with an acceptable level of quality. Chasing Shadows explores hybrid
theatrical performance, using a live actor onstage together with his projected
shadow. The shadow slowly takes on a life of its own and, eventually, a mind
of its own. Private Eyes focuses on the cinematography of a real-time movie,
created as it is acted: a misty street at night, a lonely figure and his story. It is
screen theatre, with the reality of the “live” blended with a world of cinematic
imagination. The primary case, A Brush with the Real World, is
improvisational animation that uses painting as a way of communicating to an
audience. Projected on a large screen is a cluttered 3D art studio inhabited
by a virtual artist. Armed with his paintbrushes the animated artist, Rupert,
can paint in the air around him. His movements and his painting are
controlled by a live performer who is able to set up visual, gestural and
artistic communication between Rupert and audience. This allows the
audience to influence Rupert’s world onscreen, especially his artwork.
The research I am undertaking centres on mediatized performance. It uses a
live performer to actuate movement in virtual 3D space in real-time. The
investigations are underpinned by technology with a very recent history. The
Gypsy Gyro Inertial suit used in the research was the first one commercially
produced, and arrived in Brisbane about the time I commenced the research.
Learning how to use the suit effectively required an understanding of how it
5
worked: studying software, exploring the physical parameters, learning how
the hardware interfaces, and understanding the organizational procedures for
motion capture. This was a necessary prelude to developing performances.
The research is practice-led, using a case study approach comprising four
cases to reach general conclusions on the suitability of inertial motion
capture to live performance. Inertial motion capture has properties that
suggest it could indeed be used broadly with live performance. It transmits
wirelessly, operates in real-time, fits into a small suitcase, can run on
batteries and record to a laptop. Practice-led research is still evolving as an
investigative approach, recognising and encompassing the valuable
contribution creative practitioners make to the pool of new knowledge.
The juxtaposing of disparate objects and ideas has,
after all, often been viewed as an intrinsic aspect of
creativity. The interplay of ideas from disparate areas
of knowledge in creative arts research creates
conditions for the emergence of new analogies,
metaphors and models for understanding objects of
enquiry. (Barrett, 2006: 8)
The outcome of this research is not primarily to create a performance but,
through practice, to assess and evaluate techniques that can be used in
performance. There is little information on the use of motion capture in live
performance and, for directors and choreographers, there is no clear picture
of the tools available and how they can be used. There is little commonality
of language between performative technologies and performance per se.
There is also a difference in the reflexive approaches that lead to creative
outcomes. This can result in unrealistic expectations, reinventing wheels, or
putting hybrid forms into the too hard basket. Along with some well
documented successes, the nineties are littered with performance/technology
experiments that fell short of their aims. There is a need to know more about
the mechanisms of real-time motion capture that can be used in real world
performance and included in real world events. It is important that the
6
outcomes can be practically applied to the conditions of live performance.
Motion capture is usually a highly collaborative process with technicians,
motion capture director and performance animator using costly equipment in
dedicated space in combination with teams of animators and a studio
production infrastructure. Live performance is not a mass saturation medium
and so the economies of scale necessitate a more modest operational
structure. The cases use a simplified workflow and limited collaboration to
approximate the lower budgets of live performance.
Often in the context of practice-led research into performance there has
become a de facto expectation of a completed performance from which
research conclusions are drawn. However, in emergent research generally,
the emphasis is not so much on the artifact created through practice, the
usual activity of artisans, but on matters of process or perception. In the
Methodology chapter I will show that this approach is strongly reinforced by
both practice-led theorists and supported by the Australian Department of
Education, Employment and Workplace Relations’ definition of research.
As anyone who works in the field is aware, 3D animation is time consuming
and complex. Creating a complete live performance using real-time motion
capture is more than twice the sum of its parts. It is the creation of an
animation environment, the creation of a live performance and the solving of
all problems that melding these distinct and different forms entails. With the
many techniques and methods in animation that motion capture opens up to
live performance, the series of strategically targeted cases, while collectively
providing a more sweeping evaluation than a single performance could, are
just a small peep into a field with vast potential.
7
2 Contextual Review
2.1 Introduction
We fear things in proportion to our ignorance of them.
(Titus Livy cited in Bock, 1970: 379)
This chapter surveys the historical background of motion capture, and
performance that incorporates motion capture. It explores the theoretical and
semantic foundations of the research. While animation has, at its roots, a
theatrical basis and uses many terms derived from theatre, it has developed
separately and some of the language mutual to both has meanings specific
to each. Animation has its own evolved aesthetic, governed more by its
ability to represent fantasy, to shape moments and create worlds visually
without the need for descriptions embedded in storytelling. While some see
motion capture as offering a way for live performance and animation to
interact, opening up new artistic possibilities, others see it as undermining the
art of animation, taking the “art” of animation out of the hands of the animator
and automating it.
As motion capture becomes more popular, many
animators have begun to think of it as the anti-Christ.
They talk about its instabilities, lack of flare and
expense, but in reality, they see it as a threat to their
livelihood. (Lindsay cited in Liverman, 2000: 45)
However, recent history of its use has indicated that in most areas motion
capture is not a threat to conventional forms of animation. This is partly
because motion capture has only limited application as an animation tool.
People have developed a perception of animation based on the visual and
behavioural conventions and aesthetics that have evolved with animation.
There is an evolved system of visual symbols and cues that make up a visual
language we have come to recognise and accept. When a character is
8
animated using captured motion, audiences ironically often perceive these
movements as incompatible with animation. Moreover, contrary to the notion
that actors are taking the place of animators, the overall number of animators
is growing quickly. In the U.S. the number of animators and multimedia artists
rose from 23,000 in 2005 to 26,000 in 2006 despite a growth in the number of
motion capture studios. During this same period the number of actors fell
from 59,590 to 51,880. (U.S. Bureau of Labor Statistics Office, 2007: n.p.)
The performing arts have a long history of experimenting with and integrating
new ways of working based on opportunities that arise with the introduction
of a new technology. For instance, the machinery used in some Baroque
Theatre, based on ship technology, used elaborate systems of winches and
pulleys to facilitate scene changes and enhance the visual spectacle of
theatre (Kirshenblatt-Gimblett, 2004: 49). Likewise electric lighting has
reshaped the way in which theatre is presented, embellishing the mood
onstage, changing the relationship between the actor and the audience, even
bringing about changes in the physical layout of the theatre itself. Technology
has opened new paths and led theatre practice into exciting new places as
well as dead ends. It has transmuted into forms that are no longer
recognisable as theatre. As Steve Dixon (2007) points out, the actor can
interact with his/her double in its many iterations, or can capture one moment
from a number of perspectives. Time can pass at different rates through the
same moment; narrative can accommodate concurrent multiple perspectives.
As the relationship between performing arts and a new media technology
matures, the technology can disappear behind the effect it creates, so that
audiences no longer see just the “gee whiz”, but are led once again on a
journey or engrossed in an experience. This happens when the building
blocks of the constructed reality are familiar.
The dimensions of space and time which constitute
our conventional understanding of reality have already
been destabilised, expanded and complicated so that
this virtual world seems less wholly 'other' and more
9
like just one more manifestation of the real-imaginary,
equally flexible and contingent, no more or less
constructed than our interpretations of the previous
live action. (Smart, 2005: 49)
2.2 Motion capture
2.2.1 Issues of terminology
Motion capture still lives at the bleeding edge and
most who have experimented with it have experienced
more bleeding than edge (Lindsay, 2000: 45)
Motion capture is distinct from other visual media in that it translates physical
movement to digital motion data. Motion capture or motion tracking can refer
to a broad range of data input techniques; the computer mouse is a simple
motion tracking device. In the context of this research motion capture refers
to the plotting of human motion in a form that can be represented as
performance in 3D virtual space. The three dimensional movement data
collected can be assigned to drive equivalent movement by a virtual
character. This data can also be used to control visual effects, sound, virtual
scenery, in fact anything that can be manipulated by computer data, including
things in the physical environment, from lighting rigs to robots.
Motion Capture is the process of recording a live-
motion event and translating it into useable
mathematical terms by tracking a number of key points
in space over time and combining them to obtain a 3D
representation of performance. (Menache, 1999: 1)
Unlike linear or interactive games media, in motion capture the effect created
can be driven by movement as it occurs. As the motion data is generated by
a live actor in real-time the effect on screen can be sympathetic to the
10
organic timing of live performance. The metronomic inflexibility of linear
media has been a drawback when projection is combined with live
performance, which may help to explain the limited use of synchronised
projection in theatre. This inflexibility has been overcome to some extent with
the triggering capabilities of interactive games technology. Visual media is
more commonly used in dance, an artform that usually accompanies rather
than leads the timing. This is also because dance is as much a visual artform
as a performance genre, while theatre generally relies less on spectacle and
more on the flow of human interaction.
Inertial motion capture uses strategically placed sensors at points around the
body to measure changes in orientation. These orientation values are sent
wirelessly to a receiver that is attached to a computer. This data is used in
combination with an actor file, a file that contains the performer’s body
measurements, to portray performance in digital form. The sensors only
calculate orientation values and not positional data, and so they provide no
clue as to their location in space. However, using the orientation of other
sensors and appropriate body measurements each sensor can approximate
where it is in relation to other sensors. The animation and games industries
are driving the rapid development of tools that can be used to apply motion
data to sophisticated animation environments, and many of these tools are
making their way to real-time animation environments. A fast 3D environment
such as MotionBuilder can apply a broad range of animation and scripting
options to motion data, providing a growing number of creative possibilities
for its use in live performance. However, the future probabilities are
exponentially greater as many of the extensive features available in
mainstream 3D software make their way into the real-time environment.
In creative terms motion capture is a pipeline between live performance and
animation. The motion is derived from the movements of a live performer and
converted into language that can be used by an animator. The animator is
able to create an environment that can be affected by the performer’s
movements. This environment can then be a responsive projection in a live
performance scenario. With the exponential growth of 3D animation, it has a
11
growing presence in cartoons and other animated sequences. Motion capture
had a marginal presence in movies until the very successful Lord of the
Rings trilogy when it moved to mainstream status, as much the result of
technical advances as the celebrated performance animation skills of Andy
Serkis who provided most the movement and facial expressions for the
character, Gollum.
This type of animation, where some or all of the motion is derived from the
movement of a live performer, is called “performance animation”. So “…
motion capture is the collection of data that represents motion, whereas
performance animation is the final product of a character driven by a
performer” (Menache, 1999: 1). Real-time motion capture occurs when there
is no discernible delay between the recording of motion and its conversion to
motion data. In real-time performance animation a 3D character or effect
onscreen occurs simultaneously with the movements of the performance
animator. The distinction between motion capture and performance
animation raises questions as to how motion capture relates to art. Is motion
capture a tool that can be employed in the creation of art or is it a medium for
reproduction of art? Is the art in the motions of the actor or the effect of these
motions on the screen? The actor embodies the ritualistic nature of the “live”,
the authenticity of the original. Through capture these movements become
reproducible and this reproducibility “emancipates the work of art from its
parasitical dependence on ritual” (Benjamin, 216) However, works based on
“real-time” motion capture disembody the art, but arguably preserve the
ritual, for the work is an altered vision of what’s actually happening at the
time and the possibilities for, at least, non tactile interplay with an audience
are preserved. The work itself is not just a reproduction, as in a movie; it is a
pipeline between the performer and a visual environment that creates unique,
highly responsive and ever-changing artwork. The art is both in the actor’s
movements, as live performance, and in the visual environment, as computer
art. Stripped of this uniqueness, it becomes a technology demonstration.
The terms “real-time” and “live" both have multiple definitions that overlap
and, when used with motion capture can be misleading. A physical
12
performance that happens at the same time as its digital depiction onscreen
is considered “real-time”. However, the term is also used to describe
instantaneous game-play action as a player navigates a virtual world, a
situation in which only the triggering of the animation is “real-time”. With
movies the animation derived from real-time motion capture is usually further
modified before it is presented to the public; the physical performance and
the public showing are at different times. So “real-time” has been used refer
to the entire process from live performance to screen performance as well as
only part of this process. The word “live” also has levels of meaning that are
ambivalent. When we refer to dance as “live” it indicates that the dance is
physically being presented at the same time and place it is viewed. However,
when referring to a mediated event such as live broadcasting, the event is
happening at the same time but at a different location to where it is viewed.
The term “live” was originally used in this context in the 1934 BBC Yearbook
to differentiate broadcasts that were created as they were delivered from
recorded broadcasts (Auslander, 2002: 18). The problem in the case of this
study is that it looks performatively at the intersection of performance and
broadcast media. So to avoid confusion, the word “live” is used to describe
performance that involves the physical presence of performers, and the
expression “real-time” refers to the process of representing a performance in
an animated form onscreen as it is physically performed.
2.2.2 From performance to screen
...the issue is not whether or not to uphold a particular
definition of performance over and against media and
technology, but to work with relationships between
them. (Kirshenblatt-Gimblett, 2004: 49)
13
Motion capture is suitable for certain
styles of animation and certain
circumstances in which animation is
used. It is an expensive technology
that is maturing at a rapid rate, driven
by two of the world’s most lucrative
industries, movies and interactive
games. It may one day change the
face of mediatized performance,
because it is a medium that can
respond instantly to a person’s input,
using language and gesture that is
familiar, and a sense of timing that
flows organically with the whims of
human interaction. Human motion is
captured with the subtleties of
expression and nuances of secondary
motion that are unrepeating. This is
structurally different from the triggered
behaviour used in interactive games
environments (a limited set of pre-animated responses and simple
behaviours as opposed to an unlimited set of appropriate spontaneous
responses). Appropriateness of response is intrinsic to the way we
communicate and triggered responses, no matter how well devised, are only
a limited simulation of communication. In contrast, responses in real-time
motion capture are potentially as diverse as human interaction, albeit on a
screen with a projected 2D simulation of a living being.
A screen performance using motion capture can convey paralanguage, the
behavioural and gestural language of human movement, as it is happens.
With motion capture it is possible to use an accurate screen depiction of one
person and combine it with the motions of another. Human movement can be
assigned to a bird, or bovine movement can be assigned to a human. In fact,
human motion can be isolated and used to affect a projected scene or
Figure 2: The dancer’s movements are captured digitally and then applied to a 3D character
14
character, generate sounds, change scenery, direct robots and conduct
spectacles. Furthermore, the motion of a person can potentially control any
conceivable device or digital image that can be digitally networked anywhere
on the planet without any discernible delay.
2.2.3 The actor
When an individual plays a part he implicitly requests
his observers to take seriously the impression that is
fostered before them. They are asked to believe that
the character they see actually possesses the
attributes he appears to possess, that the tasks he
performs will have the consequences that are implicitly
claimed for it, and that, in general matters are what
they appear to be (Goffman, 2004: 61)
As mentioned earlier, both actors and animators have felt some trepidation
about the use of motion capture in film. Actors see it as supplanting their
identity, and animators as automating their skills. Bill Nighy played the villain,
Davy Jones, in the Pirates of the Caribbean trilogy (2003, 2006, 2007).
Motion capture was used to give his face a realistic octopus-like appearance,
with tentacles curling and twisting in concert with his expressions and moods.
Initially, he was skeptical but came to appreciate that the facial animation,
rather than obscure the nuances of facial movement, actually enhanced
them.
They can enlarge an actor's experience. There are so
many stories we're going to want to tell that require
those kinds of skills and those kinds of images, and I
see no reason why actors and CGI can't work hand in
hand. There's no reason one should threaten the
other. (Bill Nighy, 2006: 1)
Motion capture does not replace traditional animation or acting. Both these
15
skills have traits that motion capture cannot or, at least, does not need to
emulate. It can offer an extension of what exists, new opportunities for
artisans and new possibilities for moviemakers. It helps to expand and
diversify the realm of visual representation.
…all performance involves a consciousness of
doubleness, through which the actual execution of an
action is placed in mental comparison with a potential,
an ideal, or a remembered original model of that action
(Bauman cited by Carlson, 2007: 73)
Steve Dixon, author of probably the most comprehensive compendium of
digital performance to date, expands on Artaud’s notion of “the double” to
explore the interplay between the live actor and his/her projected portrayal,
drawing on a broad range of productions including 4D Art’s Anima and the
one man show, Afasia by Marcel-Lí Antunez Roca. The actor’s digital double
can be a distinct and separate entity, as in a traditional stage role, or can play
another side to the person onstage, a role that is more common in
mediatised performance. This enables us to view multiple facets of the same
person, allows a person to interact with themselves, or creates an interplay in
which the screen double can extrapolate, intensify, magnify, diminish or
make absurd the actions of the physical self onstage.
With performance animation the performer can disappear leaving the double
as the storyteller, the puppet on the screen. A production in which the actor
can be represented on screen in real-time straddles the divide between what
is mediatised and what is media. The actor can totally control his/her screen
image while not being visible to the audience. When live actors join real-time
avatars onstage, as in Saltz’s interpretation of The Tempest or Pinhanez’s
one-man show, A Computer Theatre Play at the MIT Media Laboratory, a
dynamic is established that is unique as a mediatised experience, freeing the
actor to control the pace of the stage action. Dixon refers to this type of digital
representation, as an “avatar stage double”. A stage double can be invisible
while, in a liminal way, the subtleties of body language can still be seen. Jane
16
Harris dispenses with a visible body altogether in Portrayal, using only a
flowing dress to convey the captured dance movements of Ruth Gibson,
“emphasising the beguiling qualities of fluid material and form.” (Harris &
Walsh, 2004: 139)
The dimension of time is a playground for mediatized performance, allowing
asynchronous and multiple views of events, juxtaposing the same person in
the same space at different times simultaneously. In the play, Memorandum
by the Japanese performance group, Dumb Type, one event is shown at
different rates on multiple screens at the same time in “a montage of time
perspective” (Peterson, 2004, 32). Only a few bits of furniture litter the stage,
behind which are four projections each showing the scene onstage but at
varying speeds from very slow to very fast. All have been synchronised to
end at the same point in time on screen. This “telematic” performance shows
how visual media can introduce time, not just as a linear progression of
moments, but as another perspective.
This same juxtaposition of passing time can also be used in animated
representations of a scene onstage, opening up new ways of portraying a
double. In 3D space the same scene or action can be shown from any
perspective at the one time; so dimensionality of the space becomes a
dynamic element. Having a pipeline between the performer and a 3D
animation environment opens up the huge legacy of visual and interactive
software to live performance, making possible all sorts of anthropomorphic
and highly interpretive visual derivatives of live action. Not only can multiple
perspectives be used but distinctly different visions of the same event, each
with its own place, context and aesthetic. These visions can happen
concurrently with real life performance and, with the dimensionality of 3D
space, each can focus on a different aspect of the same set of actions.
2.3 Performativity
One is not simply a body, but, in some very key sense,
17
one does one's body and, indeed, one does one's
body differently from one's contemporaries and from
one's embodied predecessors and successors as well
(Butler cited by Felluga, 2003: n.p.)
2.3.1 Introduction
Performativity as a concept has expanded significantly from J. L. Austin's
originally intended meaning, when he coined the term to describe self-
actuating statements or "performative utterances". Judith Butler (in Jagger,
2008) used the term in a societal context to highlight how speech acts shape
who we are. We absorb the language of the social constructs that surround
us and this becomes a script for the way we act out our lives. For instance,
Judith Butler sees gender as an act rather than a category. Performativity
these days can refer to a broad range of social self-actualising processes
and has been associated with body language (Schechner, 1993: 45) and with
emergent research through creative practice (Haseman, 2006). Motion
capture can be described as a performative medium in that it facilitates a
creative transition that encapsulates the spontaneity and immediacy of
human movement. This section looks not only at the performativity of live
performance but the performativity of the medium and how this combination
can create performance that preserves a sense of immediacy while allowing
for new visual possibilities. It also looks at the social context of visual media
from a postmodern perspective, and theoretical factors that come into play
when working with mediatized performance. Finally, it outlines the history of
performance that uses motion capture including an overview of its use in the
animation industry. Motion capture is a fast evolving technology at the
beginning of its history. This contextual review takes a broad overview of this
type of performance, rather than dealing at length with specifics, because the
heritage of this type of performance is far outweighed by the scope for what
can happen in the future.
The words “convergence” and “diversification” are commonly used when
referring to innovation in visual technology, emphasising dynamic changes in
18
applications and structures in a rapidly evolving field, rich with niches for
exploration. The cases of practice used in this research are not only a
convergence of two fields, motion capture and live performance, but also part
of the diversification of each field, and highlight some antithetical and
juxtaposing concepts. Immersion/distraction, theatre/spectacle,
live/mediatized, simulation/original, presence/disappearance, reality/virtual
reality, visual/aural are some binaries at play when visual technology is used
in the performing arts. They provide an insight into the social context, the
effect on the audience, the way in which live performance and projected
images interact and the nature of the virtual world.
2.3.2 Spectacle
The visual pictorial aesthetic triumphed over the aural,
and an important reason for this victory must be the
greater degree of innovation in the technology of the
visual, above all in the production and control of
artificial light. (Christopher Kent, 1996: n.p.)
In broad social terms media is a pervasive influence that exposes us to
enormous volumes of information while at the same time subjecting us to
unprecedented levels of influence, shaping our sense of reality, removing our
lives further from the physical and, some may argue, exalting the trivial above
the consequential, hiding the “actual” under layers of subjective
interpretation. In mass visual media, it seems, we prefer to be distracted
rather than informed. Debord’s “spectacle”, Baudrillard's “simulation” and
Lyotard’s “metanarrative” allude to the mechanisms that underlie this
pervasion. In the information age, knowledge and meaning can be inhumed
by the sheer enormity of the information available and this information
typically reflects the dominant values of the technologically advanced
societies. It is integral to the mechanisms of post-colonialism and
consumerism. While an understanding of the forces at play in visual media
provides the context in which one applies the tools of art practice, this
19
research is investigating the tools as much as applying them.
Along with the growth of visual media, there has been a parallel shift towards
visual performance. The evolution of modern circus with companies like
Circus Oz, Footsbarn, Circus Archaos and Cirque du Soleil, along with the
phenomenon of performance spectacle with groups such as La Fura dels
Baus are an indication of a trend in live performance towards increasingly
awesome spectacle. This is reinforced by the greater use of visual
embellishment in stage productions such as modern renditions of Phantom of
the Opera or Les Miserables (Kellner, 2004: 8). It emphasises people’s
greater focus on the ocular that, through film, television and more recently
video games, has become so entrenched in our lives. While it can be argued
that spectacle has always been compelling, there has never been a previous
time in history where spectacle has had the means, the methods or the mass
audiences it has today.
Developing countries and the globalized world are
emerging into a culture of media spectacle that
constitutes a novel configuration of economy, society,
politics, and everyday life. It involves new cultural
forms, social relations, and modes of experience. It is
producing an ever-expanding spectacle culture with its
proliferating multimedia and interactive spectacles that
are intensifying and expanding as wireless
technologies develop. Critical social theory thus faces
compelling challenges in theoretically mapping and
analyzing these emergent forms of culture as well as
democratization and social justice. (Kellner, 2004: 17)
Spectacle, as described by Guy Debord, is both a pervasive visual inundation
and a means of social control, distracting thought and subjugating reality with
consuming images that are a reflection of power, where society is relegated
to the role of observer, displacing the real with a prevailing appearance of
reality. The receptiveness of an observer and acceptance of the unreal as a
20
part of reality are an indication both of the power of spectacle and the
message it imparts. In this world there are no longer points of reference and
so “the spectacle is the real” (Debord, 1968: 9). In a similar vein, Baudrillard
postulates the mechanisms that underlie this media inundation. Through the
media, larger-than-life models of events and people are created which are
representations of “real” events and people but the events and people do not
exist and cannot exist in the form they are shown. These “simulations” exist
only in a world which Baudrillard calls “hyperreality”. In this world signs
become more important than the commodities they signify (Baudrillard, 1988:
166). Live performance is both a victim and an accommodator of spectacle
and is reflexive in its reaction to media’s simulation.
Not only are theatre audiences seeing live
performances that resemble mediatized ones as
closely as possible, they are also apparently modeling
their responses to the live event on those expected of
them by television (Auslander, 1999: 29)
The notions of spectacle and simulation highlight how media influences our
perceptions of reality. In convergent performance that incorporates visual
technology there is a need to understand these mechanisms, recognizing the
distinction between creating a different reality and creating a different
perception of reality. The defining feature that separates spectacle, a visual
event, from spectacle, the societal organising principle, is marketing. The
economy that drives the creation of ever more eyecatching visuals is based
on selling a product or idea. This consumerist barrage has reached a point
where, according to Rushkoff, people:
…want a moment of free time and of genuine free
time. And there's no more public space left to do that
in. There is no more free space left. We've had a real
contraction of public space both mentally and in the
media and in the real world, where there's nowhere to
21
go where you're not being marketed to. (Rushkoff,
2008: n.p)
Kellner concurs with Debord in recognizing spectacle as "one of the
organizing principles of our economy" (Kellner, 2003: n.p.) but, more than
just a means of control, he believes it also reflects the nature of the
underlying culture. Along with the engineered entertainment
“megaspectacle”, the Iraq War, came the mega-exposé, Abu Graib,
indicating that there is room for opposing interests (maybe only enough to
validate the impression of impartiality). Baz Kershaw points out that digital
technology is minaturising spectacle to the small screen. This process
diminishes the individual while at the same time spreads him/her everywhere
(2003: 604). However, he recognizes that spectacles can also be a powerful
force for change as they “open up new domains for radical revisions of the
way things are” (2003: 599).
To install such humanity at the heart of spectacle in
order to give it a radical new lease on life means
continually recognising and reproducing this mutual
vulnerability, recognising in the other something akin
to the qualities of spectacle that provoke contempt and
curiosity. (Kershaw, 2003: 611)
In comparison to live performance, new media is often regarded as
subtractive, a ghostly imitation of reality. Nevertheless, the ability to reach
and influence mass audiences using media technologies makes it inevitable
that, to compete for attention and to provide alternative perspectives, new
communication tools are explored and used. New media that are at the basis
of spectacle are not the sole preserve of large corporations. An effective
alternative voice has evolved in more established forms of mass media, such
as film and television, and there is social benefit in harnessing new tools of
media for use outside the constraints of purely commercial projects.
22
While recognising the broad social impact of visual mass media, it is
important to understand the mechanisms that make visual media so
compelling. As visual mass media has matured so have our expectations and
our choices. We can be absorbed, influenced, entertained, distracted and
deluded on so many levels. For instance, movies can relate on an emotional
and empathetic level. A movie can be thought provoking. They can draw us
in to another existence, making us feel like we are somewhere else in some
other time. Visual media are no longer just an escape from day to day reality
they are now an integral part of our reality.
2.3.3 Reality
The opinion which is fated to be ultimately agreed to
by all who investigate, is what we mean by the truth,
and the object represented in this opinion is the real.
That is the way I would explain reality (Peirce, 1992:
139),
Media has greatly facilitated and democratised the availability of information
but it has also expanded the scale on which information can be manipulated.
From Adam and Eve to WMDs there have always been mass “truths” that are
detached from real evidence, but now they can be disseminated quickly on
an unprecedented scale. Reality appears more durable than truth, constantly
affirmed by the consistency of the physical world. However, even the
physicality of reality may not be so affirmable, as immersive technologies
become better at simulating visual, aural and haptic experiences. This is still
a way off (Krueger, 2002: n.p.) but there are other ways in which new media
are challenging our sense of reality. Research into “presence” is helping to
differentiate the role of mental perception, in contrast to that of physical
sensation, in shaping our reality. Studies using interaction with avatars
confirm the role of social responses in our construction of reality. Under
certain conditions people will respond to an avatar in the same way as they
would respond to another person. Experiments by Slater and Garau et al.
23
(2005: 104-116) established that people in convincing circumstances would
relate and respond to screen characters as if they were real, even if they
have a more ontological awareness that the environment they are exposed to
is virtual. This indicates that a character onscreen, using the same devices
and strategies that actors use, can engage an audience in two-way
communication. Improvisational techniques in which the actor allows the
story to unfold as it is created, set up a reality of the moment that is easily
transposed to real-time screen interaction.
Performance operates by constructing realities.
Performance can be considered to consist of special
ritual situations in which the actions are part of a meta-
fictional linguistic universe and in which, therefore, the
performers should not be understood to be executing
purposeful actions. (Peterson, 2004: 35)
When investigating a mixed reality performance, it is relevant ontologically to
distinguish physical reality from projected reality. Reality is not necessarily
being substituted by reproduction of reality but is extended to include
reproduction as part of reality. However, reality itself changes with advances
in media technology so that the infiltrating simulation and spectacle that
accompany ever more pervasive media add to what we accept as reality
(Dixon, 2007: 143). Despite some of the ominous projections in post-
structuralist and situationist theory as to the effect on society of pervasive
visual media, people in general are still aware of the distinction between
physical reality and media reality. Arguably, any blurring is as much based on
our formulation of truth as our perception of reality.
The media image may have exerted an increasing
influence or power on human consciousness, but that
consciousness has not undergone total
metamorphosis and collapsed into helpless
zombiedom…. Media’s afterimages need not
insidiously corrupt, nor be indelibly inscribed on the
24
subject’s mind and body (Dixon, 2007: 143)
This statement does not negate the influence of media rather it hints at the
compound way we relate to media. Generally, we can easily distinguish
between media and reality; we are aware that it is diversionary and we may
know the mechanisms and strategies that underlie media. However,
recognising the nature and influence of media does not preclude its ability to
influence us. To deny this influence is to negate the entire advertising
industry. Media both reflects and creates our reality. However, while people
may respond to a screen character in a screen world, or even be lured into
dialogue, this does not suggest that a sense of “being there” with a person
translates into believing the person is actually present. The interaction is not
authenticated sensually. However, it does raise questions as to where reality
ends and virtual reality begins.
Gabriella Giannachi points to the blurring boundaries between the real world
and virtual space and its impact on arts practice. This inevitably reflects on
our perception of nature, what it is to be human and the dawn of post-
humanism, and how this impacts on and is pioneered by arts practice. We, it
is argued, have already become melded with and inseparable from the
technology we have created (Giannachi, 2004). The virtual has become a
subset of our reality. This reinforces Baudrillard’s stance that much of what
we see in the media is simulation: it never happened in the terms it is
presented. The First Gulf War, a war without bodies, a war seemingly
conducted as a video game, is a compelling example.
Virtual reality is in a paradoxical relationship with the
real. On the one hand, it is part of the real; yet, on the
other, it has been constructed to be different from the
real in order to be perceived as separate from it.
(Giannachi, 2004: 123)
Dixon sums the matter up by identifying it as a question, not just of real and
virtual, but of the march of technology itself.
25
… the real has changed, as it has always done. The
real, and our consciousness of what is real, is subject
to time, and for several hundred years it has had a
conjoined twin: technological progress. (Dixon, 2007:
144)
In movies, new animation techniques have made it sometimes difficult to
discern what is physically recorded and what is animated. However, any
difficulty in recognising the difference between reality and virtual reality is a
dilemma for future generations rather than the present. Nevertheless, the
possibility that a reality constructed virtually could substitute for physical
reality provokes a serious re-examination of what reality is. Through
simulation technologies, the day we can enter an environment where we
cannot tell the difference gets incrementally closer although it is still a long
way off. In the meantime, a better point of reference artistically may be
realism rather than reality.
The flight from reality is not exactly a new issue. My
parents were once worried that I was being seduced--
by the world of books. Certainly, movies, videos, video
games, and the internet have already successfully
seduced us from the real world. However, there are no
instances of anyone being seduced by current
immersion technology. It is not good enough yet.
(Krueger, 2002: 1)
26
2.3.4 Presence
… the impression of reality we get from a film does not
depend at all on the strong presence of an actor,
rather on the low degree of existence possessed by
those ghostly creatures moving on the screen and
they are, therefore, unable to resist our constant
impulse to invest them with the "reality" of fiction, a
reality that comes only from within us, from the
projections and identifications which are mixed in with
our perception to the film. (Metz cited in Callesen,
2001: 4)
This impression has an interesting and related parallel within animation.
Highly stylised characters existing in a stylised world of the kind we see
regularly in mass media, such as the Simpsons, Futurama or SouthPark, are
readily invested with life. As animation comes closer to visually simulating the
real world, the mind switches to focus on anomalies that separate it from the
real, as pointed out by Masahiro Mori (1970: 33) in his “Uncanny Valley”
Theory; the more realistic a screen world the more we tend to scrutinise it for
faults. The fault may be a subtle lack of texture, odd reflectivity of surfaces,
shortcomings in secondary motion, moments of inappropriate timing, a slight
stiffness of mouth movement, little disparities that form fissures in the real-
worldness of the representation. When a virtual world comes close to
representing reality it is entering a realm that we are experts in: the
appearance of reality. Tiny shortcomings stand out like headlights in the night
and jolt us out of the fantasy.
Other factors affect the way we relate to projected scenarios. Using the
example of a mediatised performance where performers temporarily leave
the stage and the audience is left with media alone, Dixon in Digital
Performance points out that there is a discernible change in the posture of
the viewing audience, which, he indicates, is not a relaxation or a diminishing
of attention, but a change in “the type of attention.” This suggests broadly
27
that live performance is an inclusive experience, a form of dialogue; an
audience likes to respond in a way that communicates to the performer(s).
Viewing media may be engaging but not usually inclusive; there is no
compunction for an audience to communicate with characters on a screen.
For instance, it is uncommon for people to clap at the end of a movie but is
almost inevitable at the end of a live performance.
The extent to which we connect with the screen environment and the
characters on screen is the subject of study into “presence”. Presence is a
word used both in live performance and virtual performance but is usually
applied in different ways. Presence in theatre can refer to the aura that is
endowed by the spectator onto an actor. It is a quality rather than a quantity.
It is a magnetism that imbues a heightened sense of belief.
Since Stanislavski, when actors are observed carefully
and perform well, they are said to have "presence".
The actor achieves presence through performing as if
another. (Phelan, 1993: 117)
Presence in virtual reality is somewhat aligned but, unlike in live
performance, it has no bearing on the feelings and emotions of the spectator.
It is more akin to the level of being that an environment or a screen character
might endow. In research, as a factor in games and other interactive media,
“presence” is treated as a quantifiable entity.
Presence is the response to a given level of immersion
(and it only really makes sense when there are two
competing systems - one typically the real world, and
the other the technology delivering a given immersive
system). (Slater, 2003: n.p.)
In a virtual environment “presence” is a personal response, and can be
28
described in what Maslow (1968: 133) calls “being values”. Presence is a
sense of “being there”, while the term “co-presence” is a sense of “being with
someone”. The notion of presence in performance, therefore, has more in
common with co-presence in that it relies on the presence of both a
performer and observer(s). In fact Erica Fischer-Lichte uses it definitionally
when she states, “the bodily co-presence of actors and spectators enables
and constitutes performance.” (Fischer-Lichte, 2008: 32). In both
performance and virtual reality “presence” is also used as an umbrella term
covering a number of subtypes such as “co-presence”, “behavioural
presence” and “social presence”. In this broader sense Lombard and Ditton
define presence as "the perceptual illusion of non-mediation" (1997: n.p.).
This more inclusive meaning provides common ground, as both a virtual
environment and a live performance are mediated. If both types of presence
are viewed not as an attribute of the performer but as a response of the
spectator to the performer, the two meanings are no longer so mutually
exclusive. For instance, an observer might regard an actor as having
presence because (s)he draws them into the moment, makes them feel like
they are with the character.
Co-presence does not rely on the visible presence of two people; a person
talking on the phone can feel a sense of co-presence (Heeter, 2003: 340),
indicating that the actual presence of another person is not essential. A
sense of “being with someone” therefore relates to communication as much
as to appearance. This suggests the spontaneity of communication that is
achievable using motion capture may be very effective in establishing a
sense of “co-presence” between a person and a virtual character. In such a
production the virtual character is performative, conveying the “liveness” of
the performance animator.
Presence is crucial to the successful implementation
of virtual environments within the performing arts.
Actors are able to imagine, to role-play, to
29
communicate and create a shared environment, but
they require a high level of responsiveness from the
system and a transparent interface. Perhaps more
than other users, actors can meet the technology
halfway, forgiving it for some constraints. (Reeve,
2000: 209)
Thus how a virtual environment is integrated into a performance and how a
virtual character appears and relates are important considerations when
creating a digital performance. As pointed out earlier, a virtual environment
does not need to be realistic, it does need to be integral and consistent. For
an onscreen character to be successful in relating to an audience, there
needs to be some common social ground. Performance animation is capable
of providing the instantaneousness of verbal and bodily communication.
Nevertheless, quirks and faults in the real-time animation are common so
when a scenario is devised for this medium the choreography, direction and
visual design need to minimise the likelihood of these anomalies occurring.
This is similar to the attention to detail required in live performance when
transposing the audience to another place and time. Little things such as the
flickering of a light, the incessant hum of a generator or an inappropriate hat
can undermine the world that is created, irrespective of how good the
production is otherwise. Function and memory are also factors. Orliaguet
(2001: n.p.) points out that in representing a real object, a virtual object
needs , at least, to incorporate its function.
30
2.3.5 Liveness
Performance’s only life is in the present. Performance
cannot be saved, recorded, documented, or otherwise
participate in the circulation of representation: once it
does so, it becomes something other than
performance. To the degree that performance
attempts to enter the economy of production, it betrays
and lessens the promise of its own ontology. (Phelan,
1993: 41)
It is ironic that we could define theatre based on the irreproducibility of
performance when, in times predating film and television, through stringent
adherence to script and choreography, it has strived to be repeatable. In-
house theatre played to a different audience every night, and so there was no
advantage in incorporating variation. Film was much better at repeatability
than live performance, and in many respects, forced live performance to look
for a new rationale and to explore new approaches and venues. Live
performance has adapted, sprouted new forms that otherwise may never
have arisen, intertwining with many human endeavours and merging with
other genres to be the diverse, pervasive and often transparent fabric that it
is today.
Peggy Phelan argues that live performance is transgressive, making each
performance unique and irreproducible, separating it ontologically from the
mediatised (Phelan 1993). This is quite famously rebuked by Philip
Auslander who paints this as a nostalgic notion that bears only a residual
resemblance to the reality of modern representation. He demonstrates that
modern theatre is almost inextricably mediatised, so much so that theatrical
productions often draw their content from film and television. At the time
Auslander’s book, Liveness: Performance in a Mediatized Culture (1999) was
published the two stances, that of Phelan and Auslander, were generally
portrayed as irreconcilable. In recent years commentators such as Fischer-
Lichte and Dixon have argued that both positions can mutually exist. Dixon
31
elaborates on the liveness debate claiming that “the work of art” has been
used to affirm both arguments.
On the one hand it stands as evidence for the unique
aura and presence of live performance, which can only
be damaged and robbed by technology, and on the
other as proof that technological incursion does not
significantly alter reception of performance, since our
minds (and performance itself) are already mediatised.
(Dixon, 2007: 117)
Intriguingly, some aspects of live performance that Phelan uses to define its
uniqueness are also seen by those working in digital performance, such as
Callesen (2001) and Saltz (2001), as intrinsic to the level of “presence” in a
mediatised production. In live performance it is the variability, the
spontaneity, contextual appropriateness, the responsiveness, the timeliness
of inference and gesture, the signs in secondary motion, the proxemics, the
consequent unrepeatable interaction between the actor and the audience
that are claimed to provide an intimacy of experience and an authenticity of
interaction. In a performance in which real-time motion capture is used the
animation can move beyond the linear confines of other visual media and
interact with the audience in the moment, incorporating many of the traits that
are seen to be intrinsic to “liveness”.
These traits are all manifestations of paralanguage, the language of the
body. Through performance animation many aspects of paralanguage are no
longer only the preserve of human-to-human interaction. How effective this is
as a means of communication between a person and a virtual person is
relatively unexplored, although the way in which a person relates to the
streamed video of a friend during a “live” chat session suggests they would
feel a strong sense of co-presence. Using performance animation, the body
language of the virtual character is not generated in software but is a digital
reconstruction of a live actor’s movements. This is significant as a means of
incorporating human facets once considered exclusive to live theatre into a
32
digital environment. The underlying awareness of physical reality is always
there but, as in movies and games, people can be absorbed by the actuality
of the interaction. Unique to real-time performance animation, this media
interaction may lever the spontaneity and complexity of paralanguage.
This research is therefore not just exploring the use of real-time performance
animation in live performance but may shed some light on the nature of the
paralanguage between a person and virtual character. The potential for
motion capture in live performance is that it offers a choice of possibilities in a
way that is transgressive. There is much to be learned from the burgeoning
research on human/avatar interaction in the games industry that can help
create a broader understanding of how onscreen characters can be used
sympathetically in live performance situations. In referring to network
interaction, Garau and others highlight the importance of appropriate
responses in creating meaningful exchanges between real and virtual worlds.
The challenge with human-human communication is to
drive avatar behaviors that enrich, rather than hinder,
communication between remote participants. With
human-agent communication, the challenge lies in
simulating intelligent social responses by coupling the
agent’s dialogue with appropriate conversational
behaviors. (Garau et al., 2000: 105)
33
2.3.6 Spontaneity
There is an important difference between the way someone would relate to a
person in a recorded video sequence or movie and how they relate to a “live”
feed or video chat. This difference lies in the spontaneity of interaction that
can occur with real-time visual media. In an experiment conducted by Garau,
Slater, Pertaub and Razzaque in which participants were exposed to varying
levels of interaction with avatars it was found that all participants were able to
"react to agents both rationally as computer interfaces, and non rationally as
people" (Garau et al, 2005: 114). Generally, presence increased with the
level of interaction.
By using performance animation, actions and responses can be physically
diverse, complex and appropriate because all movements are a real-time
copy of the motions of the performer, providing a more complex,
spontaneous and, assumedly, credible environment for interaction. The
Garau, Slater, Pertaub and Razzaque study supports the notion that
responsiveness is an important factor in communication.
Our current findings support the hypothesis that
increasing responsiveness even on a simple level can
have a significant impact on certain aspects of
people's social responses, both in terms of their
perceptions of virtual humans and of their actions in
response to them. (Garau, Slater, Pertaub and
Razzaque, 2005: 105)
Saltz views live performance as “inherently interactive” (2001: 109).
Irrespective how rigid and demanding the rehearsal process is, a live
performance is always subject to variation. Spontaneity is a crucial element
of effective interaction. Saltz defines and describes many of the ways in
which media can be used in live performance, including virtual scenery,
interactive costumes, alternative perspective, subjective perspective,
34
illustration, commentary, diegetic media, affective media, synaesthesia,
instrumental media, virtual puppetry and dramatic media (Saltz, 2001: 124-
126). He argues that inherent in the perception of liveness is interactivity and
if media can comfortably sit within this interactivity then it becomes integral
and not detached. Spontaneity, as it relates to paralanguage, is performative.
The spontaneous give-and-take between performers
and spectators, and among a group of sensitive
performers, is integral to theatre's appeal as an art
form, both in the most highly stylised genres of theatre
and in “realistic” theatre. (Saltz 2000: 109)
Time is an ingredient in communication. In conversation, for instance, the
gap between a statement and a reply can radically affect the meaning. A
slightly extended pause in a conversation can, with subtle changes in gesture
and in different contexts, convey incredulity, thoughtfulness, anger,
vagueness or lack of understanding. This interplay of timing and body
language is the essence of performativity. Schechner regards performativity
as underlying theatricality. Gestures, movements and sounds, “if not
universally understood come close to conveying the same feelings
everywhere”. (Schechner, 1994: 43) Body language is a universal language.
In Schechner’s interpretation, performativity in physical communication is
transcultural, unlike verbal or written communication, which is more local.
This indicates that body language is more fundamental to communication
than speech.
35
2.4 Motion capture in the arts
2.4.1 Motion capture in visual media
From the late 1960s Myron Krueger started a series of groundbreaking
experiments in human/computer interaction that have become iconic in
technology and the arts. In 1970 he devised and directed a computer based
installation called Metaplay in which a participant could see their digitised
silhouette moving in real time on a rear projection screen. Movement of the
silhouette generated changes to the projected image. In one iteration the
person could draw lines on screen by moving his/ her finger; an open hand
would erase areas of the screen. Krueger used an artist hidden at the back to
draw on screen giving the person at the front the impression that the
computer was tracking their finger movement. (Krueger, 1989: n.p.) The
technique, while deceptive, was also a precursor of what lay in store for
computers in the future. In today's terms his set-up was a simple operation,
but Myron Krueger's experiments were conducted at the dawn of computer
graphics and provided a valuable insight into real-time interaction between
people and virtual space.
The experience during METAPLAY had demonstrated
that two people who saw their images juxtaposed
would interact as though they were actually together. It
also showed that people have a proprietary feeling
about their image. What happens to it happens to
them. If two people are together, they can see, hear
and touch each other. The ultimate consequence is an
artificial reality experienced through the participation of
one’s video image in the portrayed world. (Krueger,
1991: 37)
The first recorded motion capture system was developed by Lee Harrison III
in the early 1960s. Using potentiometers attached to a body suit he was able
36
to animate stick figures in real-time on a cathode ray tube attached to an
analogue computer (Carlson, 2007: n.p.). Much of the early history of 3D
motion capture was outlined by David Sturman in his paper, A Brief History of
Motion Capture, presented at Siggraph 1994. In 1980, Tom Calvert used
potentiometers attached to an exoskeleton that, in turn, was attached to a
person's leg to gather analogue data of human movement. Around the same
time, Polhemus developed a computer navigation system that used motion
trackers. Through triangulation among trackers, their positions relative to
each other could be established. By the late eighties several projects had the
animation industry paying close attention. Mike the Talking Head produced at
Silicon Graphics Studios was a digital puppet, a virtual head whose facial
expressions and movement were controlled in real-time by a puppeteer. This
concept was taken further by Jim Henson Studios in collaboration with Pacific
Data Images, and Waldo C. Graphic was created. Waldo was a computer-
generated puppet manipulated in real-time using a multidirectional hand-held
controller that manipulated body and mouth movement. The real-time footage
was composited with video so that Waldo was able to interact in real-time
with the physical puppets.
Influenced by this development, French company Videosystem developed a
character called Max the Ghost who appeared daily on television for three
and a half years. Other characters followed. Mario used the face tracking
system, Face Waldo, created by Simgraphics (1992). Brad de Graf
developed a face tracking system that was used for a real-time talking
spaceship. He also developed cartoon show host, Moxy the Dog, which used
electromagnetic tracking. The first full-length movie that incorporated motion
capture was Total Recall in 1990. Motion capture has refined and diversified
significantly since those early days and is now used regularly in popular
movies including Shrek and Happy Feet. The television series, Jane and the
Dragon, made extensive use of motion capture.
Being a medium that translates live performance into a digital form, it was
inevitable that motion capture would also be used as a way of digitally
mediating live performance. Projected images had been used for many years
37
in live performance dating back to at least 1911 when Loie Fuller used film
projection onto her costume in a Berlin revue. The first anthropomorphic
interaction between screen and stage was in 1914 in Gertie the Dinosaur.
Winsor McCay dressed in a costume reminiscent of a lion tamer, complete
with whip, “controlled” an animated dinosaur called Gertie. As McCay
bellowed orders Gertie the dinosaur would respond in the same way a
trained lion would. He culminates his performance by walking behind the
projection screen instantly appearing on the screen the same size but as a
stick figure, moving fluidly from real life to projected image. This
synchronisation between stage and screen has parallels with more
contemporary works using games technology to create interactive mediatised
stage scenarios.
2.4.2 Digital Performance
Theatre is not concerned with reality; it is only
concerned with truth (Sartre cited by Styan, 1981: 182)
This section chronicles live productions that use motion capture and does not
include performances that are primarily demonstrations of technology. Forms
of motion capture have been used in live performance since the eighties,
most commonly with dance. Coniglio and Stoppiello, who went on to form
Troika Ranch, used flexible rods to provide digital information on a dancer's
movements from 1989, developing software that is still the basis for what is
commonly used now for creating real-time visual effects. Dance companies
such as Riverbed, AlienNation Co. and Troika Ranch experimented with 3D
motion capture at a time when real-time use of motion capture for stage was
an expensive and technically difficult undertaking. Performances using
motion capture were uncommon and were often more a display of visual
technology than an evocative performance. Scott Delahunta stressed the
need to engage the audience more actively and to emphasise performativity
over visual display.
Since the mid to late 80's (with precedents established
38
earlier), some dancers and choreographers have been
exploring various interactive computer systems, but
their works tend to integrate these systems into
presentations in essentially proscenium-like settings
and not engage in open and participatory models
allowing the audience/user/viewer to cross the border
between performance space and spectating space.
(Delahunta, 2002: 114)
A very early performance using 3D motion capture took place when Riverbed
members, Paul Kaiser and Shelley Eshkar combined with choreographer
Merc Cunningham to create Biped (1999). This was a landmark performance
in which motion captured sequences were used to drive abstracted 'hand-
drawn' models of dancers projected onto scrim at the front of stage.
Ghostcatching, a digital dance installation, was a collaboration between
Riverbed members Paul Kaiser, Shellay Eshkar and Michael Girard, and
dancer/choreographer Bill T. Jones. A blackened room and ghostly hand-
drawn figures move about in projected three dimensional space tracing paths
as they move. These paths gradually fill the space and obscure the figures
creating them. Robert Wilson’s digital opera, Monsters of Grace also used
prerendered 3D motion capture sequences as the basis for visuals. This
blend of technology and dance was more art than performance, leading
Birringer to remark:
All of the Wilson productions I have seen, including his
collaborations with East German playwright Heiner
Müller, were marked by a depersonalized formalist
aesthetic of the painterly which, comparable to
Cunningham's choreography, emphasized technical
execution and precision, drawing lines in space and
filtering out all psychological and emotional
connotations (Birringer, 1999: 365)
Dance can more easily deploy visual technology than theatre because it
39
involves a unified sense of time in which the tempo of sound, movement and
visual elements can blend harmoniously. Programs such as Jitter, Max MSN,
Isadora, EyeCon, vvvv and EyesWeb are used commonly to create
mediatized dance performance. Western Australian troupe, Jambird, used
this type of technology to create stunning visuals in Metadance in Resonant
Light. Chunky Move incorporated visually captivating real-time imagery
developed by Frieder Weiss in their productions, Glow (1999) and Mortal
Engine (2008). These types of technology differ from 3D motion capture in
that the motion is captured in video as two-dimensional data. 3D motion
capture potentially offers more flexibility and more possibilities but it will be
years before it builds a similar artistic legacy in live performance.
ACCAD at the University of Ohio has been a hub for interdisciplinary
productions using 3D motion capture. Landing Place (2004) choreographed
by Bebe Miller and animated by Vita Berezina-Blackburn was a collaboration
among artists, performers and technicians, located in different parts of the
US. In the production, prerecorded motion capture sequences were used to
drive scenes, which were projected as visuals in dance. These sequences
were projected onto a semi-transparent scrim melding the animations and the
dancers into a visual unity.
In 2005 Paul Kaiser and Shelley Eshkar, and Michael Girard along with
composer Curtis Bahn worked with Trisha Brown and Bill T. Jones to create
the dance production, Motion-e. This was carried out as a broad
multidisciplinary collaboration through the Herbinger College of Fine Arts and
the Fulton School of Engineering. Unlike the earlier productions in the late
1990s, the stage was set up as an optical motion capture area so that the
dance onstage could drive real-time effects and sound. The real-time
element had a gentle atmospheric quality sometimes forming columns of
translucent light, sometimes radiant waves that moved in sympathy with the
dancers’ movements, a connection enhanced by the immediacy of the
visuals. Other productions that used motion capture include Point A to B
(2007) by UK based Urban Freeflow, performance visuals based on the
urban sport of parkour, a way of creatively and acrobatically getting from one
40
point to another in an urban environment, and Loops (2001-), a continuous
open-source digital artwork based on the movement of Merc Cunningham’s
hands.
David Saltz’s theatrical production of The Tempest (2000) was arguably the
first production to use full axis 3D motion capture in real-time onstage. Ariel
was played both as a live actor and a virtual actor. The actor was fitted with
electromagnetic trackers, essentially controlling the movement of the virtual
Ariel. This allowed for a consistency of motion and manner between the two
forms. The virtual Ariel was not in any way live, a creation of a 3D artist, but
Ariel’s movements, mannerisms and responses were inherently humanlike,
accommodating unanticipated variations onstage. Saltz concludes, “an age
of interactive, live media is upon us.” (Saltz, 2001: 21)
Real-time puppetry was used In Jorgen Callesen’s production of The Family
Factory (2002); a play that used animated alter egos of four family members
onstage. These animated doubles were rear-projected onto a row of screens
at the rear of stage. Each double was controlled onstage by the
corresponding family member. The technique involved the use of wired
electromagnetic sensors that could be rotated to control a range of
movements in the projected characters in real-time.
The productions after 2000 were carried out at a time when the fervour that
had surrounded the use of the latest interactive technology in live
performance waned. In many respects some of the promise of what digital
technology offered in the nineties as a tool in live performance fell victim to
overexpectation (Daniels, 1999: 7). The publicity surrounding new technology
often inflated the reality, and what was seen as achievable was caught in a
web of technological immaturity (Grady, 2003: 167). Also, movies and
advertisements were using sophisticated effects creatively with budgets live
performance companies could only dream of, so when effects were used on
stage they were often underwhelming or seen as passé.
41
Unfortunately, when an innovative new technology
such as virtual reality comes along, people often do
not distinguish between what it might be able to do
and what it truly can do. (Grady, 2003: 167)
It is early days in the evolution of real-time digital technologies and the
experimentation since the 1980s has provided the performance world with a
framework for how real-time visuals could work onstage. While some works
were pivotal and others fell short of their grand aims, all helped to shape the
parameters for successful mediatized performance. When devising visuals
for performance, it is useful to recognise a distinction between performance
and spectacle: spectacle appeals visually, whereas performance focuses
more on what is communicated. When media is introduced into performance,
performativity is easily compromised by the aesthetic and subsumed in the
image. However, most productions could not fully explore a performative link
between performance and projected image because they did not have
access to a real-time environment.
Many of the early technical limitations are being solved, allowing technology
to become an invisible element where only its effect is seen. Moreover, the
digital landscape comes closer to resembling reality as people’s reality has
shifted more closely to the aesthetic of the digital vision.
When concrete space and time is no longer an
untouchable basis on which the performance is built,
then other more abstract or functional manifestations
can become the new basics. This can then be
redefined with each new performance – and even
altered within a performance as part of the
performance narrative (Peterson, 2004: 40)
In 2008 there is a world of difference from the computing power, the
sophistication of software and the state of the technology that existed only
five years ago. Such is the speed of technological change that many earlier
42
works appear primitive in comparison to what now can be readily achieved.
Technology moves forward relentlessly in a self-perpetuating drive for greater
functionality and many of the early limitations are solved. The digital
landscape also comes closer to resembling reality as people’s reality has
shifted more closely to the aesthetic of the digital vision. The increasing
realism of 3D environments, the level of interactivity that can be achieved in
real-time are also opening up the options for exploring psychological and
emotional connotations through meaningful communication with an audience.
Digital performance itself has matured, focusing more on using technology
for artistic outcomes rather than stampeding headlong over the cutting edge
of its promise.
2.5 Conclusion
The real-time capabilities of motion capture open up new possibilities for the
way digital images can work with live performance. The contextual review
outlines the theoretical, historical and linguistic environment for practice.
Virtual reality, liveness, presence and paralinguistics are part of the web of
tensions that factor in the success of digital performance. Overarching post-
structural perspectives such as Debord’s spectacle, Baudrillard’s simulation
and Lyotard’s metanarrative contextualise the role of modern media,
shedding light on its nature and power of media, its ability to both persuade
and distract. Together they give insight into how virtual reality fits into the
physical world, how people relate to a screen environment and how visual
images can work in unison with the variability of live performance.
The way people communicate with each other involves physical signals more
primal than speech. Eye contact, mood, gesture and attitude, the primary and
secondary movements of the body, collectively called paralanguage, send
signals about our intentions, our status and our state of mind that are not
easily conveyed in words. The conveyed meaning is linked to the timeliness
of the action or response, making it difficult to re-create in media unless the
medium operates in real-time. The spontaneity of these body signals is
intrinsic to liveness in performance. With motion capture the spontaneity of
43
liveness can be reproduced in an animated environment, allowing for
sophisticated interaction between an audience and a virtual character. The
success of this interaction depends on whether the virtual environment
engenders a strong sense of presence in the viewer, and whether the viewer
feels co-present with the virtual character. Improvisation in performance uses
techniques that enable the performer to react in ways that are spontaneous
but also to incorporate the viewer in “constructing realities”. Improvisation is
inclusive, opening up performance to participation by the audience in a way
that allows them to influence the direction without wresting the control.
The contextual review presents some guidelines for how people are likely
relate to real-time performance:
Our sense of “presence” in a mediatized environment is enhanced by
the consistency of the virtual environment, by its familiarity and by the
nature and extent of the interaction.
The type of attention we give to live performance is different to the
attention we give to media.
Spontaneity is an integral element of liveness and is likely to strongly
enhance a feeling of presence when used in a virtual interaction.
The more realistic a screen world the more we tend to scrutinise it for
faults.
Experiences with new media in the nineties demonstrated the importance of
being pragmatic, not lured by the promise of what technology might do rather
than what it actually is capable of. In its totality, performance is the creative
outcome and technology is a medium for delivering it; so creative decisions
should be based on artistic intentions rather than demonstrations of
capabilities. This research is not just exploring real-time animation as an
augmentation of live performance but also the interaction between a person
and virtual character, what Steve Dixon describes as the “digital double”.
Research into presence indicates that communication is an important factor
in creating an engaging virtual experience. So the way in which a virtual
44
character communicates is just as fundamental to the success of a virtual
performance as it is to a live performance.
45
3 Methodology
...we have the habit of combining certain concepts and
conceptual relations (propositions) so definitely with
certain sense experiences that we do not become
conscious of the gulf, logically unbridgeable, which
separates the world of sensory experiences from the
world of concepts and propositions. (Albert Einstein,
1944: 289)
3.1 Research question
The research has been conducted from the perspective of a developer of
projection projects for live events. This research attempts to answer the
following question:
In what ways, and with what outcomes, can inertial motion capture be
used in live performance?
This question is explored as a practice-led case study comprising four
research cases that use real-time motion capture. Each investigates a
distinct way in which real-time animation can be used in a live performance.
Primary case
A Brush with the Real World: a real-time virtual artist.
Secondary cases
Chasing Shadows: a stage interaction between an actor and his
shadow.
Private Eyes: an instant movie experiment.
Motionics: an exploration of performance animation and dance.
46
3.2 The nature of research
The value of artistic research is related not only to the
products of creative arts practices, but also to
methodological, material and social processes through
which they operate. (Barrett, 2006:1)
Where practice-led research in the arts usually leads to the creation of an
artifact, such as a painting, a play, a game, music, a poem, there are
questions as to what aspect of this creation constitutes research. Does the
research relate to the artifact, the technique, the process of creation, the
tools used in the creation, or how any of the above impact culturally,
academically or artistically? Can research simply open up new pathways to
further research? Is the new knowledge gained through research an instance
of “knowing how” or “knowing that” (the Ryle distinction)? Does an outcome
need to be functional and what constitutes function?
Practice-led research in creative arts has been subjected to widespread
scrutiny as to how it fits into research. In general, research is undertaken to
add to what is known. The Australian Department of Employment, Education
and Workplace Relations further elucidates on the nature of research:
1. Creative work undertaken on a systematic basis in order to
increase the stock of knowledge, including knowledge of
humanity, culture and society, and the use of this stock of
knowledge to devise new applications.
2. Any activity classified as research and experimental
development is characterised by originality; it should have
investigation as a primary objective and should have the
potential to produce results that are sufficiently general for
47
humanity's stock of knowledge (theoretical and/or practical) to
be recognisably increased. (DEEWR, 2007: 8)
While methods used for practice-led research in the arts do not easily
conform to quantitative and the broad norms of qualitative research, the
boundaries that have confined creative practice as a method in research are
dissolving as the more rigid disciplines, such as the sciences, are finding it
increasingly necessary to diversify their methods to include more creativity
and extrapolation. The dynamic reordering in the structure of knowledge as
old fields merge and new fields are created has led to a situation where not
only have the arts embraced research techniques used in science (in
particular, technology) but science has embraced arts methodologies to help
solve scientific dilemmas. Hassan defines four areas of contemporary culture
characterising this flux:
The creative process in science and art
The new twilight zone of experimental science
The incorporation of technology into the arts, both as theme and form
The existential search for a unified sensibility
(Hassan cited in Dixon, 2007: 150)
While traditional research usually extracts new knowledge from what is
known, there is increasing recognition that knowledge is gained in the
process of creating. There is a blurring of the edges between art and science
and for arts practice this has led to a greater emphasis on being able to
express the methodologies of arts practice.
3.3 Postmodernism
Duchamp really abandoned the idea of making art to
take up the role of judging it and mocking it, he
opened up the door for a great deal of what we now
call Postmodernism. (Kramer, 1999: 131)
48
Common throughout postmodernist theorists is the recognition that a
reductionist approach to enquiry was inadequate; this approach leaves a trail
of exceptions that diminishes its authenticity. In culture there are no simple
answers; the search for knowledge and understanding is obfuscated by
pluralism and the indeterminate nature of truth. Postmodernism focuses on
representation and on the gaps in cultural narratives. In research it is
recognised that what is seemingly verifiable could not be described without
using language that was unverifiable. Everything is relative.
Postmodernism highlights the premises on which knowledge claims are
made by exposing the complexity of the seemingly simple, drawing attention
to the hidden inconsistencies and exceptions that undermine social
metanarratives, exposing the plurality of these social models and recognising
the subjectivity of knowledge claims. Any objective determination is based on
a subjective acceptance of a procedure towards its validity. So
Postmodernism does not logically undermine all authority or negate all
knowledge. Nietzsche rejected the nihilism of many postmodernists,
determining that we should strive for unified subjectivity. When Lyotard drew
attention to the metanarratives that preserve the power of the status quo he
also advocated a new focus on social justice.
So deterministic, empirical, rational, logical conclusions can be drawn
provided we recognise that any conclusions are an acceptance that there are
no absolutes, that knowledge is ultimately based on subjective interpretation.
As human beings we are a part of a world that we may
also observe. Therefore we can only perceive it from
the inside. (Weibel cited in Druckrey 1996: 344).
Art as a field is highly subjective. Rules are meant to be broken. The artifact
has no intrinsic value beyond the value placed upon it by the artist or the
viewer. It is not based on the materials used or necessarily the skill of the
49
artisan. It has value as the implementation of an idea. In mediatized
performance, a juncture where art and technology meet, there is a mix of
methods, some stemming from creative practice while others are rooted in
the more deterministic logic of software. The controlled chaos of creativity
often mixes with hypothetico-deductive reasoning in the pursuit of answers
that can be visualised but may or may not be possible. Postmodernism being
relative, pluralist and subjective is a sufficiently diverse umbrella to cater for
what is, in essence, an art/science fusion with a socially defined value.
The postmodern artist or writer is in the position of a
philosopher: the text he writes, the work he produces
are not in principle governed by pre-established rules,
and they cannot be judged according to a determined
judgement, by applying familiar categories to the text
or the work. Those rules and principles are what the
work of art is looking for. The artist and the writer,
then, are working without rules in order to formulate
the rules of what have been done. (Lyotard, 1979: 81)
Steve Best and Douglas Kellner (2000: n.p.) expand from Debord’s
situationism to contextualise interactive media and how it relates to the “the
spectacle”, distinguishing between “genuine interactive spectacle” and
“pseudo interactive spectacle”, the essential difference being that one is self
created and open, such as one’s website or a chatroom, whereas the other is
moderated, such as radio chatshows or email feedback, where opinions and
comments are sifted through a policy filter. “Interactivity” has become a
buzzword of the digital age but is it an ideal? The internet, for instance, offers
new ways for like minded people to organise and effect change, but there are
such volumes of opinion being created digitally that much of it ends up
spiraling down the cyber plughole or, otherwise, its relevance is diluted by the
sheer quantity of other material. Nevertheless, the ability to actively engage
rather than compliantly absorb is empowering because the internet user
becomes a participant.
50
Performance is structured play. It is a “testing ground where possible events
are tested out in a ritualised framework, so that we can weigh their
consequences without our actions becoming reality or having real
consequences.” (Peterson, 2004: 40). Interactive performance allows others
to play in the “testing ground”. The innate interactivity of real-time motion
capture offers a new way to structure play, and research based on practice
can help determine the parameters.
Postmodernism in performance is, as it is understood
today, a dramatic form that is based on a semiotic
understanding of traditional theatrical practice. The
semiotics (or visual signs) are frequently
deconstructed (taken apart) thereby opposing their
original interpretation and leaving them open to
translation. In this style of theatre the performer is
more of an artist, seldom a character like an actor, and
the content rarely being traditional in format or
presentation. (Hurstfield, 2008: n.p.)
When using technology in research there is an underlying assumption that
whatever its limitations in a particular context, these will be overcome with
the introduction of the next model. With the industrial revolution and the
growth of twentieth century capitalism came increasing pressure for science,
not just to provide knowledge about the natural world, but also to alter the
world. Technology is undergoing a runaway evolution in an economic
ecosystem that thrives on increasing efficiency and rapid turnover of
commodities. Information has itself become a commodity, and differentiation
between information and promotion is gradually disappearing. So technology
represents a landmark shift in the direction of science from a purely
inquisitive activity to a performative activity.
In the late twentieth century, however, science
occupies a crucial post in the formation of the
postmodern condition. This is less because of the
content of scientific ideas than it is due to the
51
increasingly powerful economic situation it enjoys
under a capitalism that has become wedded to
technological advance. Science can no longer afford
to be the disinterested quest for the truth about nature,
since its economic survival is bound up with the
production of new technologies: science cannot simply
desire to know; it must perform. (Hamilton-Grant,
2001: 59)
So when assessing a new technology it becomes, not just a review of its
function but, incidentally, a guide to its marketability. As scientific enquiry, in
a postmodernist sense, is no longer recognised only as an impartial
acquisition of knowledge but as a moderated activity whose direction and
application are influenced by social, political and economic priorities, so the
way in which we use and promote new technology influences the course of
innovation.
52
3.4 Performative research
The best way to predict the future is to invent it.
(Ferryman in Thacker, 2000:158)
It is difficult to address the question at the heart of this research through
purely qualitative enquiry: there are too few precedents from which to draw
any conclusions, and the most important outcomes are derivatives of practice
rather than interpretations of practice. The research centres on discovering
pathways to performance rather than assessing performance. These
pathways are based on well-founded animation techniques that, through
recently developed technology, can now be used performatively in live
performance situations, with quality and stability approaching a broadly
acceptable level. So the research focuses on the performative elements
derived from a combination of the two artforms, animation and performance.
Quantitative research is also inappropriate because the projected outcomes
are intrinsically tied to the emergent processes of creative practice and the
outcomes are neither measurable nor conclusive. However, there are
methods in both quantitative and qualitative research that can be used in a
practice-led approach. This study combines the mathematically derived field
of software, with its clearly defined and often inflexible procedures that are
sequential and repeatable, with the emergent approaches underlying artistic
investigation. This is a multi-arts investigation that uses a visual medium to
explore live performance using elite technology. Elements of the research are
quantifiable; others are interpretive, and so the research can draw on the
methods of both quantitative and qualitative research where appropriate,
along with methods derived through practice, under the umbrella of
performative research.
When research findings are presented as such
53
utterances, they too perform an action and are most
appropriately named Performative Research. It is not
qualitative research: it is itself. (Haseman, 2006: 100)
As pointed out earlier, not only arts practice, but technological innovation
itself is performative, distinguishing it from more traditional areas of science.
It seems an anomaly that human investigation has taken a seemingly
irreversible turn towards performativity and yet performativity is still struggling
to be recognised as a valid paradigm in research. This process of research
brings a product or a set of conditions into existence that did not exist before.
It uses performative methods of inquiry in that the research procedure is not
hypothesizing and proving nor investigating and hypothesizing so much as
adapting to the outcome of each step in a process that may lead the research
in an unforeseen direction. The investigations are expressed in the language
of practice and any determinations are intrinsically tied to the artifact(s)
produced and drawn from the techniques used. The methods used are based
on achieving experiential outcomes rather than reaching conclusions from a
set of opinions or subjective perspectives. First hand interpretations of an
artifact, such as a play or performance installation, can involve some
sensations that are ineffable, and will lead to a “plurality of answers” that may
be, in a Wittgensteinian sense, limited as validation. Also, “[a]udiences who
will be satisfied with the outcomes of research share a context with the
researcher“ (Biggs, 2002: 10) limiting the generality of qualitative outcomes
as a knowledge claim.
The few examples where real-time motion capture has been used in
performance do not provide a sufficiently broad understanding of the
processes that underlie its use. Few people in live performance have been
exposed to motion capture and so it would be hard for them to envisage how
motion capture can apply to performance. To be useful and understandable
to choreographers and devisers there needs to be more generalised
research that gives an overview of what the technology can do and how this
can be applied to performance. So it is from the practice rather than product
that new knowledge is likely to come. These are the techniques, tools and
54
processes gained from the experience of practice (Mafe and Brown 2005:
12).
3.5 Practice-led research
Pierre Bourdieu argues that tacit knowledge and the
alternative logic of practice underpins all discovery, yet
the operation of this logic is often overlooked because
it is subsumed into the rational logic of discursive
accounts of artistic production. (Barrett, 2003: 3).
Practice is the language of art and through practice new forms and pathways
emerge that feed the creative process. It is through practice that ideas are
investigated, hypotheses are tested and new questions emerge (Burdick
cited by Mafe & Brown, 2003: 82). Practice as research in its various forms,
while affirming its place in academic institutions is widespread throughout the
creative industries. These include movies, games, animation, print media,
advertising, fashion and web design. Along with the use of practice as
applied research, an emergent approach is a prevalent strategy driving rapid
cultural and technological change.
Although both applied research and emergent forms of research tend to use
practice as the research framework, the two approaches are different.
Applied research generally has an identified aim that helps to develop or
extend functionality whereas emergent research is more akin to prodding a
stick at the unknown to see what happens, in an informed and skilful way of
course. Brown, Gough and Roddis (2004: 5) differentiate the two by claiming
that practice-led research is a methodology whereas applied research is a
type of knowledge. It could be argued that research into technology-driven
arts projects necessarily involve both practice-led and applied research. The
methodologies used in artistic approaches are considered necessarily
emergent (Barrett, 2006: 7) in that the path and destination adapt to the
journey, and so the artistic aims of the research change with the process of
discovery. On the other hand computers are commonly used for artistic
55
outcomes and the creative process can still be emergent but the structured
logic of software with its adherence to sequence and its repeatability also
requires more deductive methods. There are many variables at play in arts
practice and the outcomes are often unique and unforeseeable so deductive
and inductive methods can only ever be a part of the research process. This
is certainly the case in performance where structure is constantly in tension
with individual expression, making portrayals of human interaction a complex
and interpretive interplay of factors. When this is combined with highly
evolved technology into a form that is related only to practice then any
meaningful insight can only be gained through practice. MacMurray stresses
the importance of practice underpinning knowledge:
...most of our knowledge, and all our primary
knowledge, arises as an aspect of activities that have
practical, not theoretical objectives: and it is the
knowledge, itself an aspect of action, to which all
reflective theory must refer. (MacMurray, 1957: 12)
Mafe and Brown (2006: 1) conclude that the significant outcomes of practice
in research are based on knowledge obtained through practice rather than
the product of practice. The chosen investigations place greater emphasis on
the development and application of techniques and processes that are useful
in the broader context of the arts in priority to the artifacts produced. This is
consistent with the emphasis by the Australian Department of Education,
Employment and Workplace Relations on generalized outcomes that are
transferable; it also reflects the need to extend beyond the mere application
of practice to “improvement of practice” (Haseman 2006:98) so that:
…the emergent core of practice-led research produces
a value in excess of any single artifact or document
produced. (Mafe and Brown, 2006: 11)
The works, therefore, need to provide outcomes that are useful and useable
to other people in the field. With emerging technology that has the potential
56
to interface with creative arts, there are opportunities not just to create new
works but to discover new ways of working in a creative context, ultimately
offering gateways to more discovery.
3.6 Investigative framework
A set of smaller performance applications were chosen in preference to a
large production as the practical basis of the research for several reasons.
Firstly, the software has many features and, in a single production with the
focus on performance, only a fraction of these features was likely to be used.
With a case study that includes a representative selection of cases, a much
broader cross-section of techniques could be investigated, because each
case can be designed with a unique set of techniques in mind. Secondly, a
full-scale production would require a collaborative pool of expertise, and
people with experience in motion capture are hard to come by; using a case
study does not require the same level of collaboration. Thirdly, there are few
precedents for using motion capture in performance and so there are likely to
be plenty of unforeseen hurdles, some of which may not be overcome.
Placing all eggs in the one basket could lead to a situation where a lot of
energy was expended for no final outcome. However, if one case cannot be
implemented then it does not risk the outcomes for the entire research. The
cases used in the research were designed around the attributes of inertial
motion capture that might adapt well to performance.
It is a real-time medium and so is projected as it is performed
It uses full axis 3D data, which theoretically allows it to use the vast
realm of 3D techniques including effects, behaviour and physics
It can simulate the nuances and spontaneity of paralanguage
It is unrepeatably interactive in real-time, as distinct from games
interaction which uses a network of preanimated actions arranged in
the form of trigger trees, along with programmed behaviours and
simulated physics, giving an impression of realness
It can visually represent conversation in real time
57
For the case study to be of value as performative research, the design of
each case needed to focus on each as a means to a performance rather than
as a technological curiosity, and so motion capture is a medium and not an
end. The ability to create real-time interaction is an important focus in the
case study as it sets up a three-way communication channel among the
actor, screen and audience. Spontaneity is its strength. With motion capture,
movies can be created that have a level of variability and responsiveness not
found in other interactive media. The technology is still evolving but it heralds
a plethora of possibilities that may change the way we perceive arts and
entertainment.
This research was set up from the early stages as an “enabling” industry
partnership with visual technology company, VRS, and much of the technical
equipment and logistical support through the investigations was based on
this arrangement. The software engineer at VRS took an active interest and
spent time in the early stages helping me with the motion capture software.
However, his time was limited so it was necessary to be self-sufficient,
learning the software and solving technical hurdles involved in each
investigation. In the latter stages, once the technical aspects were mastered
and real-time environments were at a stage where performance-based
experimentation could begin, several performers were invited to work in a
number of collaborative movement/dance improvisations to determine the
types of movement that worked best with motion capture and to uncover new
possibilities.
The overall innovation performance of an economy
depends not so much on how specific institutions
perform, but on how they interact with each other as
elements of a collective system of knowledge creation
and use, and on their interplay with social institutions
(Johnston cited by Grigg, L, Johnston, R & Milsom, N
2003: 15)
58
The research is interdisciplinary involving performance, animation and visual
technology. Interdisciplinary research is, according to Julie Thompson Klein, “
the necessary ‘churn’ in the system”. It taps into “a set of dynamic forces for
rejuvination and regeneration, pressure for change and the capacity for
responsiveness” (Thompson Klein, 2000: 22). An interdisciplinary approach
tends to prioritise the problem ahead of the artificial categories of knowledge
organisation. Once the problem is recognised experts from relevant
disciplines can then collaborate to solve the problem. However,
interdisciplinary research does not need to be collaborative (unlike
multidisciplinary) so long as the researcher conducting the research has the
skills and the knowledge to solve the problem. The rapid pace of innovation
is driving structural changes in the way knowledge is organised and the way
research is conducted. Across many areas of research, multidisciplinary,
cross-disciplinary and interdisciplinary approaches involving collaboration
have become the norm. Convergence and diversification reflect the state of
the flux in research driven by the growth of the knowledge industry and the
rapid evolution of digital technology. The growth of computing power and a
widespread acceptance of some of the major tenets of Postmodernism have
opened the way for the exploration of complex problems that transcend
traditional disciplines.
Over the course of this century, metaphors of
knowledge have shifted from the static logic of a
foundation and a structure to the dynamic properties of
a network, a web, a system, and a field. Perceptions of
academic reality, though, are still shaped by older
forms and images. (Thompson Klein, 2000: 22)
59
3.7 Methods
3.7.1 Case study
If you tell me, I will listen. If you show me, I will see.
But if you let me experience, I will learn. (Lao-Tse
cited by Clarke, 2000: n.p.)
Lao-Tse’s philosophy on learning became the basis of a method of enquiry
practiced by ancient philosophers, Confucius and Han Fei-Tzu, that is now
referred to as “case study”. Case study is a method often associated with
sociology but has been used extensively as a research method in many
areas of research, from business to nursing. Stake (1995) and Yin (1994)
identify sources of evidence for case study that include documents, archival
records, interviews, direct observation, participation observation and physical
artifacts.
The case study, I argued, is best defined as an
intensive study of a single case (or a small set of
cases) with an aim to generalize across a larger set of
cases of the same general type. (Gerring, 2007: 65)
Yin (1993) identifies three types of case study: exploratory, explanatory and
descriptive. The case study used in this research is exploratory in that, while
the projected outcomes are clear, their achievability, effect and function are
not verified until the case study is complete. As demonstrated by the works of
Darwin, Freud and Piaget, case study is an effective methodology for
exploratory research. This is in part because a case study is more likely to
expose causal mechanisms than, say, cross-case research. Case study is
useful when detailed knowledge of a small sample is more insightful than
targeted knowledge of a large sample. It is therefore an appropriate research
method for a topic of study that is complex and has few representative cases.
Much of what has been written on case study as a method in research
60
relates to its use in the social sciences where qualitative determinations can
be drawn. However, the method can apply just as well to performative
approaches, drawing conclusions, not from social contexts but from
processes in practice. Case study is generally an empirical process that, in
this research, can be based on observations of the processes, functions,
techniques and procedures as well as in observations of the performance.
As the case study is also the creative practice, however, these results are
performative. That is, the creative process towards a practice-led outcome is
the research framework, and the findings are in the language of practice.
Grounding the research through practice-led enquiry was necessary because
there were insufficient examples of its use in the field from which to draw
conclusions by purely empirical means. Also, the creative practice covers a
whole range of software attributes in combination with a high level technology
to map a creative performance framework. This is a complex blend that is
hypothesis-generating rather than hypothesis-solving (Gerring, 2007). The
case study is not representative of a population of similar existing
performances but provides a useful set of templates for future creative
endeavours.
Each investigation is designed to reach separate performance objectives but
within a framework common to all; it is the extent of all observed and
performed outcomes from all of the investigations that are used, in a holistic
sense, to map a creative landscape. The case study focuses on the
functionality of inertial motion capture as a medium only, and the
performances resulting from the use of that medium affirm its suitability. The
standard functions of the software are not objects of study per se, as these
have been developed long before to carry out these functions. What is
relevant is the context in which these functions are used to meet a creative
objective.
Case study is less of a methodological choice than a
61
choice of what is to be studied. (Stake 2000: 435)
The research is structured as a practice-led case study comprising four
cases. Each case investigates a different performance scenario and each
sheds light on the processes, the attributes and the performance outcomes,
including ways in which this medium can be used to create a performative
interaction between the world of screen and the real world. The case study
comprises a major case that explores interactions between a real-time
screen character and audience, and three minor cases, each exploring a
different interactive dynamic.
The relationships discovered among different
elements of a single case have a prima facie causal
connection: they are all at the scene of the crime. This
is revelatory when one is at an early stage of analysis,
for at that point there is no identifiable suspect and the
crime itself may be difficult to discern. (Gerring, 2007:
41)
These cases help to determine ways in which motion capture can be used at
live events, and give an indication as to whether it is ready now for a broader
role in live situations and whether it is likely to become widely used at live
events in the future. As there were precedents to draw from, the aims of the
research are broad, determining in general terms the technology’s suitability
to live performance as well as exploring pipelines to creative output, the
factors influencing this type of mediatised performance and a spread of
creative possibilities. Nevertheless, it was important to gain sufficient insight
to be able to construct, or at least know how to construct, real-time
narratives. As there were few existing examples to draw upon and these
together did not cover a broad range of uses, the cases were devised with
the aim of representing an overview. The cases, therefore, needed to be
diverse and general in order to explore:
62
the dynamics of hybrid performance
animated environments that were conducive to live performance
ways of engaging and intercommunicating with an audience
the strengths and limitations of the software and hardware
presentational considerations such as the visibility of the technology
strategies that avoid the splitting of focus
differences in the way an audience perceives onscreen performance
compared to live performance
the use of light in a situation where the two forms have conflicting
requirements
In a performative sense the case study is not just based on observing
phenomena to find causal mechanisms, but also envisaging effects and
determining how to cause them. It is a similar situation with process-tracing
which, in a performative case, becomes process-constructing. So, rather than
observing existing cases and examining them for causal links, the cases are
constructed, and it is in the process of construction that experiential
knowledge of the creative processes of practice is gained and causal
mechanisms are understood. In this respect, the case study is the
construction of the cases and this process of construction exposes new
knowledge.
Each behaviour has a side effect and the sum of the
side effects gives the desired functionality (Steels,
1991: 454.)
The case study follows the “diverse” model in that each case differs from the
other cases so that, in combination, the cases cover “maximum variance
along relevant dimensions” (Gerring, 2007: 97). It is through the diversity of
the cases representing maximum variance that an overview of the processes,
63
as well as the causal and performative mechanisms can be framed. The
experience gained can be used to construct more complex and
comprehensive projects such as complete “live” movies that can be
manipulated at any moment in a story: participatory cartoons in which
members of the public can join in as a character: or virtual installations that
are a hybrid of animation and reality television built on spontaneous
interaction and a heritage of improvisation. Each case in the case study
explores a broad range of software features that are likely to be useful in a
live performance scenario and these are listed below:
A Brush with the Real World: a real-time virtual artist. Interactive installation
Character interaction using voice, expression and body language
Movement and dance using a Digital double
Painting using a proxy
Gestural interface
Spatial triggers
Virtual improvisation
Participatory animation
Chasing Shadows: a stage interaction between an actor and his shadow
Mediatised theatre
Shadow puppetry
Swapping between recorded and live action
Theatrical interplay between physical and virtual character
Movie clip triggering
Private Eyes: an instant movie
Interactive movie
Real-time acting
Multiple characters in the one scene
64
Complex real-time environment
Cinematography
Atmosphere
Motionics: exploring techniques for creating real-time 3D visuals for dance
Multiple virtual bodies
The use of video in animation
Dynamic camera effects
Reactive scenery
Targeted projection
Motion trails
Interpretive and liminal bodies
Virtual lighting effects
3.7.2 Creative practice
Practice-led research builds on the fact that creative
practice has always been a knowledge generating
activity. (Mafe and Brown, 2006: 10)
Creative practice in research needs to be differentiated from the normal
practice of the artisan and have identifiable goals. The outcomes need to be
accessible and open to scrutiny and the structure of the research and the
written exegesis need to be clear in theorizing, explaining and contextualizing
the research and its outcomes. The outcomes should be sufficiently general
to be useful beyond the scope of the research and be able to be applied to
other research (Mafe and Brown, 2006: 4). This formal approach to
scholarship expected in research driven by a rationale of consistency and
order is in contrast to the more chaotic and reflexive approach typical of arts
practice.
65
The primary research methodology is problem-solving
itself, which often, in practice, cannot be fully
explicated. There are, of course, methods, techniques
and tools that can assist the researcher, but in many
cases, the problem finding and solving processes
retain an element of the black arts. (Scrivener, 2000:
n.p.)
The practice uses both animation and performance as well as sophisticated
emerging technology to bond them into a coherent performance framework.
Creative practice, in this convergent structure, requires a range of methods
as each area has its own unique workflows. Firstly, the design of each
investigation looks at the attributes of real-time animation and the unique
strengths of live performance to formulate a common ground upon which to
create. The animation is devised, storyboarded, characters modeled,
environments built, and, in parallel, the performance is devised and
developed through rehearsal and improvisation. Through motion capture the
animation and live performance are brought together regularly to ensure that
they work in combination.
The methodologies of creative practice are generally considered to be
emergent (Barrett, 2005: 6). However, I would argue the approach could be
either emergent or conjectural. In an emergent approach to creative practice
the creative pathways adapt to unexpected insights that may not be part of
the original intention. With a conjectural approach to creative practice the
creative pathways adapt to unexpected insights but without altering the
original intention. The emergent approach is led by inspiration, or what
Gerring calls “lightbulb moments” (2007: 5), while the conjectural approach is
visionary, envisaging a creative outcome and working towards it. In this
respect the approach to creative practice used in this research is conjectural,
using emergent methodologies.
Performance and dance were used in the research for testing the virtual
environments and characters as well as developing the performance
66
framework. Improvisational techniques were particularly useful in helping to
formulate ways of interacting with an audience that would emphasise the
interactivity of the virtual environments in performance.
We rarely want “a” motion, but rather want “the”
performance. The specifics of the performer and
performance are important. Just as actors are trained
to know that there is no such thing as a “generic”
movement, the sense of character, context, and intent
should be conveyed in movements. Just as with stage
and film actors, motion capture directors work with
performers to achieve what they want to see. Good
motion capture preserves this. (Gleicher and Ferrier,
2002: 2)
Each case had a specific performance style. A Brush with the Real World
uses improvisational techniques to lure passers by and involve them in visual
and verbal exchanges. These techniques stem from the work of Canadian
Keith Johnstone whose book, Improvisation, sparked an interest in ways to
deal with spontaneous performance situations, and my years of working with
Tony Kishawi using mask, mime and clowning techniques. Chasing Shadows
is built on interplay between two onstage characters, one real and one virtual.
It is a choreographed sequence, and motion capture is used to synchronise
movement between the actor and the virtual character. In traditional clowning
(as with Commedia dell’Arte), differences in status present fertile ground for
comedy sequences, and this piece explores a change in the normal
relationship between a man and his shadow, the actor and the virtual actor.
The other theatrical piece, Private Eyes, uses reflective monologue and is
more subtle, sober and mood evoking. Onscreen, this piece is in the form of
a movie, using a progression of scenes and camera positions, along with
rehearsed movement and gesture. Motionics explores a range of visual
techniques that could be used with dance, and dance was used to test each
technique in a performative way. Each environment has its own flavour, and
67
dance is used improvisationally to help determine a movement style or
dramatic feel that suited each of the visuals.
3.7.3 Industry input
The role of knowledge in economic development has
been transformed from a minor player to a driving
force. The industrial economy, based on goods and
services, is being matched, and in some cases
displaced, by the global knowledge economy, based
on the production, distribution and use of knowledge.
(Grigg, Johnston & Milsom, 2003: 15)
Industry input was sought initially for determining what the research question
should be. I had accumulated a number of ideas for using animation at public
events and, rather than rely on informed intuition as to which was most
appealing as a public project, I presented the ideas as a folio of prospective
projects to a selection of people involved in performance and festivals to get
feedback on the type of animated production they would envisage as being
successful. On the basis that people may not be aware of some technologies
available and ways in which these technologies could be used, I created nine
Flash animations, each demonstrating a way of using projection at live
events, including some that used live performance. Animations representing
hybrid shadow puppetry, performance animation, previsualisation and 2D
event projection evoked the most interest. One person with a long
involvement in circus was keen to discuss the use of virtual simulation as a
method of trialing a potentially dangerous risk to the performer. This method
is already used for defence, emergency services and pilot training and, if
applied to circus, it could help drive innovation. New acts that are potentially
dangerous could be simulated first, minimizing the risk factor. In combination,
the industry feedback helped identify a practical case(s) that had industry
application and could be used as the basis for research.
Below are images from the folio of Flash animations presented to people in
68
key arts groups for feedback.
Figure 3: Using rear projection animation as a public window installation
Figure 4: Large-scale shadow puppetry using animation
Figure 5: Using effects triggered by movement
69
Figure 6: Projected background moves with person’s stage orientation
Figure 7: Projection within sets onstage
Figure 8: Animated shadow
70
Figure 9: Shadow projection on the tent wall in circus
Figure 10: Improvisational screen character
3.7.4 Applied methods
There is an increasing role for applications-driven
approaches in many spheres of research, and a
growing share of the national research effort is
occurring in commercial settings. These trends are
particularly pronounced in the fields of science and
technology research, but are by no means confined to
them; similar trends can also be observed across a
range of the social sciences and humanities. (Kemp,
1999: 1.10)
The investigations each had fundamental technical challenges and,
frequently, new solutions to problems created new problems to solve. Each
time a new workaround was developed it was tried with the inertial suit to
determine the effect and what unforeseen consequences might be caused.
71
Some problems were suited to a holistic approach whereas others needed to
be isolated because it was too hard to determine the mechanics of the
problem otherwise. While technical issues were very important to resolve, the
theatrical parameters also needed to be determined; these were limited to
the scope that the technology allowed, as well as the way audience and
actors related to screen environments. Many of the technical hurdles and
artistic aims could be trialed to gain a better understanding of how they
worked. Some of these were:
Developing a gestural interface to operate a colour pallet
Tracking anomalies in movement data
Exploring ways in which effects could be used
Determining improvisational techniques and theatrical strategies that
would prove effective in holding people’s attention
Assessing how people would react to a real-time screen character
Working out the dynamics between a live actor and the virtual
environment
Identifying moments where the continuity of dynamics or aesthetics might
be broken
The flux caused by rapid innovation as different fields converge frequently
pushes researchers into pockets of unfamiliar territory. Through the
collaboration with visualization hardware company, VR Solutions,
opportunities arose to apply and expand on the skills and knowledge needed
for the research in a real-world context. On a number of occasions early in
the research I worked as the performer in motion capture sessions and, after
becoming more familiar with the motion capture software, demonstrated real-
time motion capture at a number of games studios and at the GamesConnect
Conference at the Brisbane Convention Centre. Later I worked on motion
capture for a UQ/industry collaborative project, creating and editing motion
capture data. A new beta level ultrasonic tracking system was sent from the
UK to augment the inertial system and I was employed to help set up the
hardware and software. Each of these experiences broadened my knowledge
72
of motion capture and improved workflow in practice. In combination they
provided an overview of the entire process in the context of industry
workflow, providing know-how not readily available in tutorials and books,
and were valuable in reaching a level of competence to be able to complete
the investigations. Late last year I traveled to Perth to set up motion capture
sessions for a major games company. Since then I have regularly been
involved in demonstration and content creation with motion capture. In
August I attended Siggraph 2008 in Los Angeles where I was able to see first
hand new hardware and software developments in motion capture as well as
aligned technologies and presentations of the latest research. Each of these
industry-based experiences has provided critical learning experiences in time
based workflows and an overview of the entire process of motion capture.
3.8 The working environment
The inertial motion capture suit used in the research is the Gypsy Gyro 18,
designed and produced by UK based Animazoo. It is an inertial system that
uses 17 inertia cubes placed at strategic points around the body. The inertia
cubes, made by Intersense, use accelerometric, magnetic and gyroscopic
data to determine 3 axis orientation (pitch, roll and yaw). These small cubes
are attached by leads to a transmitter worn on the chest or back. The
transmitter sends data to a receiver, which is plugged into a computer. This
Gypsy Gyro was the first commercially available suit produced and was
shipped to Brisbane in late 2004, just before the start of this research. The
software used for the motion capture was Animazoo’s Cobra 7.11 and
Autodesk Motion Builder 7.5.
The most widespread type of motion capture is optical motion capture. It is
commonly used to create animation for movies and games. Optical motion
capture typically requires dedicated studio space and this limits its application
to live situations. Inertial motion capture, while not as accurate as optical and
73
subject to drift, has no problems with obstructed sitelines (occlusion) and the
unedited real-time data is acceptable. The suit, fitted with a transmitter and
cables, is marginally restrictive but it allows the performer to operate
wirelessly hundreds of metres away from the receiver. A big advantage is its
portability. The entire system, including laptop, can fit into a medium size
case. It can be up and running within half an hour and can be used in a wide
variety of environments.
74
4 Description of practice
4.1 Overview
The practical research explores four cases in which motion capture can be
applied to live situations. These examples primarily focus on interactive
relationships between the actor and the audience or the actor and the visuals
rather than the artistic merit of the visuals.
(1) Interactive installation
(2) Mediatised theatre
(3) Instant movie
(4) Interpretive visuals
Many of the techniques in these cases are well known to those who work
with animation. Applying these techniques using motion capture to create
real-time animation is less well known. There are a number of well-
documented examples of real-time animated characters being used in a
public forum; these are typically real-time applications by animation
production houses rather than hybrid performances by live performance
troupes. This difference in emphasis is important in determining how the
subject matter is approached and how the technology is used. In live
performance the “liveness” is primary and animation is an augmentation,
whereas in animation the emphasis is on the animation, and the liveness is
an attribute that helps the workflow but is disconnected from the final product.
The implication that programming is the actual art and
what artists make of it will always remain secondary
matches up with Friedrich Kittler’s suggestion that only
our ignorance makes us confuse the products of
media with art. (Daniels, 1999: n.p.)
75
For motion capture, or for that matter any form of media, to have a
connection with live events it needs to be more than a fusion exercise. It was
important to explore composite forms where motion capture was a tool rather
than the focus of the visual experience. Otherwise its interest may lie only in
its novelty. A performance can “wow” an audience with cutting edge effects
that are only captivating until the next technological breakthrough consigns it
to obsolescence and, with the frequency of innovation in visual software and
exponential increases in computer speed, it is inevitably a fate just around
the corner.
Chasing Shadows and A Brush with the Real World are theatrically based
installations that use animation techniques applicable to dance. Both in
different ways explore how a projected story can be created live, including
how scene changes can occur, how a scene can operate with more than one
character and how the characters can interact with objects in the scene. The
installations are both highly portable, and require few special conditions
outside those that would normally be expected when using projection. The
performance is shown as it is created and so the content cannot be edited.
There were often glitches but the chances of them occurring or being seen
were minimised in the choreography and camera perspective. The
technology presented opportunities to develop or enhance ways of
interacting and this was the primary consideration when looking at how these
cases were constructed. Although there was a need to adapt to the
limitations and to frequently use workarounds, there was an assumption that
any problem could be solved and any inadequacies of the hardware and
software would gradually be eliminated as the technology improved; this
often proved to be the case.
76
4.2 A Brush with the Real World
Figure 11: A Brush with the Real World
A Brush with the Real World is a real-time performance installation. The
screen character, Rupert, is an artist and the screen is a window into his
world. He works in his small studio painting on his canvas but Rupert can
also paint in the air around him. His brush becomes his main form of
communication with his passing audience. Rupert is also able to
communicate through actions, moods and, most importantly, his paintbrush.
He can paint on the inside of the screen and in the virtual air of his studio.
A Brush with the Real World allows for spontaneous visual exchanges
between screen character and audience. All movement is transposed from
the actor onto the screen character allowing him to communicate directly with
the audience. The actor can respond to the audience and that response is
portrayed by the double on screen in the visual language of the artist. The
actor’s role is integral but invisible and the audience perceives itself
communicating with the artist and the art. It is a virtual theatre improvisation,
where the rules of an improvisational game apply, structured around the
77
characteristics, the quirks, foibles and activities of the screen character and
his ability to express through the paintbrush.
The communication between Rupert and the viewer subtly unfolds. Initially
Rupert is engrossed in his own world, painting and pottering about in his
studio. His first awareness of the audience is like someone who is suddenly
aware that someone is peering into
their window; he tries to block them out
and when that becomes unsuccessful
he attempts to ignore them. This can
only go on for so long before he feels
the urge to express his indignation.
However, he gradually warms to those
peeping into his world and starts a
dialogue, firstly through gestures and
then through his art. Finally, he uses his voice, talking directly to them. So the
sequence allows the interaction to evolve as the screen character, at first
detached from any involvement with the audience, progressively adds to his
methods of communication until he includes them wholly into a spontaneous
and engrossing exchange.
A Brush with the Real World uses a gestural interface to control some actions
in the projected scene. A gestural interface is an interface in which actions
are triggered by the body movements of the operator. With Rupert, positions
of the non-painting arm were used to control colour while forearm rotation
was used to control flow. The paint was activated when the hand was a
certain distance forward of the body and any distance beyond that
determined the extent of the paintflow.
Late in the development of A Brush with the Real World real-time speech
was added to Rupert’s communication repertoire. This gave Rupert another
layer of communication that could add to the dramatic depth of the character
and his ability to interact. In a performance situation Rupert could join in with
dialogue onstage, adding another step in an unfolding communication
Figure 12: Layout for A Brush with
the Real World
78
dynamic with an audience/observer. Facial expressions that express feelings
and emotions were also added and these could either be triggered by a third
party with a keyboard or game controller, or by the performer using a hand
controller such as a thumb joystick. The speech used MotionBuilder’s voice
recognition engine to trigger blendshapes, giving a realistic impression of
speech. In live situations this can work well but, because the mouth
movement is triggered by sound, any stray sounds such as audience noise
can also cause the mouth to move. So when using this voice engine in live
performance precautions should be taken to filter out extraneous sounds.
Although A Brush with the Real World was designed as a theatrical
installation, the outcomes from this type of performance are also applicable
to dance. The screen is a visual extension of the stage and can be used to
enhance the plot, the characterisation or context of the live action. With
Rupert’s ability to speak and to express moods, characterisations shown
onscreen through dance can be quite complex, extending the nature of
dance visuals to being a sophisticated communications medium. The real-
time nature of the visuals may help to overcome the limited ability to
emotionally connect with the audience, prevalent in dance productions of the
past that used motion capture. According to Birringer these productions
“were marked by a depersonalized formalist aesthetic of the painterly” (1999:
365).
The A Brush with the Real World environment was also used to explore the
“digital double” as a way of conveying dance in an animated form. Through a
digital double the dancer can be represented visually in an illusory world. In a
series of trials, dancers were involved in creatively exploring the virtual
character as a real-time visual element in dance. Using Rupert and his virtual
paintbrush, the dancer could dance his/her way through a painting as it is
created. The trail of paint became a map of the dance. Other relationships
between dance and the paintstream were explored, including one iteration
where the colour of the paint trail changed according to the height of the
brush, and in another the paint was constrained to a set of planar surfaces.
Other experiments explored the use of dynamically changing textures.
79
While the aesthetic may be the same whether the visuals are real-time or not,
the formalism is not a big factor in real-time because the dance is liberated
from the pace of the visuals. The onscreen environment is the puppet of the
dancer. So when paint trails from Rupert’s hand as he dances through his
studio it is the dancer controlling its path and its timing. Through this flow the
dancer can manifest precision or symmetry, or create recognisable artifacts
that can be woven into the choreography. The paint can also be used as a
colour trail of the dancer’s movements, a representation of movement over
time, as has been used previously with Riverbed productions of the late
nineties.
80
4.3 Chasing Shadows
Figure 13: Chasing Shadows
Chasing shadows is more traditionally theatrical, a combination of drama and
shadow puppetry. The audience members are onlookers and the interaction
is all on stage between the live actor and his shadow. A detective-like figure
stands near a lamppost waiting for a bus. His shadow casts on the wall of the
building behind. The shadow gradually takes on a life of its own, becoming
another actor onstage, and a scenario ensues among the inadvertent actor,
his clumsy shadow and other shadows that pass in the night.
With Chasing Shadows the technology is invisible and the focus is the
theatrical action onstage. Motion capture is used to generate the shadow of
the actor and this is projected in real-time on the wall (projection screen)
behind. The actor’s shadow is in fact a real-time projection of the actor’s
movements. It is worth noting here the reason for choosing a story about a
shadow. One inescapable problem whenever there is interplay between live
performance and projected images is that they exist in different dimensions,
the world of live performance and the world of cinema. A shadow, however,
is a naturally occurring two-dimensional projection that exists in a three
dimensional reality making it a natural player in a hybrid performance that
does not strain the bounds of credibility.
81
The shadow becomes aware of a sense of independence, gradually
exhibiting more individuality until it becomes a different player. However, as
the plot concludes the shadow recedes to its traditional role. The real-time
generation of the shadow is intertwined with movie clips to create the subtle
shift from obedient mimic to independent entity. Technically, a smooth
transition is difficult to achieve with inertial motion capture because the
motion data is relative. As an example: if the performance animator walked in
a circle back to the exact same position (s)he started, the character on
screen may not be in the exact same position it started. While there are
probably ways of automating the transitions, there was insufficient time to
explore this area and so the changes from live motion to movie clip in
Chasing Shadows rely on the actor’s intuition.
82
4.4 Private Eyes
Figure 14: Private Eyes
Private Eyes started as an offshoot from Chasing Shadows. It is a foggy night
in the inner city and a lone misty figure stands under lamplight on a street
corner waiting for a bus. He slowly becomes aware of and gradually
preoccupied with eyes watching him. The headlights of a passing bus or dark
figures passing in the foreground occasionally punctuate the scene. Lighting
and visual effects are used to create a Singing in the Rain atmosphere.
Private Eyes uses particle effects, background video, scene changes,
camera switching and mood lighting in real time to test the extent to which an
animated movie can be performed “live”. Scenes with multiple characters,
using a blend of movie clips and real-time motion capture, also added to the
range of possibilities.
From a dramatic perspective Private Eyes sees into the thoughts of the actor
on stage. These thoughts transport the audience to another time and place
as the actor reminisces about the past. While the structure relied less on the
animation being real-time than the other investigations, it provided a guide as
to how far effects could be pushed before the visuals were degraded by
dropped frames or stalling. Within the limitations of this study Private Eyes
explored a range of movie techniques with the emphasis on determining
83
whether they could adapt to live performance scenarios.
84
4.5 Motionics
Figure 15: Motionics 1. By attaching a camera to the head of an avatar the audience
can follow the eyes of the live actor through a journey of his/her imagination.
Figure 16: Motionics 2. This is a simple experiment in which particle emitters are
constrained at varying distances from parts of the body.
Figure 17: Motionics 3. This example uses simple emitters that change colour over
time to visually represent a dancer’s movement.
85
Figure 18: Motionics 4. Software mirrors are used to show different perspectives at
the same time.
Dance is usually set to metronomic rhythm of music, and visual media is
often used as a synaesthetic enhancement to the movement of the dancers
onstage. These visuals are sometimes generated using 2D motion capture, a
form of motion capture that uses a single camera source to map the shapes
of the dancers or a series of identifiable points visible on the dancers so they
are able to control the motion and timing of the visuals. 3D motion capture
extends this into the three dimensions of virtual space. Motionics is not a
single performance structure but a series of trials testing a wide range of
visual tools individually. These trials use particle emission and more
conventional animation methods to determine ways in which animated
visuals could interact with dance. There are many techniques in animation
that can be applied to visuals in dance and so this study, rather than focus on
a single performance-based outcome, explores a range of visual techniques
to reach an overview of real-time performance. It is a study portfolio rather
than a single case per se.
Visual effects, driven by motion on stage, can be viewed in real-time from a
range of virtual camera perspectives that can be toggled or animated.
Examples were recorded of scenes created from the view of a character on
screen, from multiple camera perspectives that could be switched, from the
perspective of a stationary camera locked to the movement onscreen, and
from an animated camera. Also, projected video was used, either as a
background within the 3D environment, a dynamic projection onto the body of
the character, or way to add Flash animation to the scene. Basic video and
86
particle effects were used in scenes without discernible deterioration in the
real-time performance. This was subject to such factors as the number of
video and particle effects used and complexity of the scenes. Multiple
characters onscreen were driven by a single dancer and software mirrors
were used to create the illusion of multiple dancers or to extend the scene
without adding more geometry. Reactive approaches were also used in
several ways. One scene used a sea of blinking eyes that were locked onto
the actor; wherever the actor moved the eyes followed. Trigger points were
set up at several locations in a scene comprised of a black flashing, reflective
figure dancing with staccato movement in front of continuous looping
lightning; when the dancer entered certain points in the scene, it caused
sparks to fly.
There are many attributes of the MotionBuilder environment that could be
explored artistically in a real-time projected environment and so a broad
determination of its use in live performance is difficult to assess other than by
using a broad performative approach. The case in dance explores in broad
terms the accepted capabilities of the software and how they work with
relatively unexplored hardware. While each of the cases was successful, at
least in principle, an acceptable visual performance can only be achieved if
there is consideration given to a whole range of factors. Variables such as
the complexity of geometry, resolution of textures, the extent to which video
and CG effects are used all influence the quality of the output. These are
factors that make up the balance between technical capability and artistic
intention.
87
4.6 Log of practice
Early in 2005 I set up an initial contact with Jon Newell from VRS, hoping to
establish a collaboration with the company to gain access to their motion
capture technology, and most importantly, their Gypsy Gyro 18 MOCAP suit.
It was April before any practical interaction commenced and, despite many
setbacks with both software and knowledge of the technology, it opened up
possibilities as to what could be achieved in a real-time interactive hybrid
performance. As I learned more about the technology I decided to aim for a
project that explored the potential of the MOCAP suit and my background in
both theatre and illustration. This gelled into the notion of creating a real-time
interactive painting environment, A Brush with the Real World.
The motion capture equipment was only available when it was not being
used for income generation and so the research needed to be flexible
enough to adapt to the windows of opportunity that arose. The technology
was new, entailing a lot of experimentation and learning. Early in these
studies persistent glitches occurred when generating real-time animation with
the inertial suit. Some of these were minimized by workarounds. For
instance, the inertia cube placed at the base of the spine was pivotal but was
subject to shift. I tried using Mastix to directly glue it onto the skin and this
helped. Later, I contoured a pad to sit snuggly in position and this worked just
as well. I used a similar method for the foot trackers and this marginally
improved the results. The angular rates and calibrations were persistently
tweaked; this also gradually improved the motion.
The latter part of 2005 was spent developing a painting environment.
Through early trials it became clear that all aspects were feasible but
progress came to a standstill when unexpected software conflicts cropped
up. In December 1995 Ali Kord, head of UK based Inertial suit manufacturer
Animazoo, visited VRS and demonstrated an upgrade that alleviated the
more glaring deficiencies in the software and provided a more stable
environment. I was now sufficiently familiar with the software and equipment
to handle motion capture sessions unaided. However, in March 2006 the
88
inertial suit was leased to Irrational Studios in Canberra for 4 months and so
the project again came to a temporary standstill. This called for a strategic
change. While the equipment was still available I arranged to capture all
motion for two of the proposed studies, Chasing Shadows and Room Rage,
just in case A Brush with the Real World was unable to be finished on
schedule. This way, if there were further problems with availability, the data
could at least be used for pre-rendered sequences in performance.
During the time the inertial suit was away I focused on modeling the
characters and environments, assisted by Kim Allen and Vana Prayitno.
Shadow models were built and rigged for Chasing Shadows and the first
“Rupert the artist” character was built for A Brush with the Real World. When
the inertial suit returned it became clear that the characters needed
significant revision to work well in a real-time environment. Changes were
trialed in the studio and by late September the characters were working
sufficiently well to tackle the more tricky technical questions associated with
using effects, triggers and behaviours. Chasing Shadows used video clips
interspersed with real-time action. The transitions needed to be unnoticeable
to an audience and this presented a range of technical challenges. In A
Brush with the Real World there needed to be a quick easy way of swapping
among colours on the brush, preferably a method that could be handled by
the actor. Another major hurdle presented itself around this time; Jon
Newell’s availability was greatly reduced when his role was changed from
motion capture specialist to content developer. As a result I needed to spend
large amounts of time through the rest of the year learning more about the
software until I was self-sufficient.
Each change was trialed in the studio and any results provided feedback for
further refinement. During this time a number of short motion capture projects
came up and so I cut back on performance work to participate in these. This
work ranged from performance animation, working as the performer in the
suit and demonstrating motion capture to games studios and human
movement related organisations. Further work arose in software
demonstration (GamesConnect Conference, Brisbane), running motion
89
capture sessions, edit motion data for animation (Bippo the Hippo) and run
training programs. The work give me a more comprehensive understanding
of all facets of motion capture, from both the perspective of the actor and that
of the animator, and a clearer picture of how it was deployed commercially.
By February 2007 A Brush with the Real World, while still needing
refinement, was working in principle. By April movie clips for Chasing
Shadows had been created and arranged in a trigger tree so they could be
activated from the keyboard. By July the environment for Private Eyes was
built and several effects-based scenarios had been created as proof of
concepts for dance. There was plenty of scope for further refinement, visually
and performatively, but all investigations were at a stage where they could be
demonstrated to be technically successful. Several upgrades in the software
have since reduced these problems further. The real-time performance
animation produced on screen was of sufficient quality for at least proof of
concept performances and, with Animazoo recently receiving extensive
development funding, the software was expected to undergo further
improvement. The inertial suit was now more regularly available and this
allowed me to spend more time in the studio.
Late in 2007 and through to March 2008 a basic gestural interface was
created for A Brush with the Real World to control the paint colours and the
thickness of the paintflow. Rupert was given a voice using the speech
recognition engine in MotionBuilder that recognises phonemes and triggers
appropriate blendshapes (shapes of the face representing the corresponding
phonemes). This would allow for greater dramatic flexibility with
communication between Rupert and audience. Facial expressions and eye
movement were also added. With the help of dancers Bridget Fiske and Joe
Lau, along with choreographer Rosetta Cook, real-time dance effects were
trialed in the studio. These trials were not so much focused on implementing
creative ideas as exploring the technical attributes of the software that were
likely to be effective as tools for creating visuals. In essence, they looked at
what was performatively possible and what was visually and theatrically
appealing but the intention was not to produce finished performance pieces.
90
The movie clips for Chasing Shadows were cleaned up, edited and arranged
in a trigger tree. The motion sequences for Private Eyes were also finalised
and other characters added.
Each of these investigations contributed to a broader understanding of how
motion capture could be used at live events. Scenery and characters could
be changed at an instant; interaction with the screen character could extend
beyond the interaction that games have made familiar. In April 2008 more
visual possibilities for dance were explored, including the use of video
projection and camera animation. As video could be projected on specific
objects in the scene or assigned as a texture to those objects in a number of
different ways, it was not difficult to create many types of video effects
created using real-time 2D. However, the 3D effect was navigable so that the
created effect could be viewed dynamically from any perspective. Through
2008 the focus was to incorporate operating methods that gave the performer
as much control over the environments as possible. The more control the
performer had the more spontaneously they were able to interact with the
environment. Control methods developed earlier in the research, while giving
the actor control, required too much thought and dexterity, compromising the
purpose of using them in the first place. So time was spent developing more
intuitive controls: those that more closely mimicked real-life actions.
91
5 Evaluation
There is no particular mystery in animation...it’s really
very simple, and like anything that is simple, it is about
the hardest thing in the world to do. (Tytla cited by
Graham, 1937: n.p.)
5.1 Inertial motion capture
The aim when commencing the research was to investigate a set of public
event ideas that incorporated projection. Motion capture, at the time, was
seen as a means to create some of these performances. As I began to work
with motion capture it became clear that, before it could be an effective and
familiar tool for live performance, a lot more needed to be uncovered as to
how it could work in live performance. There were questions about its
robustness and the quality of the real-time data produced. The performative
aims were speculative, as there were few precedents that provided insight
into whether the aims were technically achievable or qualitatively acceptable.
As a result the research first needed to explore whether inertial motion
capture was a useful tool for live events and, if so, in what ways it could be
used. So the research focused on examples that explored broad ranging
techniques applicable to live arts rather than the creative outcomes. Thus, in
the context of the research, the cases are archetypal more than artistic. They
answer questions of process more than product.
The quality of captured data depends on the real-time performance and this
is partly dependent on the performer’s awareness of the inertial suit's
movement parameters. While the inertial motion capture is able to capture
non-vigorous movement with acceptable accuracy, it has limitations.
Jumping, or in fact any movement where both feet leave the ground, is
predictive and often will not be portrayed accurately onscreen. Highly
acrobatic or high impact activities are generally not suitable. This might seem
a major drawback in, say, action games, but it is surprising how many of the
movements in this context are not in this category. Walking in a crouched
92
position was considered a move to avoid but we had few problems. The
screen environment that the actor controlled had definite limitations, some
that are intrinsic to the technology and its application, some that are likely to
be overcome in the near future; and others that have recently been solved.
Inertial motion capture has some inherent traits that differentiate it from the
more commonly used optical motion capture. Inertial systems have active
sensors that measure orientation. This orientation data is transmitted
wirelessly to a computer. Optical motion capture uses an array of cameras to
plot the changing locations of points in space. So inertial data is rotational
whereas optical data is translational. Optical systems use visible light as a
means of detection and so are subject to “line-of sight”, contrast and
reflectivity problems. With inertial motion capture the data has no global point
of reference and this can lead to a range of translational errors. However, it is
not the thrust of the research to delve into an in-depth comparison. Instead,
inertial motion capture is viewed simply in terms of whether or not it is
suitable for live performance. The attributes are set out in the chart below.
The second chart shows some of inertial motion capture’s limitations as well
as some specific recurring problems that are likely to affect real-time
performance.
93
Chart 1: Attributes of inertial suit
Chart 2: limitations and problems
This evaluation of inertial motion capture has been conducted over several
years using the first inertial suit available commercially. To gain a better
Latency Operated in real-time with few frames dropped
There is no significant lag even with more complex 3D
environments
Range Can operate over distance effectively
Signal interference Is not affected by line of site problems
Venue Is not subject to jitter or occlusion
Ease of use Can be used in a large range of live environments
Equipment Entire performance can run in real-time on standard computer
equipment fitted with a graphics card capable of running two
monitors.
Control In the right situation the entire performance animation process,
including the acting, can be operated by one person
Updates The software is regularly improved and with each new version
improves the real-time output
Complexity Can use elaborate 3D environments, multiple characters and
complex effects
Foot slide The foot slides on the floor with each step
Foot drop The foot drops a few centimeters to the floor with each step
Jumping/displacement Jumping is highly predictive. While vertical jumping is
interpolated well, jumping horizontally is not very accurate
Actor file It took a number of attempts to get an accurate “actor” file
Frame dropping The hardware or software cannot handle the rate of information
and drops frames
Relative positioning The data only tracks relative positioning. Depending on the
accuracy of the calibrations, the location of the virtual character
in the scene shifts with movement.
Vertical tracking Tracking is based on the assumption that one foot is touching
the ground at all times. When both feet leave the ground, say,
while climbing stairs, the software will locate the data with one
foot at y=0
Physical limitations The suit, while wireless to the receiver, has significant amounts
of hardware attached. This limits the range of costuming that
can be worn with it for a live performance situation
94
perspective on the problems and limitations listed, it should be noted that
since this study began there have been significant advances in the
technology that have reduced or eliminated many of these problems and
limitations. At Siggraph 2008 all systems presented, both optical and inertial,
were able to apply a broad range of vigorous movements to complex 3D
scenes in real-time reliably and with few discernible glitches. When trialing
the cases it also became clear that, by adapting the visual environments,
modifying characters, carefully selecting camera angles, and adapting the
choreography, interruptions to the visual continuity of each piece were
minimised. While the animation produced is visually impressive and is
sufficient to create an engaging interactive real-time production, the
technology is still not at a level where it can generate a reliable continuous
uninterrupted feeling of co-presence with a character on screen. Small
anomalies become obvious. To create a sense of realism or imbue a feeling
of co-presence with a screen character in real-time the level of precision in
the motion of the displayed animation needs to be high.
Animation has direct needs for precision that stem
from the sensitivities that viewers have in experiencing
motion. For example, a viewer is likely to notice
imprecision in a character’s interaction with its world. A
foot floating slightly above the floor or sliding a small
amount, or a hand not quite reaching the doorknob,
are tiny imprecisions yet can completely destroy the
illusions of realism. (Gleicher and Ferrier, 2002: 1)
The primary aim of this practice-led case study, however, is not to create an
illusion of realism as this would be an environment too complex to run in real
time and would require more resources and know-how than a study of this
nature is likely to attract. The cases are not designed to explore aesthetic
frontiers or even move beyond the visual conventions of 3D animation. The
focus is on communication between observer and screen and, as a
consequence, on the many facets of conventional 3D software and how well
they adapt to a real-time environment. The worlds that can be created in 3D
95
are years ahead of what can be represented in real-time and so the visual
quality of the real-time environment is necessarily less important than the
level of interactivity that can be generated.
5.2 The case study
Many of the techniques here are well known to those who work with
animation. Applying these techniques using motion capture to create real-
time animation is less well known. There are a number of well-documented
examples of real-time animated characters being used in a public forum;
these are typically real-time applications by animation production houses
rather than hybrid performances by live performance troupes. This difference
in emphasis is important in determining how the subject matter is approached
and how the technology is used. In live performance the “liveness” is primary
and animation is an augmentation, whereas in animation the emphasis is on
the rendered image; the liveness is an attribute that helps the workflow but is
usually invisible in the final product.
A real-time environment is not capable of reproducing the depth and
sophistication of animation people have come to expect through media.
These environments take many hours to render. In time the combination of
what can be produced in real-time and the tools available in a 3D
environment will no doubt create their own tree of aesthetic directions in live
performance. The case study looks at the potential of interactivity that this
technology offers.
96
5.2.1 A Brush with the Real World
A Brush with the Real World was designed to look at ways in which dramatic
interaction can take place between a screen character and an audience and
so it incorporates several communication techniques. The virtual artist,
Rupert, can communicate physically and verbally but, most of all,
communicates through his paintbrush.
Figure 19: Trialing the paintbrush
Using particle effects as “paint” was sufficient for the study but had some
limitations. Firstly, the paint was RGB colour. As a consequence, when one
colour was blended with another it did not behave the same as real paint. As
an additive system blending one RGB colour with another resulted in a
lighter, brighter colour. Real paint is CMYK colour and blends subtractively;
the resultant blend is darker and duller. As a compromise Rupert used matt
colours that did not blend at all. A CMYK colour system could have been
simulated but there was another hurdle: any change to the particle emitters’
colours affected all its particles onscreen universally. For instance if the paint
changes from red to blue all red paint already onscreen will also change to
blue. Each colour required its own emitter (figure 20). The other major
limitation when trying to paint using an onscreen artist is that the system
97
controlling the hand movement is not highly accurate. This is partly because
the data does not represent the subtleties of hand and finger position, and
because an inertial system relies on the body measurements of the actor.
Being at an extremity of the body, the position of the hands are calculated
from all such measurements back to the feet and there is likely to be a
significant degree of error.
Late in the development of A Brush with the Real World real-time speech
was added to Rupert’s communication repertoire. This gave Rupert another
layer of communication that could add to the dramatic depth of the character
and his ability to interact. Facial expressions that express feelings and
emotions were also added and these could either be triggered by a third
party with a keyboard or by the performer using a hand controller such as a
thumb joystick. Altogether, eight expressions were created and these could
be used in combination with speech so that a character could talk
expressively.
The characterisation was important in creating a sense of connection with an
audience (co-presence), so the quirks and foibles of Rupert were explored
theatrically. A set script or scenario would have detracted from the necessary
improvisational quality that made this type of media unique and so loose
scenarios were devised and active elements were included in the scene to
steer communication between Rupert and the audience. Also, the scene and
the viewing position could be switched easily at any time, opening up
opportunities to further develop a more narrative and visually diverse
approach.
No capture device was used for the hands; they were set in a neutral stance
and changes were kept to a minimum. The eyes were operated in several
different ways; they could be fixed so they always faced forward, automated
so they “led the way”, or controlled by a joystick or mouse. The eyelids
randomly blinked. Objects in the scene could be picked up and put down in
real-time. Rupert could also trigger actions by entering a particular location
on screen or by putting his hand on a certain virtual object. As an example,
98
active objects in the scene, such as light switches, could be “switched on” by
the character.
The primary study, A Brush with the Real World, was performed as an
improvisational window installation at the Judith Wright Centre for
Contemporary Art in Brisbane in July, 2009. The animation was rear-
projected onto two large screens, one facing the street for interactions with
passing street traffic, and the other set-up in the foyer for more in depth
dialogue with a more stationary audience. Cameras and microphones were
set up so the actor could see and hear those watching. Sound from a
wireless headset worn by the actor was fed through speakers on either side
of the foyer screen and also back to the computer to generate mouth
articulation in Rupert so that Rupert could talk to the audience. His virtual
studio included a range of active elements to engage people in participating.
Rupert could turn lights on and off, change appearance, pick up objects and
put down objects. He had a range of moods and spoke with articulate mouth
movements. He could paint with a range of colours and could dance to create
colourful 3D trails. All these elements helped to make it a complex multilevel
improvisation where people were generally more engrossed in
communicating than passively watching.
Four scenes were projected at different times during each two-hour session:
the studio, the cellar, an empty black scene as a backdrop for dancing and
painting and a 2D poster. The last scene, Brushalex, is presented in the style
of an Art Nouveau advertising poster with Rupert as a prominent figure.
Council regulations prohibited the use of amplified sound in the street so, for
the window projection, the interaction focused more on visual elements. In
Brushalex Rupert could move letters in the text of the poster engaging
passersby in creating new expressions. In the cellar scene Rupert wandered
about in the dark with a flickering candle. He could pick up and put down the
candle on several boxes scattered around the space. If he sat on the candle
the back of his jacket began to smoke. It could be extinguished by repeatedly
hitting it with his hand. The studio scene was the most commonly used
because it was more in context with Rupert’s nature and occupation,
99
providing easy conversational cues for the audience.
While experimental, A Brush with the Real World accentuated that
improvisation works well with animation, whether the form of improvisation is
physical, visual or verbal. The performance relied on attracting the attention
of passersby, holding their attention and drawing them into participating. This
provided, at least in quantitative terms, an ongoing indication of the success
of the installation. After the first night it was concluded that the audience for
the outside screen was comprised mostly of drivers sitting in traffic and
diners at the restaurants and cafes across the road; people walking on the
footpath directly past the screen frequently failed to notice it. On the second
night we lifted the blinds on the windows either side of the projection so
people had a clear view of the actor driving the animation. This had an
immediate effect as people could clearly see the connection between the
actor and the character, realizing the projection was not just another patch of
inner city light flickering in their peripheral vision, but something intriguing.
figure 20: The two screens
Gestural control of the paint worked well. The painting environment was
relatively simple but sufficient to demonstrate that it worked as an
improvisational tool and, based on its use with an audience, is a device well
worth refining and structuring theatrically for future public performances.
There was little opportunity to practice the painting before the performances
but, based on our work in the studio, the articulation of the brush grew better
100
with practice. In this way the process is similar to drawing with a graphics pad
or mouse in that it relies on watching the progress of the image being created
onscreen, a situation where the cause and the effect are separate.
The project was technically challenging and there were a number of
equipment and software conflicts and failures over the three days of set-up.
Fortunately, Queensland Academy of Sport and VR Solutions provided
additional equipment and timely assistance enabling the performances to go
ahead in a modified form. We also had excellent ongoing assistance from the
administrative and technical staff of the Judith Wright Centre. Throughout the
week we operated with a four person crew consisting of Rosetta Cook, co-
performer and public relations, Adrian Green as technician and digital
puppeteer, Miriam Hanmer as assistant stage manager, and myself.
5.2.2 Private Eyes
Some aspects of 3D animation software environment are metaphorically
modeled on film production; these include actors, lighting, cameras and
effects. Private Eyes uses particle effects, background video, scene changes,
camera switching and mood lighting in real time to test the extent to which an
animated movie can be performed “live”. The environment used was able to
support multiple particle effects, lighting and video at the same time with
acceptable real-time playback. Scenes with multiple characters were created
using a blend of movie clips and real-time motion capture. Rather than
overload the environment with so many dynamic elements that real-time
rendering was affected, certain elements in the environment were only used
in specific scenes or shots where they were a prominent feature, or when
they important to the storyline, mood or effect.
In this study there were many variables at play and so it was approached
reflexively to get a broad overview of whether all of these factors could work
together without degrading the quality of the real-time screen rendering. As
would be expected there is a tradeoff between complexity of the scene and
the quality of the animation; the juggling of the components of each scene to
101
achieve an optimum result is an artistic as much as a technical decision.
What can be drawn from this is that these components can all be used
together; the extent to which they can is a question of balance.
5.2.3 Motionics
Motionics uses simple constructed environments as well as some generic
assets from the application, ‘MotionBuilder’, to explore a range of software
features including virtual cameras, triggering, video, particle effects, scene
changes and relationship constraints, using more evident ways in which each
can be used to connect the motions of the dancer and the mood of the dance
to the projected visuals. The outcomes give an indication of how well each
element performs in real-time.
Figure 21: Reactive background The way a camera in the scene is used can determine the relationship
between the dancer, accompanying imagery and the viewer. Using a first
person view, the audience is transposed into the eyes of the dancer as they
move through the virtual scene, so it becomes a mutual journey between
audience and dancer. This technique was used in a night-time scene, set in a
forest of blinking eyes. By animating the camera the movement of the dancer
could be viewed in ways that were almost impossible to achieve by other
102
means: the camera becomes part of the choreography as the perspective
can pan, zoom and follow complex paths around the motion represented
onscreen. This flexibility is further enhanced by toggling between multiple
cameras, each of which can have unique visual elements. A moving camera
constrained to a path is used in one sequence to show the virtual dancer
from continuously changing perspectives.
There were several ways of using a dancer’s movements to drive multiple
virtual dancers. The obvious way was to use several characters, but mirror
surfaces were also used to create the appearance of multiple characters.
Abstracted liminal bodies were created by constraining particle emitters to
the motion. Particle effects were also used to create motion trails and more
interpretive visual imagery. Images were applied to particles and these could
be dynamically constrained by the dancer’s motions. There were several
ways in which video can be technically applied in a scene, and these
provided some interesting visual outcomes. Video was projected as a
background element as well as directly onto a moving character. By applying
it to the surface mapping of the character reflective and transparent effects
were created.
Figure 22: Eyes follow the dancer
Figure 24: Triggering effects
Figure 23: Moving through a sea of eyes
FigFigure 25: Using video
103
Using relations (and expressions) constraints offer many ways of connecting
motion data to the attributes of any element in a scene. This is an area that
could provide a large range of interesting visual possibilities. However, it was
only used in a couple of instances, one for causing a shower of sparks when
the character moved through certain points in the scene and to change the
colours of particles using hand gestures. These video and particle effects
were used in real-time without discernible deterioration in the real-time
performance. Simple relationships between the dancer and the environment
were also trialled. One scene used eyes that were locked onto the dancer;
wherever the dancer moved the eyes followed. Another used a projected
backdrop that exaggerated the effect of travel onstage.
Motionics provided an opportunity to explore the visual landscape, free from
the constraints of a set scenario or a particular choreography. The program,
Motion Builder, has an extensive range of tools for working in real-time and
the ways these can be used to create interactive visuals can lead to a great
range of creative outcomes.
Figure 26: The Paint colour changes universally
104
5.2.4 Chasing Shadows
Chasing Shadows experiments with interplay between live action, real-time
animation and movie clips to create the dramatic illusion that the actor’s
shadow takes on a life of its own. This was attempted by alternating between
capturing live action and triggering movie clips. To make the transitions as
smooth as possible, the changeovers from real-time shadow to movie clip
shadow took place either out of view or at a set place in the virtual scene.
Blending movie clips with real-time motion and live action was a challenge, as
the position of the actor’s double onscreen had no absolute point of reference
and, over time, was subject to drift. To reduce this problem the actor on stage
had a set location that he kept returning to. Changeovers between clips and
real-time performance animation happened either at this point or when the
shadow was out of view. With more time these transitions could have been
made smoother by setting the horizontal coordinates of the characters point of
reference in each subsequent clip to the final horizontal hip coordinates of the
character when driven by the actor.
The performance was designed to minimise the audience’s awareness of the
motion capture equipment. The scene was set in an age of overcoats, broad
hats and loose trousers, which made it easy to obscure the motio0n capture
equipment worn by the actor. Also, the actor’s digital double was a shadow,
reducing the likelihood anomalies in the animation would be noticed. A
shadow is a naturally occurring two dimensional projection of a 3D object and
so long as the application of the shadow is well executed, an audience may
accept the scene purely as a physical world performance, rather than be
diverted by the technology driving it. Footslide was a recurring problem that
was likely to occur during performance, breaking the continuity and the
fantasy, so the shadow was projected low enough so that the feet were often
out of frame.
105
5.3 Observations of the performer (A Brush with the Real World)
You feel very light, a sense of freedom. When you
move it feels easier than doing it yourself. Embodying
the character is almost instant but there’s also a
disconnectedness, like in puppetry, where you’re inside
the scene but watching it at the same time. So you’re in
two places at once. (Rosetta Cook, 2009: n.p.)
While the scene bears little resemblance to the real world, recognising the
mechanical relationship to a character onscreen seems to require no
adjustment process. Normally the movement of the character onscreen is a
diametric opposite of the actor’s movements. It is much easier working with a
mirror opposite and this easily done by reversing the projected image. The 3D
world onscreen is not really a 3D environment and while there is an
appearance of depth there are only two dimensions; so the whole sensation is
like looking into a psychedelic mirror with one eye open. This is difficult at first
but, providing the character is oriented correctly in relation to the screen, it
becomes quickly familiar and easy to accommodate.
You feel yourself inside and see yourself in the
distance, like a mirror except the mirror can move so
you can see yourself from any angle. Dancers spend a
lot of time in front of a mirror and to be able to see from
any angle at will.. dancers don’t get to do that. It’s a
great choreographic tool. (Rosetta Cook, 2009: n.p.)
When working in a real-time environment with an audience two screens are
needed, one for projecting the animation and the other for monitoring the
audience, creating a split focus. All these differences can create anomalies
such as walking through a wall or struggling to pick up an object in the scene.
In the performances this did not appear to affect the public perception and,
after a while, choosing which screen to watch became second nature.
Working with a virtual prop meant working with nothing but an image of it in a
106
projected scene and so the temptation was to look at the screen instead of the
relative location of the prop. Over the nights of performance it became easier
to locate the object first onscreen then rely on memory to interact with it,
making the process unnoticeable to the audience.
To monitor the audience a camera needed to be placed as close as possible
to the character view. The ideal position would be the centre of the projection
screen, a problematic undertaking. The camera monitoring the foyer was
placed to the side of screen and this worked well making it easy to respond to
people's gestures, expressions and movements. Due to council restrictions
the camera facing the street needed to be set inside looking from an adjoining
window well away from the screen. From this monitor the location of people
watching was difficult to determine and this hindered any gestural gameplay.
Despite these drawbacks, as improvisation the animation was successful in
engaging the attention of passersby often for extended periods. People were
generally willing to suspend belief and immerse themselves in the interaction.
By the third night the blinds covering the shopfront windows either side of
street screen were opened and the actor's performance area was lit. The two
screens then each had a different but complementary purpose. The foyer
screen was used for improvising with the public whereas the street screen
showed the same animation but as an exposé of the process.
The mouth animation generated by voice generally worked very well. It
fractionally lagged behind the voice but this was barely noticeable. On some
occasions if the character talked during action that was processor intensive
such as when complex particle effects were used the lag between voice and
animation became apparent. Sometimes these conditions would also cause
dropped frames. All these anomalies did not have affect the overall
interaction. Once set up improvising was as fluid as it would be as a roving
character.
In reference to performers using motion tracking to affect digital media in real-
107
time Coniglio states:
..we cannot a priori rely on the expectations and
understanding the jazz audience brings to a
performance. This is because:
1. the audience may not be aware there is some level
of improvisation occurring, and
2. because the audience has no prior understanding of
the 'instrument' with which the performer controls that
manipulation. (Coniglio, 2004: 9)
So people are likely to assume the animation is prerecorded unless it is made
clear that it is not. Several strategies were used to make it more obvious that
the animation was interactive, some that worked very well. The most obvious
and effective way is to strike up a conversation. Mimicking, and setting up
short scenarios that beg a response were also used effectively to initiate
interplay. Once the penny dropped people either watched intriguingly or
became enthusiastic players.
5.4 General findings
Many animation techniques can be applied to live performance using inertial
(and other forms of) motion capture. These techniques are well known to
animators but little known in live performance. As a way of adapting this
medium to live performance many of the actions onscreen were operated by
the performer, so there was a spontaneous relationship between the
performer and the visual scenario. Gestural control, voice driven blendshapes
and hand held devices were used to control events onscreen. These events
range from switching camera perspective to triggering animation. The
intention was to minimise the presence of technology, prioritising the
“liveness” of the performance over the perceived impact.
The ways in which animation techniques could be used at live events are as
broad as the imagination but there are some important limitations. Projection
108
is affected by light, so the places and the times for using projected visuals is
restricted. Connecting a projector to a computer can substantially increase the
workload of the graphics card if used as a second monitor and so it is
important to trial the technical set-up well before a performance.
A complex blend of models, effects, lighting and camera movement can be
achieved in real-time, but there is a threshold at which the quality of the real-
time animation becomes too problematical to use. For this reason
performance needs to be planned around the capabilities of real-time
animation. The integrity of character movement is affected by the calibration of
the actor driving the motion; the fluid movement of a character is an art as
much as a science. The quality of the real-time animation is affected, among
other factors, by the battery voltage, magnetic field and the initialization
process. Even with all care taken anomalies can adversely affect the integrity
of the animation. These are listed on Chart 2. The animation environment and
real-time interface in Motion Builder are very sophisticated and, though there
is a steep learning curve and limited learning resources available, its
capabilities are impressive. One glaring limitation with using motion capture
live onstage is that a performer needs to wear the inertial suit. The range of
effective movements is limited and the suit itself is a visual statement that
needs to be included as an aspect of the design, covered by an appropriate
costume or not visible to the audience. Any choreography that uses this type
of equipment needs to be devised around these limitations. From what we
found this still leaves a good range of artistic possibilities to explore.
Another limitation when using any form of projected image as a part of live
performance is the unobstructed space required to project the images. This
problem is inherent in the nature of projection but some advances in
equipment over the past couple of years may improve the way projection can
be used on a stage or other public event space. Rear projection requires
ample room behind the projection screen. The focal lengths of more recent
projectors have significantly reduced this distance. Projectors have been
developed that, as an array, project upwards from the floor in front of the
109
screen at distances of a couple of metres or less, making it feasible to have
front projection and live action on stage work hand in hand in places where set
up space is limited. Digital projectors have been unable to recreate true black
and this affected the quality and intensity of projected images, especially in
situations where background light was an issue. Expensive CRT units were
used for high quality projection. However, in the past couple of years new
lightweight digital projectors have been released with high refresh rates and
the ability to project black areas on screen.
While a mid-level graphics card is adequate for content creation and viewing
animations on computer monitor, it may not be suitable for projecting in real-
time. In a perfomance set-up it was important to access the menus on
computer screen while projecting the animation from a projector. This required
a computer and graphics card that could smoothly handle this output.
5.5 Implications for live performance
Animation is a field undergoing exponential development based on massive
budgets. It has a rapidly evolved heritage built on a complex ecosystem of
software. To address the implications of 3D motion capture for live
performance is, in effect, to look at the entire range of creative possibilities for
3D animation. Although there are computational issues that limit what can play
in real-time and certain areas of content creation are not supported in a real-
time environment, this is changing.
The form in which images are presented is integral to the artistic and
performative outcomes. In stage performance scrim, cycloramas and
transparent screens have long been used to add depth, to circumvent the
dimensional limitations of projection. In one visual sequence in the production
Glow by Chunky Move, a video template of the dancer’s movement was
superimposed onto the dancer as she moved about the floor. This was
combined with a spyrographic effect that continuously circumnavigated the
extremities of her projected frame. Projection used in this way, as a visual
110
extension of the performer, ensures that there is no dilemma in matching the
two dimensional projection with the three-dimensional dance and no split
focus between the screen and the stage; it is all one. In the production,
Motion-e, the movement of dancers shaped shards of soft light and gentle
motion patterns on levels of scrim to create a spatiality that melded easily with
the performance, removing the projections from their planar origins. The way
in which projection can be used onstage is showing artistic maturity born out
of years of experimentation. However, the capabilities of projection are also
expanding. New research has demonstrated that smart projection can not only
follow the shape of a dynamically moving object, it can adjust to the contours
and colours of arbitrary surfaces (Bimber, 2008); it can intelligently blend with
other projections (Majimber et al., 2008); and it can produce a stationary
image from a mobile projector (Rasker, 2008) opening up further ways in
which images can be presented in performance.
The field of gestural interface, when used in live performance is, in itself, a
trove for further development. Through movement an actor/dancer can control
any conceivable action in a performance or at a public event that can be
connected to a digital network. Live human motion can be used to control the
lighting rig, or trigger remote actions, steer robots and puppets. Through
performance animation, it is possible for the mood of an actor or dancer, using
body language, to initiate changes to the lighting, sound and set, generating a
sympathetic mood on stage. Also, based on the established pace of
technological development the range of visual possibilities will expand rapidly.
The whole mood of a production at any particular time can be in the hands of
the actor or triggered by the actor’s situation. However, this may only be an
advantage when there is an intuitive relationship between the action of the
actor and the event triggered. For instance, the brush in A Brush with the Real
World worked when the brush was a beyond a certain distance from the body
and within a natural range of arm movement. This is similar to the way an
artist would normally use a brush and was very easy for the actor to adapt to.
On the other hand the colours of the paint were controlled by the position of
111
the non painting arm, an interface that has no parallel in the physical world.
This was difficult for the actor to grasp without a significant amount of practice.
The prospective use of real-time performance animation at public events,
particularly live performance, is related to the sum total of its unique attributes
that can be adapted to performance. Conceivably, other art forms such as
virtual sculpture (modeling) can also be carried out in the same way, using a
digital double in combination with a gestural interface. In combination with 3D
printing, this may become a common sculpting technique in the future. It is the
ultimate intuitive interface paralleling the way we interact digitally with the way
we interact with the world. Gestural recognition is an aligned area of research,
investigating ways in which a person’s body language can be digitally
recognized. In a performance, these algorithms can be used to determine
consequences, such as the direction a narrative takes, how an audience is
responded to, or the atmosphere created onstage.
With current 3D scanning techniques it is simple to recreate the appearance of
a person as a 3D object. Hand scanners can scan a body and render it in
minutes including the textural appearance; the accompanying software can
intelligently re-sample the mesh to optimize it for animation. A character driven
by motion capture does not need the kinematics normally applied to a 3D
character and so such characters are simpler to create. Consequently, using
available technology, it is a straightforward process to create a 3D image of
oneself that can move in real-time in digitally constructed worlds. “Mirror,
mirror on the wall’” will one day be reality. (The question then is: who will
control the mirror?)
The semantics of performance are a core part of the language of animation;
this has not been through a usurping of the language but is an indication of
the roots of animation. Many skills in animation, in its procedures and
particularly in its performative form in motion capture, are the same as those
used in live performance and so convergence between animation and
performance is a rejoining of separately evolved branches of a common
112
evolution. Phillip Auslander (1997: 7) argued that the nature of performance
has changed with the advent of new media. As a result performance has
become more mediatized. In 2004, based on statistics of the National
Endowment for the Arts (NEA), Auslander also pointed out that 78% of people
in the United States never attend live theatre. This is, in part, a problem that is
more uniquely American because theatre in the U.S. has never held the same
cultural status that it does in Europe where the traditions of theatre are woven
into the culture. Also, the low participation rate in America is not so much an
indication of the demise of theatre as an indication that theatre, and live
performance generally, has long left the theatre space; it has both diversified
in form, adapting to a more mobile, more convergent, more visually focused
world, and has evolved along with and as part of new media. The traditional
skills of performance are integral in the processes for creating content in all
genres of new media.
As digital games become physically performative, the applications for motion
capture in live performance may reach the household games console before
they ever make it to a live performance. The new level of physical participation
that motion capture will open up will once again blur the boundaries of what is
live performance and what is new media. Already the Wii has swept the
games industry with its ability to engage a person in physical play. In the near
future, markerless motion capture using a set of web-cams or “active” clothing
may be part of household living. This will make collaborative scenarios in
which people physically participate with other people in what is unmistakably
performance, an unremarkable activity of lifestyle. If performance genres such
as dance and theatre continue to focus on the changing nature of human
activity then they will be part of the fabric of these changes rather than left
marginalized by what may seem an encroachment. (Motion capture
technology may also one day spell the end of such ubiquitous items as light
switches and remote controls).
113
5.6 Summary
In the grand scheme this research is a tentative foot in the water. It attempts
to focus on 3D animation that can run in real-time to enhance live
performance but, as a relatively unexplored domain, the outcomes can only be
a peep into the future that, combined with many other peeps, starts to give a
broader picture of what, how, when and where motion capture will be used. A
gestural interface when used in live performance is, in itself, a very promising
area for further development. It allows the performer using subtleties of body
movement and voice to control actions in a virtual scene, onstage or
elsewhere. The whole mood of a production at any particular time can be in
the hands of the actor. Whether this is desirable or practical The prospective
use of real-time performance animation at public events, particularly live
performance, is related to the sum total of its unique attributes that can be
adapted to performance.
Optical or video-based motion capture can provide more accurate data than
inertial motion capture. However, at a public event in which real-time motion
capture is needed, the attributes of inertial motion capture, including ease of
setup, ease of use, cost and adaptability, and quality as a real-time
environment (unedited) make it a better choice. Also, unlike optical motion
capture, there are no line-of-site problems; motion data is sourced from the
movement of the body directly rather than interpreted from light emitted from
the body. Therefore, objects and other people can obscure a person wearing
an inertial system without affecting the motion data.
5.7 The future of motion capture
In the past two to three years there has been a rapid expansion in the number
of universities and colleges with interactive arts labs that have motion capture
studios. These labs have been used for analyzing, recording and documenting
movement in live performance. For instance, the Advanced Computing Centre
for the Arts and Design at Ohio State University recently recorded some of
114
Marcel Marceau’s mime routines and these are now stored as 3D motion data
that can be used to teach, demonstrate and reconstruct in perpetuity. This
growth in motion capture studios on campuses is an indication of the growing
importance of motion capture as a tool in research.
Several Australian educational institutions have motion capture systems
installed, from the small eight camera optical systems at QANTM and QUT to
the twenty four camera system at Deakin University. These systems are used
in areas such as film, games, biomechanical research, human movement
studies and animation. Motion capture is increasingly used in live arts for
analysing and documenting performance as well as creating visual effects for
use in performance. When exploring a maturing technology in which the
possibilities grow regularly many questions arise.
Visual technology is already ubiquitous. Film, television and games are some
of the world’s largest industries. 3D technology has had a major impact in
these industries but this has masked its growing contribution in other areas. A
quiet revolution is taking place as 3D technology is infiltrating many other
fields. Real-time visualisation is already establishing itself as a major tool for
large-scale building projects, mining, defence, flight training, population
dynamics, human movement, and social modeling. It has recently proved
useful to differentiate between Autism and Asperger syndrome. 3D scanning
and printing are revolutionising archeology, reconstructive surgery and
prosthesis, and is becoming more widely used for rapid prototyping. An
important feature unique to 3D environments is not only the ability to represent
an object realistically but to simulate the properties of the object. This makes it
possible to visually simulate complex interactions among objects enabling 3D
environments to be used not only for vastly speeding up experiments that can
be done in other ways, but also for conducting experiments visually in virtual
macroscopic environments to simulate weather patterns, global warming,
solar flares or black holes, and submicroscopic environments to model
aspects of quantum mechanics or organic chemistry. It may be used to
visually represent the dynamics in non-physical, hypothetical, abstract,
115
conceptual, or non existent environments. The other attribute of the 3D
environment is that it can simulate interactions instantly; real-time remote
motion data can be used to carry out robotic rescue operations, work in
dangerous environments or manipulate tools on space probes.
Many of the factors that currently restrict use of inertial real-time 3D
performance animation in live performance are diminishing:
The price of the technology is declining
Devices that use tracking sensors are becoming more widespread
The quality of the motion captured is improving
The range of uses is expanding
The technology is diversifying
Systems are becoming easier to use
Systems are becoming more robust
The software is becoming more stable
The software capabilities are expanding
The hardware is miniaturizing
Small wireless sensors with on-chip intelligence are becoming available
The hardware worn by the actor is becoming less obtrusive
Computers are becoming more powerful
(Most of these factors also apply to optical motion capture)
Motion capture technology is shifting from being an elite technology to
becoming a pervasive and diverse aspect of everyday life. Already computer
games consoles such as the Wii are incorporating basic motion tracking
technology into movement-based games. iPods use an accelerometer to
change the orientation of the screen when it is rotated. A range of pointing
devices or “wands” use motion tracking as a 3D computer interface device.
Motion capture in major movies has steadily increased over the past ten
years, and in 2006 two of three Best Animated Feature Film nominees for
Academy Awards, Happy Feet and Monster House, used motion capture (von
Riedemann, 2007). In some movies the effects and atmospheres as well as a
116
character's appearance and movement have become so realistic the
animation is almost indistinguishable from live video action. The range of live
performance situations in which motion capture can be used is limited by
economies of scale. It is also limited by the quality of the animation that can
be produced live. However, with new developments, these hurdles are
disappearing.
Several breakthroughs have been made in the last two years in motion
capture. Companies such as Mova (‘volumetric capture’), Organic Motion has
successfully produced a markerless motion capture system which can capture
a person’s movements accurately without the use of specialized clothing or
markers and this can drive a character with reasonable accuracy in real-time.
This has tremendous implications for the way we can interface with the world
around us. It could lead to more ubiquitous use of gestural control as a means
of interacting with a whole range of electrical and electronic devices: a
preemptive living style where things go on and off because of what a person is
about to do rather than because a button was pressed. Intelligent software
can be used to learn people’s habits and act according to their needs. This
sounds like technological submission but it has practical application for, say,
intelligent energy saving, impact studies, taking care of mundane repetitive
tasks or catering for people with disabilities. Large sampling of complex
movement data for, say, better town planning and more ergonomic design as
well as health and sports research, becomes practical. It also has implications
for performance and entertainment: animation has moved from a linear form
(television and cinema) to an interactive form (as in computer games) and,
through markerless motion capture, it can move to a participatory form. With
markerless cameras set up in a public space anyone can join an animation as
a performer.
Price is another hurdle that is shrinking. The Optitrack Arena is an eight
camera full axis 3D motion capture system, selling in its basic form for under
ten thousand dollars, perhaps heralding new opportunities for cottage
animation and the spread of the technology to smaller learning institutions.
117
More importantly in the context of this research, it makes motion capture
accessible to a broad range of performance companies. Dutch company,
Moven (recently acquired by Xsens) released an inertial system that has
solved many of the problems with inertial capture including off-ground capture
and accurate foot movement. A prototype portable system was demonstrated
at Siggraph 2007 that combined inertial trackers with ultrasonic emitters
enabling it to be used in a large range of outdoor situations without the need
to transmit data to a nearby computer (Vlasic et al., 2007). Also demonstrated
was a technique that uses very inexpensive photosensing markers for
accurate outdoor motion capture (Raskar et al., 2007). Measurand have
improved exoskeletal (mechanical) suits by using flexible fibre-optic tape in
combination with inertial sensors in place of rigid telescopic frames. Since the
release of its first Gyrosuit four years ago, Animazoo’s software has
undergone improvements and their newest suit has the option of combining
with an ultrasonic tracking system for more accurate translation. All major
types of motion capture can now minimally produce reasonable quality real
time animation. Along with improvements in quality and accessibility there are
two distinct trends in motion capture innovation: inertial motion capture is
becoming location independent and optical motion capture is becoming
costume independent.
In the past couple of years the term “performance capture” has become
prevalent. Performance capture has come to denote a shift in emphasis from
capturing motion to representing performance. By realistically approximating
the motion of other elements such as hair and clothing, a more fluid and
holistic representation of a performance can be created. Research at the
Massachusetts Institute of Technology entitled “Articulated mesh animation
from multi-view silhouettes“ (Vlasic et al., 2008) presented at Siggraph 2008
showed that by plotting silhouettes and combining this information with motion
capture data, not only the body movements of an actor but the movement of
clothing worn by the actor could be depicted onscreen. A second paper titled,
“Performance Capture from Sparse Multi-view Video” (de Aguiar et al., 2008)
showed that by using algorithms a performance could be captured as surface
118
deformation, obviating the need for a skeleton and providing a detailed
representation of performance.
Not only are we moving towards more realistic, more manipulatable and more
instantaneous animation, we are also moving towards “intelligent” animation,
with characters that have complex reactions and responses. In games, for
instance, behaviours and physics allow characters, objects and other
elements to respond to conditions within a scene and this extends the user
experience well beyond direct responses from triggered clips. This is an
intense area of research that is rapidly evolving and includes aspects such as
behaviour planning techniques (Lau and Keffner, 2005), learned character
behaviour (Park et al., 2005) and imitation of another character’s behaviour
(Blumberg et al. 2005). In the near future these and other developments are
likely to lead to a rapid evolution in the ways in which independent decision-
making by screen characters can be integrated into real-time scenarios in live
performance.
The locations and the contexts in which real-time motion capture can be used
are also expanding. The use of projection at public events has grown and
evolved. Large-scale digital screens are becoming more commonly used at
major public events and are often installed at major sporting venues and
exhibition centres. Sydney based Electronic Canvas use large format
projectors and proprietary software to create huge visual spectacles on the
sides of buildings. Curved and hemispherical projection, techniques for
projecting onto complex surfaces, algorithms for blending multiple projections,
3D projection, and mobile projection, along with systems for integrating
multiple image sources are expanding the ways in which images can be
displayed. The drive to deliver more, better and bigger spectacles is creating
opportunities both in and outside the driving force of mass
entertainment/infotainment that is open to exploration by live arts. Much of the
infrastructure and demand is already there. Art galleries, performance
complexes, libraries, museums, convention centres, theatres, tertiary
institutions and festival venues all, to a growing extent, use visual display
technologies and are friendly environments for the use of new technologies in
119
live arts. The creative possibilities for the use of motion capture in these
environments are extensive because motion capture raises the bar on the
level of interactivity that can be achieved.
There is a growing interest in the use of real-time virtual hosts such as
mascots and animated presenters at corporate events and festivals. In the
future there may be caricaure doubles of Ritchie Benaud and Wasim Akram
commentating the cricket live onscreen at the Multan Stadium, cartoons that
kids can participate in at children’s festivals, digital paintings that react to the
observer, actors controlling the mood onstage or controlling physical events
offstage, a dancer in London controlling a fleet of dancing forklift trucks in
Tokyo, a conductor controlling a virtual orchestra or a virtual conductor
controlling a real orchestra at the Sydney Opera House. Projections can follow
and deform to the shape of a moving changing subject; objects can move,
lights can go on and off, and effects can be triggered with the gestures of the
hand. The use of hand gestures as an interface is depicted in the movie,
Minority Report, in which Chief John Anderton, played by Tom Cruise,
manipulates a complex array of screen images using his hands. This, while
fictional, was based on the work of John UnderKoffler, technical advisor for
the film.
120
6 Conclusion
The opportunity to learn and use motion capture equipment as a tool in
research has provided an insight, not just in the techniques investigated but
into a much broader range of possibilities that came to light as the parameters
of the software became more familiar. Synaesthetic connections, gestural
control, the use of CG and particle effects as real-time visuals, and real-time
voice animation are all areas that can be used easily in a real-time 3D
environment. The artistic outcomes that can extend from these elements
provide a glimpse into the future; a vision of what will be possible once real-
time technology has caught up with current 3D rendering capabilities.
Through the exploration of motion capture it became clear that multi-scene,
multi-person narratives could be enacted that can combine live action with
real-time screen representation. This approach can also apply to narrative
dance in which the dancers onstage are enacting the scene onscreen.
Multiple characters can be used in a projected scene either by operating
several motion capture suits, an expensive option, by using a scenario where
no two characters are on stage at the same time, or by the mixing of real-time
animation with triggered motion clips. With dance it is a simple operation for a
troupe of virtual dancers onscreen to be driven by the movements of one
dancer onstage.
Dance is clearly a field in which the use of real-time motion capture is likely to
become more common. Real-time 2D effects are already used to great effect
in dance. This is evident in the work of interactive software creator, Frieder
Weiss, whose highly visual motion tracking effects have been used in
Mordake, Glow, Onskebronn and Mortal Engine. The huge range of visual
tools available in 3D software along with the dimensionality of 3D and the
responsiveness of real-time rendering offer an environment that will, in time,
greatly extend the way in which visuals are used.
121
Live human motion can be used to control the lighting rig, or trigger local and
remote actions or control puppets. Through performance animation it is
possible for the mood of an actor or dancer, using body language, to initiate
changes to the lighting, sound and set, generating a sympathetic mood on
stage. The technology is still relatively new and there are some issues with the
quality of the real-time animation produced and limitations on the types of
movement that can be captured. However, it is likely, based on the pace of
technological development, that these limitations will be overcome in the short
term and, with a migration of existing techniques in 3D animation to real-time
performance animation environment, the range of visual possibilities will
expand rapidly.
The primary investigation A Brush with the Real World focused on the use of a
real-time animation environment to interactively communicate with an
audience. It was also an example of the use of improvisation in animation,
using paralanguage as a means of two-way communication between the
onscreen character and the audience. Gestural controls can be applied in
other situations where it is desirable for a person, either on or offstage, to
trigger changes onstage or onscreen, through their movements, their gestures
and their interaction with virtual objects. Dance, with its emphasis on the
controlled motion of the body, may find a gestural interface especially useful.
Chasing Shadows was successful as an experiment but the quality of the
animation is still short of being able to create a seamless interaction between
live and computer generated performance in this form on stage. The ability to
swap between movie clips and live action was established but this transition
needs to be explored further to work seamlessly. Private Eyes is an
investigation into digital storytelling. The misty streetscape, using multiple
particle effects, challenged the real-time capabilities of the software and
hardware. Visually, there needed to be a balance between smooth animation
and the atmosphere created. However, it demonstrated that the parameters
for creating a visual mood that augments a live event are potentially diverse,
particularly when taking into account the huge library of CG effects available in
122
a 3D environment. Dramatically, the onscreen scene can depict many aspects
of the actor onstage, from scenes of reminiscence, thoughts and schemes, to
facets of the alter ego.
Other investigations under the title Motionics have been a practice-led search
for ways in which 3D motion capture can augment dance that are unique, and
distinct from the mature technology of real-time video and 2D effects used
commonly as visual augmentation to modern dance. The ability to dynamically
change camera perspective was important as it allowed for first person and
third person perspectives as well as animated camera tracking. Motion
capture was very useful in creating reactive elements such as scenery that
responds to movement onstage, actions triggered by location and the eyes
that follow the actor in Private Eyes. Other useful elements explored in the
case study were the use of video within the projected scene, as well as
particle trails that accentuate movement over time Through this limited study
a range of tools and techniques were explored, including a visualisation from a
dancer’s view, movement of virtual objects generated from the dancer’s
movements, creating motion trails onscreen and using avatars to augment the
theme onstage.
The findings from the creative practice provide the basis for further exploration
and application of performance animation as a part of live performance. While
the inertial suit was a breakthrough technology at the commencement of the
project, in the past year or so several other suits have been produced that
have solved some of its earlier problems. Each investigation provided some
affirmation and creative direction that, in total, demonstrate that motion
capture has a promising future in live performance and at public events. In the
not-too- distant future there are likely to be highly portable lightweight systems
that are unrestrictive and accurate. Based on current trends these systems will
gradually reduce in cost as the technology matures and the demand moves
outside niche markets into more ubiquitous uses, paving the way for its
greater use in live performance. The impact of motion capture on
performance, however, will not so much open up new ways for performances
123
to be presented, as open up new ways for the public to perform. So what
directions will performance take in its relationship with real-time animation
technologies? That is for performance to explore and for audiences to judge.
124
8 References
2007 Higher Education Research Data Collection, Innovation and Research
Systems Group: Department of Education Science and Training, <http://www.dest.gov.au/Search.htm?query=research%20publications>. Accessed 8/7/08
U.S. Bureau of Labor Statistics Office of Occupational Statistics and
Employment: May 2006 National Occupational Employment and Wage Estimates, 2006, 4/2/2008, <http://www.bls.gov/oes/2006/may/oes_nat.htm#b00-0000>. Accessed 4/2/08
Auslander, P 2002, 'Live From Cyberspace or, I was sitting at my computer
this guy appeared he thought I was a bot', Performing Arts Journal, vol. 24, no. 1, pp. 16-21.
---- 1999, Liveness: Performance in a Mediatized Culture, Routledge, New
York. ---- 'Looking at Records', The Drama Review, vol. 45, no. 1, p. (T169) p77. Barrett, E 2006, Creative Arts Practice, Creative Industries Method and
Process as Cultural Capital, Queensland University of Technology Australia.
Baudrillard, J 1988, 'Simulacra and Simulations (selection)', in Stanford
University Press, pp. 166-84. Benjamin, W 1935, 'The Work of Art in the Age of Mechanical Reproduction',
in H Arendt (ed.), Illuminations, New York. Berezina-Blackburn, V & Miller, B 2005, LANDING/PLACE: �animation in
dance performance, ACCAD, Ohio State University, <http://accad.osu.edu/~vberezin/bm/landing.html>. Accessed 6/3/08
Best, S & Kellner, D 2000, Debord, Cybersituations, and the Interactive
Spectacle, <www.jstor.org/pss/3685437>. Accessed 20/5/06 Biggs, M 2004, Learning from Experience: approaches to the experiential
component of practice-based research, <http://sitem.herts.ac.uk/artdes_research/tvad/mb/2004a.pdf>. Accessed 7/11/06
Bimber, O 2008, 'Visually augmenting the real world with projectors', paper
presented to International Conference on Computer Graphics and Interactive Techniques
ACM SIGGRAPH 2008, Los Angeles.
125
Birringer, J 1999, 'Contemporary Performance/Technology', Theatre Journal,
vol. 51, no. 4, pp. 361-81. Bock, PK 1970, Culture Shock. A Reader in Modern Cultural Anthropology,
Alfred A.Knopf, New York. Brown, B, Gough, P & Roddis, J 2004, Types of Research in the Creative Arts
and Design, University of Brighton, Sussex, UK. Calleson, J 2001, 'Virtual Puppets in Performance', paper presented to
Proceedings from Marionette: Metaphysics, Mechanics, Modernity. International Symposium, 1/4/01.
Carlson, M 2004, 'What is performance', in H Bial (ed.), The Performance
Studies Reader, Routledge. Carlson, W 2007, A Critical History of Computer Graphics and Animation,
<http://design.osu.edu/carlson/history/lesson12.html >. Accessed 15/7/08
Carver, G & Beardon, C 2004, New visions in performance the impact of
digital technologies, Innovations in art and design, Swets & Zeitlinger Publishers, Lisse; Exton, PA.
Clark, D 1998, Case Studies,
<http://www.nwlink.com/~Donclark/perform/casesty.html>. Accessed 17/9/08
Cohen, D & Nighy, B 2006, 'New techniques make visual effects more actor-
friendly: Actors' chops meet techno shops', Variety, 11 December 2006.
Coniglio, M 2004, 'The importance of being interactive', in G Carver & C
Beardon (eds), New Visions in Performance. The Impact of Digital Technologies, Swetz & Zeitlinger B.V., Lisse, Netherlands, pp. 5-12.
Cook, R 2009, interview with the author (spoken, unpublished), 8 August
2009 Daniels, D 2002, ‘Strategies of Interactivity’, transl. T Morrison
<http://www.medienkunstnetz.de/source-text/65/>. Accessed 16/7/05 de Aguiar, E, Stoll, C, Theobalt, C, Ahmed, N, Seidel, H-P & Thrun, S 2008,
'Performance Capture from Sparse Multi-view Video', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2008, Los Angeles.
Debord, G 1967, Society of the Spectacle, Black & Red,
126
<http://www.marxists.org/reference/archive/debord/society.htm accessed Accessed 26/3/05
Delahunta, S 2002, 'Virtual Reality and Performance', PAJ: A Journal of
Performing Art, vol. 24, no. 1, pp. 105-14. Dixon, S 2007, Digital Performance: a history of new media in theatre, dance,
performance art, and installation, MIT Press, Cambridge, Mass. Druckrey, T 1996, Electronic Culture: Technology and Visual Representation,
Aperture Press, New York. Einstein, A 1944, Remarks on Bertrand Russell’s Theory of Knowledge,
<http://evans-experientialism.freewebspace.com/einstein_russell.htm>. Accessed 11/11/06
Felluga, D 2003, Modules on Butler: On Perform, <
<http://www.purdue.edu/guidetotheory/genderandsex/modules/butlerperformativity.html>. >. Accessed 12/5/08
Fischer-Lichte, E 2008, The Transformative Power of Performance: A new
aesthetics, Routledge, London, New York. Garau, M, Slater, M, Pertaub, DP & Razzaque, S 2005, 'The Responses of
People to Virtual Humans in an Immersive Virtual Environment', Presence: Teleoperators and Virtual Environments, vol. 14, no. 1, pp. 104-16.
Gerring, J 2007, Case Study Research: Principles and Practices, Cambridge
University Press, Cambridge, UK. Giannachi, G 2004, Virtual Theatres: an introduction, Routledge, London,
New York. Gleicher, M & Ferrier, N 2002, Evaluating Video-Based Motion Capture,
<http://www.cs.wisc.edu/graphics/Papers/Gleicher/Mocap/videomocap.pdf>. Accessed 7/2/06
Goffman, E 2004, 'Performances: Belief in the part one is playing', in H Bial
(ed.), The Performance Studies Reader, Routledge. Grady, S 2003, Virtual Reality: simulating and Enhancing the World of
Computers, New Edition, Facts on File Inc., New York. Graham, D 1937, transcripts of action analysis class with Bill Tytla, Walt
Disney Studios, <http://www.soe.ucsc.edu/classes/cmps160/Spring05/p35-lasseter.pdf>. Accessed 17/6/06
127
Grigg, L, Johnston, R & Milsom, N 2003, Emerging Issues for Cross-
Disciplinary Research, Higher Education Group, DEST No. 7044.HERC03A.
Hamilton-Grant, I 2001, 'Postmodernism and Science and Technology', in S
Sim (ed.), Routledge Guide to Postmodernism, Routledge, New York, London.
Harris, J & Walsh, B 2004, 'Sorcerers’ Apprentice: reactive digital forms of
body and cloth in performance', in GaB Carver, C (ed.), New Visions in Performance. The Impact of Digital Technologies, Swets & Zeitlinger B.V., Lisse, The Netherlands.
Haseman, B 2006, 'A Manifesto for Performative Research', Media
International Australia incorporating Culture and Policy, theme issue "Practice-led Research", February (118), pp. 98-106.
Heeter, C 'Reflections on Real Presence by a Virtual person', Presence, vol.
12, no. 4, pp. 335-45. Horkheimer, M & O'Connell, MJ 1982, Critical Theory: Selected Essays,
Continuum International Publishing Group. Hurstfield, T 2008, Postmodern Performance Opposing the Aesthetic (the
Beautiful) <http://moderntheatre.suite101.com/article.cfm/postmodern_performance>. Accessed 27/8/2008
Jagger, G 2008, Judith Butler: Sexual politics, social change and the power of
the performative, Routledge, London, New York. Kellner, D 2004, Media Culture and the Triumph of the Spectacle,
<http://www.gseis.ucla.edu/faculty/kellner/ >. Accessed 17/5/05 Kemp, D 1999, New Knowledge, New Opportunities: A Discussion Paper on
Higher Education Research and Research Training, Commonwealth of Australia, DETYA No. 6376HERC99A, <http://www.dest.gov.au/archive/highered/otherpub/greenpaper/fullpaper.pdf>. Accessed 18/5/08
Kent, C 1996, 'Spectacular History as an Ocular Discipline', Wide Angle, vol.
18, no. 3, pp. 1-12. Kershaw, B 2003, 'Curiosity or Contempt: On Spectacle, the Human and
Activism', Theatre Journal, vol. 55, pp. 591-611.
128
Kirshenblatt-Gimblett, B 2004, 'Performance Studies Reader, Second Edition',
in H Bial (ed.), Routledge, New York. Kramer, H 1999, Conversations on art and Performance. , John Hopkins
University Press, Baltimore and London. Krueger, M 1989, Video Place,
<http://au.youtube.com/watch?v=A6ZYsX_dxzs>. Accessed 3/3/07
Krueger, M 1991, Artificial Reality II, Addison-Wesley Publishing Company,
Inc., Reading. Krueger, M & Turner, J 2002, Myron Krueger Live, 23/1/2002 edn, a104,
Arthur and Marilouise Kroker, 13/2/08, <http://www.ctheory.net/printer.aspx?id=328>. Accessed 11/11/07
Lau, M & Kuffner, JJ 2005, Behavior Planning for Character Animation,
<http://graphics.cs.cmu.edu/projects/behavior_planning/behavior_planning_sca05.pdf>. Accessed 7/9/08
Lindsay, A 2000, 'Thinking Inside the box: Kaydera's Filmbox2.0', 3D
Magazine, no. April 2000, pp. 45-7. Liverman, M 2004, The Animator's Motion Capture Guide: Organizing,
Managing and Editing, Charles River Media, Hingham Mass. Lombard, M, Ditton, T., 1998, 'At the Heart of It All: the Concept of presence',
Journal of Computer Mediated Communication, vol. 3, no. 2. Lyotard, J-F 1979, The Postmodern Condition: a Report on Knowledge,
Manchester University Press, Manchester. MacMurray, J 1957, The Self as Agent, Handbook of Action Research, Faber
and Faber, London. Mafe, D & Brown, A 2006, Emergent Matters: Reflections on Collaborative
Practice-led Research, QUT Brisbane, Australia. , <http://eprints.qut.edu.au >. Accessed 25/5/07
Majumder, A, Bimber, O, Raskar, R & Lensch, HPA 2008, 'Projectors for
graphics', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2008, Los Angeles.
Mamoca, <http://www.mamoca.com/>. Accessed 18/7/07 Maslow, A 1968, The farther reaches of human nature, Viking, New York.
129
Menache, A 1999, Understanding Motion Capture for Computer Animation
and Video Games, Morgan Kaufmenn,, San Diego, CAL.; London. Mori, M 1970, 'The Uncanny Valley', Energy, vol. 7, no. 4, pp. 33-5. Mova, <http://www.mova.com>. Accessed 20/7/07 Organic Motion, <www.organicmotion.com>. Accessed 18/7/07 Orliaguet, J-M 2001, Virtual Reality and Peircian Phenomenology, Chalmers
Medialab, Chalmers University of Technology, <http://www.ckk.chalmers.se/ckk/pdf/hci2001-JM_Orliaguet.pd>.
Accessed 5/4/05 Peirce, CS 1978, 'How to Make Our Ideas Clear', Popular Science Monthly,
no. 12, pp. 26-302. Peterson, KY 2004, 'The Emergence of Hyperreality in Performance ', in GB
Carver, C (ed.), New Visions in performance: the Impact of the Digital Technologies, Swets and Zeitlinger B.V, Lisse, The Netherlands.
Phelan, P 1993, Unmarked: The Politics of Performance, Routledge, New
York. Rasker, R 2008, 'Mobile projectors and optical communication', paper
presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2008, Los Angeles.
Reeve, C 2000, 'Presence in Virtual Theatre', Presence: Teleoperators and
Virtual Environments, vol. 9, no. 2, pp. 209-13. Rushkoff, D 2008, The Merchants of Cool,
<http://www.pbs.org/wgbh/pages/frontline/shows/cool/interviews/rushkoff.html>. Accessed 5/10/08
Saltz, DZ 2001, 'Live Media: Interactive Technology and Theatre', Theatre
Topics, vol. 11, no. 2, pp. 107-30. Schechner, R 2002, Performance Studies: an introduction, Routledge,
London and NY. ---- 1994, Performance Theory, New York and London, Routledge.
Scrivener, S 2000, Reflection in and on action and practice in creative-
production doctoral projects in art and design, ISSN 1466-4917, <http://sitem.herts.ac.uk/artdes_research/papers/wpades/vol1/scrivener2.html>. Accessed 19/3/07
130
Slater, M 2003, A Note on Presence Terminology,
<http://www.cs.ucl.ac.uk/research/vr/Projects/Presencia/ConsortiumPublications/ucl_cs_papers/presence-terminology.htm>. Accessed 21/4/06
Smart, J 2004, 'The technology of the Real: Wayne McGregor’s Nemisis', in
GB Carver, C (ed.), New Visions in Performance. The Impact of Digital Technologies, Swets & Zeitlinger Lisse, The Netherlands.
Stake, RE 1995, The Art of Case Study Research, Sage, Thousand Oaks,
California. Steels, L 1991, Towards a theory of emergent functionality, MIT Press,
Cambridge, MA, USA. Sturman, D 1994, 'A Brief History of Motion Capture for Computer Character
Animation', paper presented to Siggraph 1994, Boston MA. Styan, JL Modern Drama in Theory and Practice, Cambridge University
Press, Cambridge UK. Tellis, W 1997, 'Application of a Case Study Methodology', The Qualitative
Report, vol. 3, no. 3. Thacker, E 2000, 'The Science Fiction of Technoscience: The Politics of
Simulation and a Challenge for New Media Art', Leonardo, vol. 34, no. 2, pp. 156-8.
Thompson Klein, J 2000, 'A Conceptual Vocabulary of Interdisciplinary
Science', in N Stehr & P Weingart (eds), Practising Interdisciplinarity, University of Toronto Press, Toronto, Canada.
Turpin, APT, Aylward, MD, Garrett-Jones, DS, Speak, MG, Grigg, DL &
Johnston, PR 1999, University and Industry Research Partnerships in Australia, Commonwealth of Australia, <http://www.dest.gov.au/archive/highered/respubs/uni_partner/execsum.htm>. Accessed12/5/08
Vlasic, D, Adelsberger, R, Vannucci, G, Barnwell, J, Gross, M, Matusik, W &
Popović, J 2007, 'Practical motion capture in everyday surroundings', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2007 papers, Los Angeles.
Vlasic, D, Baran, I, Matusiky, W & Popović, J 2008, 'Articulated Mesh
Animation from Multi-view Silhouettes', paper presented to International Conference on Computer Graphics and Interactive
131
Techniques archive ACM SIGGRAPH 2008, Los Angeles. von Riedemann, D 2007, Is Motion-Capture Animation?, November 13,
Suite101.com Media Inc., Vancouver, Canada, <http://hollywood-animated-films.suite101.com/article.cfm/is_beowulf_animation_pt1>. Accessed 1/10/08
Weibel, P & Druckrey, T 2001, Net_Condition: art and global media, MIT
Press, Cambridge Mass; London. Weiss, F 2008, <http://www.frieder-weiss.de/>. Accessed 8/10/08