Narrativity and the Senses in Virtual Reality

Abstract

This paper will explore the possible ways that storytelling

experience can be more immersive with the implementation of

multiple sensual stimuli. The visual and auditory elements in

Virtual Reality (VR) and their implications on the story and

human perception (through psychological and brain studies)

will be analyzed. The human perception and the brain's best

function in VR will be centralized since a multisensory

experience provides more input for the user to feel familiar and

immersed in the story. The importance of the correlation of the

narrative and the senses will be highlighted since in the virtual

world the senses are the ones that can make the user immerse

by sending feedback to his/her brain. The qualitative approach

to VR storytelling aims to combine existent studies with the

Project Orpheus' production documentation and theoretical

hypotheses.

Author Keywords

Virtual, reality; narratives; soundscape; perception,

multisensory, experience; brain, studies; GPS, cells.

Introduction

Virtual Reality is a medium that tried to emerge several times

but unsuccessfully until now. At the present moment, VR

passes its more fruitful period because the entertainment

industry (games, VR films) is interested in it. VR has been

struggling for decades to surface, the present moment, with

the contemporary technological leaps, seems to be the perfect

timing to investigate through its infinite possibilities regarding

Irene Kalesi

MediaLAB Amsterdam

irenekalesi@hotmail.com

Jorrit Groot

MediaLAB Amsterdam

jorritgroot@hotmail.com

Peter McLaughlin

MediaLAB Amsterdam

petermirrorimageuk@gmail.com

viewer’s engagement through senses’ stimulation and

structured narratives.

Virtual Reality has been viewed by many as an extension of

other media like cinema or just a paratext to video game and filmic/serial franchises. However, things are not that simple. VR is much more than people think. Virtual reality shows how different it is from its predecessors since it requires a new language and a different artistic approach. VR exercises its remediating power on theatre, cinema and video games. It is a medium that has incorporated others (theatre, cinema, games) but at the same time the created amalgam has different possibilities, implications on the user and the overall experience. Bolter and Grusin (2010) stated that “mediation is the remediation of reality because media themselves are

real and because the experience of media is the subject of remediation” (59). Therefore, it is important to view Virtual Reality as an experience-oriented medium that focuses on empathy and improvement of the user’s position and involvement in each storyworld.

The VR world is an experimental ground on which

developers focus on visual gratification and exploration of the

visual aspect of VR relegating the rest of the senses to

second place. Virtual bedazzlement has been the easy way

out for the VR developers but not taking into consideration

the users' limitations, needs and preferences can only

produce bad content which can potentially lead to VR’s

detriment. As James Milward, creator of the Sleepy Hollow

VR Experience said "Bad Content will destroy VR” (BBC news,

2016). Thus, the concentration on visual representation

should be accompanied by narrative depth.

Methods

The paper will follow:

•  a theoretical qualitative approach Virtual Reality viewing it as a new storytelling medium following McLuhan's theory on the power of the medium, the concept of Remediation formulated by Bolter and Grusin as well as Elsaesser’s theory on sound.

•  Brain studies and perception researches will assemble the ways that the user behaves and accepts Virtual Reality.

•  The case study of Project Orpheus will offer the quantitative side of the paper since the visual and auditory examples will be taken and analysed from it.

•  The theoretical research will be tested with the Project Orpheus Experience as its main case study.

1. Visualizing a Narrative

Narratives are the spinal cords of experiences. They represent

ways of approaching the real and fantasy world. Paul Cobley

(2001) explained the term as “a particular form of representation

implementing signs” (6). In most of the media, narratives can be

verbal, gesticular, personal or abstract with the incorporation of

colors and concepts. Therefore, because of their importance, the

narration of stories in VR is even more important since the

medium offers different possibilities for the user’s experience. The

biggest difference is that the position of the viewer is annulled and

the user becomes an organic part of the virtual reality

environment and story.

Since Virtual Reality puts the user in a subjective but

simultaneously ubiquitous position, he can be paralleled to

the omniscient narrator in stories. Acquiring the ability to

have peripheral view in the story’s environment is pivotal in

the way that the story should revolve around the user or

involve her/him in an integral way. According to Sandy

Louchart and Ruth Aylett (2001), “the consideration of the

user and his or her behaviour as a primary resource for the

storytelling system, brings a different perspective to the role

of the user within the story, a character based interactive

storytelling system (18). The narrative should be structured

as if the user is a “visitor” and thus should be guided through

visual/auditory guides or an “investigator” who has to

familiarize with the environment and follow the story on

his/her own understanding. Therefore, when the narration

has an abstract nature, it implies an increase of the flexibility

of the user’s position as well as of the interest levels which

make the people pay more attention to the actions happening

around. Thus, the way the story is presented can help the

person assume the role and experience the formation of

his/her presence around his/her virtual character.

If Marshall McLuhan (1964) considered technology as the

extension of the human central nervous system, then Virtual

Reality could be thought as the best example of a living

organism. In VR, the spherical immersive aspect of the

experience can be linked to self-revitalization and not auto-

amputation since it is a puzzle where human perception

along with the senses and the environment are essential to

the successful experience of VR. The person is not a

spectator or a mere critic but s/he is part of the story and

above all the organic part of the whole technological venture.

However, stories in VR cannot be considered complete

without the careful implementation of sensual stimuli. Sight

has the primary role in the construction of a virtual world

since every VR experience focuses mostly on the visual part.

The narratives that are visualized in VR should be intrinsically

intertwined with visually interesting cues and interactions

with the user. If the user is unconvincingly or without the

proper technological tools introduced to different places then,

the level of immersion decreases and thus the loss of

attention is next to follow.

In Virtual Reality, the human brain, especially the region of

the hippocampus, is trying to map out the reality that it’s

encountering. Space mapping is the process during which,

depth and distances between objects and people are

calculated, so that the brain can move on to memorizing

details and immersing in the environment. According to a

study published in Journal Nature Neuroscience (2014),

experiments on rats have shown that virtual and physical

environments trigger different cognitive processes. In VR, the

place cells or else called GPS cells of the brain function

irregularly in VR in comparison to real environments, so the

brain needs more diverse stimuli in order to space map and

thus be able to immerse in the environment and

consequently the story. The human brains are in need of all

the senses to make sense of the environment around them

and thus when Virtual Reality simply offers visual stimuli, it is

not enough for the cognitive processes to substantiate the

virtual surroundings Thus, the role of the sound is the one

that attains the improvement of the space mapping and the

whole experience.

2. The Power of Sound

A visual narrative can seem interesting to our eyes but the

isolation of the sense of sight is not a right move and most

importantly a challenging process to our brain. "Spatial

perception depends on a synthesis of information from

multiple sources, including all the senses” Loren Frank told

Cynthia Fox from Bioscience Technology. Especially in

Virtual Reality, the auditory experience is the one to be

explored deeply. According to Elsaesser (2010), “It is the

ear that renders the image visible” (145). Thus, if sound

makes our vision better and our sense of presence stronger

then the developers should direct themselves towards

exploring the infinite possibilities of sound.

The implementation of sound and most importantly binaural

sound can improve the brain function of space mapping and

perception that lead to immersion. As Jason Dorrier from

singularityhub.com (2014) explained "positional 3D audio

tricks the ears that it originates in specific locations". This

illusionary recreation of a reality is the key element to

immersion, memory creation and storing. The hippocampus

which is responsible for these processes should be the

object of research because the way this part of the brain

works will provide solutions beyond entertainment purposes.

The most recent advancement in Virtual Sound is the

soundscape which is an organic way of producing and

localizing sound in the space around the user. The sound

distance, the echoing, the sudden changes of the source of

the sounds create a whole new field of possibilities for

developers to measure the levels of immersion and

perception of virtual reality. The sound design progress

offers the users a stimulating new way of interacting with

the environment in VR since the user, the sound and the

surroundings form a living sphere. This experiential sphere

should be orchestrated properly because timing is of the

essence when sound and vision are mingled. In

cybertherapy.info, it is stated that "the feeling of presence

cannot be reached by one sense it is a straightforward

approach to couple different senses to reach an optimal

feeling of presence within a virtual reality".

3. The Curious Case of Project Orpheus

Creating a Virtual Reality experience connected to a television

series as a transmedia paratext but at the same time as an

autonomous product is an interesting and challenging process.

Throughout all the phases of a production, there are some

hypotheses that really make sense on how the film’s narration

would be successful in VR. The premise of the Project Orpheus’

virtual experience is to give the audience an imaginary glimpse of

how an after death experience can be visually represented and

sensed. Since the VR experience is linked to the titular series, the

final film should help the users immerse in an ambiguous after

death environment with eerie and vaguely threatening elements

but also create direct and implicit references to the Project

Orpheus series. More specifically, the concept of Death and the

afterlife is a non-existent, extremely ambiguous formulated by

cultural/religious beliefs and prejudice but also offers a certain

freedom to create a virtual environment.

The visual stimuli of Project Orpheus vary from direct interaction

with the viewer to the formation of a stimulating environment ofr

the user to immerse into it. For example, the main character, the

young boy functions as a visual guide but at the same time as a

multi-positional stimulus. Even though the script of the

experience does not contain speech, the movements of the boys

can be seen as a substitution of dialogue that and a form of

interaction. Moreover, the multiple visual cues in the corridor and

office scene challenge the user’s ability to receive and analyze

different stimuli simultaneously in order to create suspense and

provoke an unsettling atmosphere.

The successful combination of these two senses will be the

gateway for the users to feel immersed and ready to be part

of the virtual world. For instance, the sound in Project

Orpheus is multiple accompanying the correspondent visual

cues but also "misleading" the user in order to create

suspense and awake the sense of exploration in the unknown

territory of VR. During the editing phase of the film, the lack

of sound decreased the impact of the experience. Thus, the

addition of the heartbeat of the user, the heavy breathing

coming from the entities in the film as well as the footsteps

creates an ominous environment. They transmit emotion

since natural noise helps the audience immerse in the

environment and atmosphere. The implementation of the

subtle violin and string instruments fading in the air adds an

eerie layer as transitions in each of the scenes of the

experience.

On the whole, the power of sound is evident when with

primal instincts and connected with fear. Even in the realm of

film, as Elsaesser (2010) indicated “One of the features of

the so-called “new sound picture” is that it puts the spectator

into “free-fall”, in time as well as in space.” (144). Therefore

if you transfer the feeling of free-falling experienced by

spectators in a movie theatre to a virtual reality context, the

sense is magnified and the impact on the user is gravely

amplified. The user doesn’t have to empathize with what the

camera is showing because s/he controls the narrative

he/she is being exposed to. So in the case of the Project

Orpheus is not just an immersion tool but also is linked to

fear and the uneasy feeling of danger.

Conclusion

The combination of multiple senses will improve the users' experience. Storytelling is an organism that feeds from the senses and thus the creation of a stimulating narrative can benefit its meaning. The brain function and the role of psychology should be centralized during the formation of narratives.

The spherical storyworld of VR will bring new dimensions to world building in order to visually create a reality but also acoustically since the sound localization will be the key element to the engagement of the users. Narratives are interesting and possible in VR, so storytellers should invent new techniques to tell stories and implement different sensual input. If the presence of a user is stronger, the story becomes

more interesting and vice versa. Virtual Reality is a multimedium and should be treated as such, thus the narratives should be viewed from multiple perspectives as if in a spherical environment.

Further Discussion

The different forms of narratives that can succeed in VR

should be explored and all the possible combinations of

sensorial stimuli should be tested in order to have a more

complete and immersive experience.

Even if the senses of taste or smell are not to be

implemented in an organic way that does not, the proper

manipulation of sight hearing and touch can stimulate the

other senses too by association or through digging in the

user’s memory storage.

Acknowledgements Thanks to Avrotros, NL Films, for providing everything needed for our case study “The Project Orpheus Experience” a reality. Special Thanks to We Make VR for introducing us to the technical side of Virtual Reality which helped us understand and appreciate the medium even more.

Last but not least, thanks to MediaLAB for funding and support in our project since the beginning

References

1. Aylett, R., and S. Louchart. "Towards a Narrative Theory of Virtual Reality." Virtual Reality 7.1 (2003): n. pag. Web.

2. BBC News. BBC, 29 Sept. 2015. Web. 18 Jan. 2016

3. Bolter, J D, and Richard A. Grusin. Remediation:

Understanding New Media. Cambridge, Mass: MIT Press,

1999. Print.

4. Cobley, Paul. Narrative. London: Routledge, 2001. Print.

5. Dorrier, Jason. "What's Missing from Virtual Reality?

Immersive 3D Soundscapes - Singularity HUB." Singularity

HUB. N.p., 06 July 2014. Web. 28 Jan. 2016.

6. Elsaesser, Thomas, and Malte Hagener. "Chapter 6:

Cinema as Ear." Film Theory: An Introduction through the

Senses. New York: Routledge, 2010.

7. Fox, Cynthia. "Virtual Reality vs. Real Life: How Brain

Neurons Light Up." Bioscience Technology. N.p., 01 Dec.

2015. Web. 28 Jan. 2016.

8. McLuhan, Marshall. Understanding Media: The Extensions

of Man. 1964. Print.

9. "SOUND in VR." Sound in VR. N.p., n.d. Web. 28 Jan.

2016.

10. Wolpert, Stuart. "Brain's Reaction to Virtual Reality

Should Prompt Further Study, Suggests New Research by

UCLA Neuroscientists." UCLA Newsroom. N.p., 24 Nov. 2014.

Web. 28 Jan. 2016.

11. Wlassoff, Viatcheslav. "Brains Aren’t Fooled By Virtual

Reality." Brain Blogger Brains Arent Fooled By Virtual Reality

Comments. N.p., 24 Feb. 2015. Web. 28 Jan. 2016.

.