media transformation: the transfer of media characteristics among media

DOI: 10.1057/9781137474254.0001

Media Transformation

DOI: 10.1057/9781137474254.0001

Other Palgrave Pivot titles

Gerbrand Tholen: The Changing Nature of the Graduate Labour Market: Media, Policy and Political Discourses in the UK

Aaron Stoller: Knowing and Learning as Creative Action: A Reexamination of the Epistemological Foundations of Education

Carl Packman: Payday Lending: Global Growth of the High-Cost Credit Market

Lisa Lau and Om Prakash Dwivedi: Re-Orientalism and Indian Writing in English

Chapman Rackaway: Communicating Politics Online

G. Douglas Atkins: T.S. Eliot’s Christmas Poems: An Essay in Writing-as-Reading and Other “Impossible Unions”

Marsha Berry and Mark Schleser: Mobile Media Making in an Age of Smartphones

Isabel Harbaugh: Smallholders and the Non-Farm Transition in Latin America

Daniel A. Wagner (editor): Learning and Education in Developing Countries: Research and Policy for the Post-2015 UN Development Goals

Murat Ustaoğlu and Ahmet İncekara: Islamic Finance Alternatives for Emerging Econo-mies: Empirical Evidence from Turkey

Laurent Bibard: Sexuality and Globalization: An Introduction to a Phenomenology of Sexualities

Thorsten Botz-Bornstein and Noreen Abdullah-Khan: The Veil in Kuwait: Gender, Fashion, Identity

Vasilis Kostakis and Michel Bauwens: Network Society and Future Scenarios for a Collaborative Economy

Tom Watson (editor): Eastern European Perspectives on the Development of Public Relations: Other Voices

Erik Paul: Australia as US Client State: The Geopolitics of De-Democratization and Insecurity

Floyd Weatherspoon: African-American Males and the U.S. Justice System of Marginaliza-tion: A National Tragedy

Mark Axelrod: No Symbols Where None Intended: Literary Essays from Laclos to Beckett

Paul M. W. Hackett: Facet Theory and the Mapping Sentence: Evolving Philosophy, Use and Application

Irwin Wall: France Votes: The Election of François Hollande

David J. Staley: Brain, Mind and Internet: A Deep History and Future

Georgiy Voloshin: The European Union’s Normative Power in Central Asia: Promoting Values and Defending Interests

Gerbrand Tholen: The Changing Nature of the Graduate Labour Market: Media, Policy and Political Discourses in the UK

Aaron Stoller: Knowing and Learning as Creative Action: A Reexamination of the Epistemological Foundations of Education

Carl Packman: Payday Lending: Global Growth of the High-Cost Credit Market

DOI: 10.1057/9781137474254.0001

Media Transformation: The Transfer of Media Characteristics Among Media

Lars ElleströmProfessor of Comparative Literature, Linnæus University, Sweden

© Lars Elleström 2014

All rights reserved. No reproduction, copy or transmission of this publication may be made without written permission.

No portion of this publication may be reproduced, copied or transmitted save with written permission or in accordance with the provisions of the Copyright, Designs and Patents Act 1988, or under the terms of any licence permitting limited copying issued by the Copyright Licensing Agency, Saffron House, 6–10 Kirby Street, London EC1N 8TS.

Any person who does any unauthorized act in relation to this publication may be liable to criminal prosecution and civil claims for damages.

The author has asserted his right to be identified as the author of this work in accordance with the Copyright, Designs and Patents Act 1988.

First published 2014 byPALGRAVE MACMILLAN

Palgrave Macmillan in the UK is an imprint of Macmillan Publishers Limited, registered in England, company number 785998, of Houndmills, Basingstoke, Hampshire RG21 6XS.

Palgrave Macmillan in the US is a division of St Martin’s Press LLC, 175 Fifth Avenue, New York, NY 10010.

Palgrave Macmillan is the global academic imprint of the above companies and has companies and representatives throughout the world.

Palgrave® and Macmillan® are registered trademarks in the United States, the United Kingdom, Europe and other countries.

ISBN: 978–1–137–47425–4 PDFISBN: 978-1-349-50153-3

A catalogue record for this book is available from the British Library.

A catalog record for this book is available from the Library of Congress.

www.palgrave.com/pivot

doi: 10.1057/9781137474254

Softcover reprint of the hardcover 1st edition 2014 978-1-137-47424-7

DOI: 10.1057/9781137474254.0001 v

Contents

List of Figures vii

Acknowledgments viii

1 Introduction 1

Aims 4

Mapping the Field 6

2 Two Types of Media Transformation 11

Transmediation 20

Media Representation 27

3 The Transmedial Basis 36

The Four Media Modalities 37

Compound Media Characteristics 39

4 A Model for Media Transformation 46

Technical Media and Modality Modes 47

Formula for the Transfer of Media Characteristics 56

5 Three Analyses 62

J. S. Bach – Fantasy in G Minor 64

Jabberwocky 71

Dimensions of Dialogue 80

vi Contents

DOI: 10.1057/9781137474254.0001

6 Conclusion 86

The Border Zones of Media Transformation 88

Final Remarks 93

Bibliography 95

Index 101

DOI: 10.1057/9781137474254.0002 vii

List of Figures

2.1 Transmediation and media representation 162.2 Pure media representation 184.1 Mediation and representation 534.2 Transmediation 544.3 Media representation 544.4 Transmediation 554.5 Media representation 564.6 Figure 4.5 tilted 574.7 Example of extended transmediation 615.1 From Švankmajer, J. S. Bach – Fantasy in

G Minor (3:37) 655.2 From Švankmajer, Jabberwocky (10:20) 735.3 From Švankmajer, Dimensions of

Dialogue (00:46) 81

DOI: 10.1057/9781137474254.0003viii

Acknowledgments

I wish to thank all of my dear colleagues who helped and encouraged me during the research process, particularly those who kindly agreed to comment on the manuscript at a late stage: Anette Almgren White, Jørgen Bruhn, Øyvind Eide, Alice Jedličková, Asunción López-Varela, Liviu Lutas, Beate Schirrmacher, Regina Schober, and Emma Tornborg. Their assistance was invaluable.

DOI: 10.1057/9781137474254.0004

1Introduction

Abstract: In the introductory chapter, I argue that basic research in the area of media transformation – the transfer of media characteristics among media – is vital for further progress in understanding communication in general. I describe the aim of the study as twofold: first, to fuse a number of study areas that have been unduly separated into one overarching field of media transformation research and, second, to form a conceptual model that facilitates a analysis of transfers of media characteristics. The ultimate goal is a theoretical framework that provides a thorough explanation of what happens when cognitive import is changed or corrupted during transfers among different types of media. This chapter also includes a critical discussion of previous attempts to map the field of media interrelations.

Keywords: intermediality; transmediality; media transformation; multimodality

Elleström Lars. Media Transformation: The Transfer of Media Characteristics Among Media. Basingstoke: Palgrave Macmillan, 2014. doi: 10.1057/9781137474254.0004.


DOI: 10.1057/9781137474254.0004

All human beings use media, from gestures and speech to newspapers and computers, and the collaboration of all these media is essential for living, learning, and sharing experiences. Understanding mediality is the key to understanding how meaning, or cognitive import, is created in human interaction, whether directly through the capacities of our bodies (what I call our internal technical media) or with the aid of traditional or modern external technical media.

In the broadest sense of the word, media may be understood as communicative tools constituted by related features. All media are multimodal and intermedial in the sense that they are composed of multiple basic features and are understood only in relation to other types of media. Such an understanding cannot be achieved without adequate and detailed descriptions of common media features. Basic research in this area is vital for further progress in understanding communication in general. However, much of the exploration within the field of media studies – to be understood as the study of all types of so-called art forms (literature, music, film, and so forth) and other types of media (such as speech, gestures, blogs, news, or advertisements, whether or not perceived to be aesthetic) – is carried out without a sound conceptual groundwork.

Although advanced terminology and theoretical sophistication are certainly not lacking, the vast majority of researchers still use largely undefined and deeply ambiguous layman’s terms, such as text and image, to describe the nature of media products. Such terms refer to notoriously vague concepts and, consequently, misunderstanding and confusion are standard features of academic discussions. Attempts to create systematic and comprehensive methodologies and theoretical frameworks fail because the most basic concepts are not clearly delimited. For instance, the term text may refer to media with fundamentally different material, spatial, and temporal properties that are perceived through different physiological senses. Similarly, terms such as image and picture may refer to very different types of media. Consequentially, efforts to understand the relationship between so-called texts and images are doomed to fail. One is referred to nebulous and inadequate ideas of ‘mixtures’ of text and image.

In this study, I attempt to dig deeper. I avoid much of the standard terminology and instead delineate the essential concepts using technical terms and doing some difficult theoretical work. Readers seeking easy solutions should beware. Although brief, this work is not a quick read.

Introduction

DOI: 10.1057/9781137474254.0004

It focuses on the notion that media characteristics – what we accept as information and meaning mediated by separate media – may be trans-ferred to other media.

I suggest that theoretical studies on intermediality may be roughly divided using two main perspectives. The first is a synchronic perspec-tive: how can different types of media be understood, analyzed, and compared in terms of the combination and integration of fundamental media traits? This viewpoint emphasizes an understanding of media as coexisting media products, media types, and media traits. The second is a diachronic perspective: how can transfer and transformation of media characteristics be comprehended and described adequately? This viewpoint emphasizes an understanding of media that includes a temporal gap among media products, media types, and media traits – either an actual gap in terms of different times of genesis or a gap in the sense that the perceiver construes the import of a medium on the basis of previously known media. The focus in this study is on this second, diachronic perspective, including an emphasis on both the notion of transfer – indicating that identifiable traits are actually relocated among media – and the notion of transformation – stressing that transfers among different media nevertheless always entail changes. Whereas I use the term intermedial to broadly refer to all types of relations among different types of media, the term transmedial should be understood to refer to intermedial relations that are characterized by actual or poten-tial transfers.

During the investigation, distinguishing on many levels and creating a number of categories is necessary. However, note that these distinc-tions and categories are primarily intended to facilitate thinking about and analyzing media; they should be understood as distinctions and categories of theoretical perspectives and not as categories of media. Although thinking in terms of different categories of media products or dissimilar types of media is important and far from futile, this type of classification is secondary and becomes problematic unless it is based on a fundamental recognition of the fact that cultural phenomena, such as media, are creations of mental activity – although the mental activity is triggered by material phenomena. Therefore, we must begin with an attempt to understand the manners in which we think about media and how our cognitive activity interacts with the materiality of media during the acts of perception and interpretation. Constructing media categories is possible and meaningful only when we have a fairly solid knowledge


DOI: 10.1057/9781137474254.0004

of how essential concepts should relate to one another and how different theoretical perspectives on media may be distinguished.

Hence, I do not seek to isolate certain media products or media types and state that they are, as such, containers of transferred media charac-teristics. I view the notion of transfer and transformation of media as an analytical perspective: a way to methodically explore media interrela-tions (Elleström, 2010a: 27–35; cf. Schober, 2013: 109). All media prod-ucts can be investigated from both a synchronic perspective in terms of combination and integration and a diachronic perspective in terms of transfer and transformation. Without a doubt, certain media products tend to be remarkably apt for diachronic analysis, with an emphasis on their relations to other, previously existing media products; however, no media product exists that cannot be treated in terms of media transfor-mation without some profit. Media transformation should be understood as a term that covers the diachronic perspective on media interrelations, which goes beyond the general field of media history, that is, the study of how media types evolved and were transformed throughout the centuries.1

Whereas the theoretical investigations in this treatise are most certainly deeply colored by my scholarly background in literature, art, music, philosophy, semiotics, interart, and intermediality studies, their results are widely applicable to the study of all types of media. To the best of my knowledge, the most acute theoretical questions regarding media transformation have been posed in these areas. Works of art constitute only one type of medium but are highly complex creations that differ in degree rather than in type from other media; therefore, wrestling with them teaches us to identify critical problems and theoretical complica-tions that are valid for encounters with a variety of media.

Aims

The aim of this study is twofold. The first aim is to fuse a number of areas of investigation that have been unduly separated into a single overarching model of media transformation research. The second and more ambi-tious aim is to develop a conceptual framework that facilitates a detailed analysis of transfers of media characteristics – traits that may be under-stood as media form or media content. Many characteristics are actually viewed as both form and content, depending on the perspective.

Introduction

DOI: 10.1057/9781137474254.0004

The intermedial field of research includes diverse phenomena marked by transfers of media characteristics. Yet, no systematic account exists of all of these phenomena. The research is compartmentalized, which does not favor a general understanding of media transformation. Most existing studies focus on either just a few media and their specific interrelations or delimited study areas, such as ekphrasis and adaptation, which are not properly related to one another. Unless one compares notions to other notions that involve the transfer of media characteristics, the risk exists of getting stuck in isolated concepts. Delimiting a research area is rather pointless if the delimitation is not based on a thorough understanding of the broader field surrounding it.

Hence, I suggest that existing areas of research such as adaptation and ekphrasis should be fused into a broad conglomerate of transme-dial research based on a common understanding of notions such as medium, mediation, transmediation, and representation, and a wide range of other important notions. In this way, essential correlations and conceptual overlaps among existing research areas, and among the many isolated studies of media transformations that do not fit very well into established research areas, can be detected. Furthermore, an overarching systematic approach to the field of media transformation is more faith-ful to the historic reality of transmedial relations: the limited types of media transformations selected for intensive study may be important but nevertheless exist in a historical context full of related phenomena that risk expulsion from the attention of research simply because they do not fit into existing categories.

The second aim of forming a conceptual framework that facilitates a detailed analysis of transfer of media characteristics should be viewed as an attempt to begin to solve the problem of not yet having a compre-hensive theory for understanding the complex interrelations among the material and the cognitive aspects of these transfers. We need a deeper understanding of how information and meaning are modified – some-times dramatically – when transferred among different types of media (an exceedingly common phenomenon). Ideally, such a theory must include media materiality, sensory perception, semiotics, and cognitive aspects. Beyond the borders of this brief investigation, the ultimate goal is a new and more adequate theoretical framework for media studies that provides a detailed explanation of what happens when cognitive import is changed or corrupted during transfer among different types of media. I believe that an in-depth understanding of such processes appears to


DOI: 10.1057/9781137474254.0004

be an acutely important matter with far-reaching consequences for our understanding of communication at large.

The aspects of similarities and differences among media are necessarily fundamental to this approach. To succeed, a transfer of media character-istics from one medium to another and with different traits requires that the two media nevertheless share similar capacities that, to some extent, bridge their differences. Thus, the axiomatic starting point of the inves-tigation is that a transfer of media characteristics among different types of media always involves transformation to some degree: something is kept, something is added, and something is removed.

In the following section, I critically discuss previous attempts to map the field of media interrelations. In Chapter 2, I begin my own investiga-tion with a rather detailed overview of the field of media transformation in a broad sense, including two fundamental types of media transfor-mations: transmediation of media characteristics and representation of media. In Chapter 3, my next step is to explore the transmedial basis of these phenomena: what are the fundamental media properties that make media transformation possible? In Chapter 4, I sketch an elementary model for analyzing media transformations, which requires a deepened discussion of notions such as technical medium and mediation. This model is envisioned to work for all conceivable media types and to facili-tate a systematic approach to media transformation. In Chapter 5, three short films by Jan Švankmajer as empirical examples are analyzed briefly to illustrate the applicability of the model and the concepts developed. Chapter 6 rounds off with a condensed demonstration of how the theo-retical notions of the treatise may be used to map the border zones of intermedial and intramedial transfers of media characteristics (transfers among different media and among similar media, respectively) and final considerations.

Mapping the Field

As previously stated, the perspective of media transformation, under-stood as the transfer of media characteristics among media, may be understood as one of the two principal parts of the field of media inter-relations – the other being the perspective of media combination and integration. I prefer to conceptualize the notion of media interrelations in this manner given its convenience and fundamental importance in

Introduction

DOI: 10.1057/9781137474254.0004

distinguishing between diachronic and synchronic perspectives on intermedial relations. However, this method is certainly not the only possible way of categorizing media interrelations. In this section, I first rudimentarily sketch the background of media transformation studies and then provide a brief account of a few previous efforts to map the wider field of media interrelations. Needless to say, my surveys are very far from complete; I only attempt to highlight the theoretical approaches that come closest to and are most relevant for the development of my own theoretical framework regarding the transfer of media characteris-tics among dissimilar media.

Although Roman Jakobson was certainly not the first to dwell on diachronic media interrelations, his statement in a linguistic article on translation that ‘intersemiotic translation or transmutation is an interpre-tation of verbal signs by means of signs of nonverbal sign systems’ could be viewed as the starting signal of semiotically oriented media transfor-mation research (Jakobson, 1971 [1959]: 261). However, Jakobson was restricted by the perspective of ‘verbal’ versus ‘nonverbal’ and offered no theoretical tools for analyzing intersemiotic translation. Several decades later, Claus Clüver (1989) developed the notion of ‘intersemiotic trans-position’, which was outlined as an attempt to sketch a general approach to transmedial relations broader than Jakobson’s. Despite its merits, the approach is delimited by the misleading dichotomy of ‘verbal’ versus ‘visual’ texts that obscures the complex nature of overlapping media characteristics.2 Regina Schober (2010) used yet another term, ‘interme-dial translation’. A recent article clarified that this term is designed to be an umbrella term covering various types of ‘intermedial transformation processes’ (Schober, 2011: 77). Because the word translation provides strong associations with transfers among different verbal languages, I prefer to refer to the notion of ‘intermedial transformation processes’ simply in terms of media transformation.

Currently, the most popular study in the field is Remediation: Understanding New Media by Jay Bolter and Richard Grusin (1999), which has been influential and important in highlighting the wide area of media transformations. The book represents an inspiring inquiry, focused on but not delimited to visual new media types. It is full of relevant observa-tions but severely lacks in-depth theoretical discussions on the nature and different forms of ‘remediation’. The authors’ notions of media and remediation are acutely vague. In a way, my own study is an attempt to develop more finely tuned notions that rival the all-embracing concept


DOI: 10.1057/9781137474254.0004

of remediation of Bolter and Grusin, which primarily aims to explain the mechanisms in contemporary media culture.

The work of Werner Wolf is a sharp contrast to Remediation. Wolf ’s The Musicalization of Fiction: A Study in the Theory and History of Intermediality (1999) aims to categorize and systematically investigate intermedial phenomena, and is not restricted only to the field of music and fiction. My distinction between combination and integration of media on the one hand and transfer and transformation of media on the other hand partly corresponds to Wolf ’s distinction between ‘overt’ and ‘covert’ intermediality. However, I find it deeply problematic to state that an arti-fact of ‘overt’ intermediality is distinguished by qualities that are ‘imme-diately discernible on its surface’ and to delimit ‘overt’ intermediality to a narrow category of cases in which ‘the signifiers of two media are apparent and distinct’ (1999: 40, 50). Actually, what is the ‘surface’ of a multimodal medium such as, for instance, film (which includes sound), and what does it mean to say that the signifiers of a medium are ‘appar-ent and distinct’? I argue that, on the surface, signifiers are very rarely distinct in relation to one another, even if one broadly understands the notion of surface. The distinction between overt and covert intermedial-ity requires that the perceiver has a very clear perception of the exact end of the characteristics of one conventionally distinct media type and the start of the characteristics of another conventionally distinct media type.

In an article from 2002, Wolf launched another intermedial typol-ogy. Nonetheless, his new distinction between ‘intracompositional’ and ‘extracompositional’ intermediality can be criticized in the same manner as his distinction between ‘overt’ and ‘covert’ intermediality; they are both based on an understanding of media as containers with clearly perceptible surfaces that separate an inside from an outside. Because media must be understood as cultural phenomena that are constituted within a field that covers cognitive and semiotic aspects and material and sensorial aspects, the notion of media as containers is severely reductive. Although the distinction between ‘intracompositional’ and ‘extracom-positional’ intermediality may be applicable to a certain extent, it is problematic to assume that certain types of intermedial traits are found ‘within’ the media products, whereas others are the results of interactions with something ‘outside’ the media products. The distinction breaks down as soon as one investigates what the ‘inside’ of a media product actually is and finds that it consists only of substance that comes from

Introduction

DOI: 10.1057/9781137474254.0004

the ‘outside’, albeit from different types of external sources (although ulti-mately from our own minds). According to Wolf, a phenomenon such as a novel transformed to an opera – ‘intermedial transposition’ – should count as extracompositional intermediality, whereas the musicalization of fiction – ‘implicit intermedial reference’ (imitation) – should count as intracompositional intermediality (2002: 27–9). Yet, both types of intermediality presuppose an awareness of preexisting media products or media qualities (a certain novel or certain musical characteristics) and, hence, point to something beyond the ‘intracompositional’.

Additionally, Irina O. Rajewsky tends to think of media as contain-ers, which is useful and perhaps even necessary to some extent but also restraining in the long run. Hence, my critique of Wolf ’s categoriza-tions is, to a certain extent, also valid for Rajewsky’s way of reasoning. Nevertheless, my own main category of media combination and integra-tion clearly resembles Rajewsky’s first intermedial category as outlined in her book-length study Intermedialität and in subsequent articles: ‘media combination’ (Medienkombination), exemplified through opera, comics, and so forth (2002: 15–16, 2005: 51–2). Rajewsky’s two other intermedial categories are ‘medial transposition’ (Medienwechsel), which is ‘the trans-formation of a given media product’ as in the case of adaptation (2002: 16, 2005: 51), and ‘intermedial references’ (intermediale Bezüge), which I argue is her most heterogeneous category exemplified by phenomena such as ekphrasis, narrativization of music, and references to painting in film (2002: 16–17, 2005: 52–3).

Because I find it difficult to uphold a difference between ‘medial transposition’ and ‘intermedial references’ in both theory and analyti-cal practice, I choose to make other distinctions within the broad field of transfer and transformation of media. Although I believe that Rajewsky’s two categories should be understood within the frames of media transformation – as outlined in this study – the resemblance between my distinction between transmediation and media representa-tion to be subsequently developed and Rajewsky’s distinction between ‘medial transposition’ and ‘intermedial references’ is only superficial. My division between transmediation and media representation is prob-ably closer to some of Wolf ’s distinctions, although we conceptualize the issues very differently. My notion of transmediation includes both Wolf ’s ‘implicit intermedial reference’ (imitation) and his ‘intermedial transposition’ despite the fact that Wolf treats these two phenomena as very different (intracompositional vs. extracompositional). My notion


DOI: 10.1057/9781137474254.0004

of media representation resembles Wolf ’s ‘explicit intermedial reference’ (thematization), although the semiotic notion of representation as I use it is certainly not equivalent to reference (Wolf, 2002: 27–9).

Theoretical categorizations are vital for all types of science; however, in the end they do not represent absolute realities. They are tools for thinking, indicating that their validity is only proven if they turn out to be helpful for discriminating among things that are worth being discriminated among, and if they help avoid confusion and misconcep-tions. In Chapter 2, I argue for one way of creating order in the complex area of media transformation. Whereas categorizations in studies of intermediality normally aim at classifying media products, media types, or actual media qualities as such, my aim is rather to classify analytical viewpoints. Discussing her intermedial types, Rajewsky importantly notes that ‘a single medial configuration may certainly fulfill the criteria of two or even of all three of the intermedial categories’ (2005: 53). By the same token, I wish to emphasize that even if I may occasionally express myself in ways that seem to indicate that a medium is inherently like this or like that, these qualities are always also a result of an analytical perspective – in this treatise, the diachronic viewpoint of transfer and transformation.

Notes

We find the latter sense of a diachronic perspective on media in, for instance, Rajewsky, 2005: 46–7.Cf. my criticism of oppositions such as this one in Elleström, 2010a: 13–17.

DOI: 10.1057/9781137474254.0005

2Two Types of Media Transformation

Abstract: The basis of this chapter is the distinction between mediation and representation. Whereas mediation is a presemiotic notion that captures the material process of media realization, representation is a semiotic notion designed to explain the process of meaning making. On the basis of this distinction, I launch a fundamental distinction between two types of media transformation: transmediation of media characteristics and representation of media. Whereas transmediation is repeated mediation by another type of medium (exemplified by adaptation), media representation involves the notion of one medium that represents another type of medium (exemplified by ekphrasis). Both types of media transformation may be present in simple and complex forms. This chapter includes investigations into both specific media products and qualified media: historically and culturally formed media types.

Keywords: mediation; transmediation; media representation; semiotics; adaptation; ekphrasis



DOI: 10.1057/9781137474254.0005

My first step toward the goal of forming a conceptual framework that facilitates a detailed analysis of transfers of media characteristics among media is to establish a distinction between transmediation (repeated representation of media traits, such as a children’s book that tells the same story as a computer game) and media representation (representation of another medium, such as a review that describes the performance of a piece of music). Detailed analysis requires an understanding of the fundamental processes of various forms of media transformation. The distinction between transmediation and media representation – thor-oughly elaborated on in this chapter – is based on the fundamental distinction between mediation and representation that has not, according to my knowledge, been properly accounted for by prior research. If these two notions are conflated, a certain degree of confusion may result.

As I define it, mediation is a presemiotic phenomenon and should be understood as the physical realization of entities (with material, senso-rial, and spatiotemporal qualities, and semiotic potential) that human sense receptors perceive within a communication context. For instance, one may hear the sound of a voice.

Representation is a semiotic phenomenon and should be understood as the core of signification, which is delimited in this research to how humans create cognitive import. As soon as a human agent creates sense, sign functions are activated and representation is at work. In this study, the interest is in representation that occurs within the context of communication. For instance, one may interpret the sound of a voice as meaningful words.

As discussed in this research, representation is the creation of mean-ing through the perceptual and cognitive acts of reception. I submit that to say that a media product represents something is to say that it triggers a certain type of interpretation. This interpretation may be more or less hardwired in the media product and the manner in which one perceives it with one’s senses, but it never exists independently of the cognitive activity of the recipient. When something represents, it calls forth some-thing else; the representing entity makes something else – the repre-sented – present in the mind. In terms of Charles Sanders Peirce, a sign or representamen, stands for an object (see, for instance, 1932: CP 2.228–9 [c. 1897]); Peirce’s third sign constituent, the interpretant, the mental result of the representamen–object relation, is bracketed in this study. Whereas representation, the very essence of the semiotic, constantly occurs in our minds when we think without having to be prompted by sensory

Two Types of Media Transformation

DOI: 10.1057/9781137474254.0005

perceptions, it is also triggered by external stimuli; in this context, focus-ing on external stimuli resulting from mediation is appropriate. In other words, representation also occurs in pure thinking and in the perception of things and phenomena that are not part of mediation, but the account of representation in this research is limited to the creation of cognitive import on the basis of mediated sensory configurations – stimuli picked up by our sense receptors.

Therefore, all media products represent in various ways as soon as meaning is attributed to them. Again, following Peirce, I maintain that three main types of representation exist: iconicity (based on similarity between representamen and object), indexicality (based on contigu-ity between representamen and object), and symbolicity (based on a habitual connection between representamen and object) (see, for instance, 1932: CP 2.303–4 [1902]). Furthermore, I suggest three terms to denote the processes of iconic, indexical, and symbolic representation. These terms are widely used for different purposes in diverse contexts, but they fit the rationale of this study: to make possible a systematic approach to the area of media transformation. Hence, I propose calling iconic representation depiction, referring to indexical representation as deiction, and denoting the process of symbolic representation with the term description. Icons, indices, and symbols work together in intricate ways, and most media products create cognitive import through the cooperation of depiction, deiction, and description, even though one type of sign function commonly dominates. Note that the manner in which I use these three terms makes their significance both broader and narrower compared with many other contexts; I annex them only for the purpose of being able to efficiently distinguish verbally among the three main types of signification.

I return to this foundational semiotic trichotomy later. My current emphasis is on the notion that both a presemiotic and a semiotic side exist to basic encounters with media. Whereas the concept of mediation highlights the material realization of the medium, the concept of repre-sentation highlights the semiotic conception of the medium. Although mediation and representation are clearly entangled in complex ways, upholding a theoretical distinction between them is vital. This theoreti-cal distinction is helpful in analyzing complex relations and processes. However, in practice, mediation and representation are deeply interre-lated. Every representation is based on the distinctiveness of a specific mediation. Furthermore, some types of mediation facilitate certain types


DOI: 10.1057/9781137474254.0005

of representation and render other types of representation impossible. As an obvious case in point, vibrating air emerging from the vocal chords and lips that is perceived as sound but not as words is well suited for the iconic representation of bird song, whereas such sounds cannot possibly form a detailed, three-dimensional iconic representation of a cathedral. However, distinctive differences among mediations are frequently subtler and less easily spotted without close and systematic examination.

Hence, I use the term mediate to describe the process of a techni-cal medium realizing presemiotic (potentially meaningful) sensory configurations. For instance, a book page is able to mediate visual sensory configurations that are (once perceived and rudimentarily interpreted) taken to be a poem, a diagram, or a musical score. If equivalent sensory configurations (sensory configurations that have the capacity to trigger corresponding representations) are mediated for a second (or third or fourth) time and by another type of technical medium, I say that they are transmediated: the poem on the page is later heard when it is trans-mediated by a voice. In other words, the poem’s vital characteristics are represented again by a new type of sensory configuration (not visual but auditory signs) mediated by another type of technical medium (not a book page emitting photons but sound waves generated by vocal cords).

In this study, the notion of a technical medium should consistently be understood not as a technical medium of production or storage but of ‘distribution’ in the precise sense of disseminating sensory configura-tions. Without a doubt, a book page is a technical medium of storage and furthermore a technical medium of distribution that disseminates visual sensory configurations. In contrast, sound waves generated by vocal cords do not store sensory configurations but only disseminate them.

The concept of transmediation involves two ideas. Transmediation is not only re-mediation – repeated mediation – but also trans-mediation: repeated mediation of equivalent sensory configurations by another tech-nical medium (remediation should not be understood in the open-ended sense of Bolter and Grusin [1999]). Hence, the composite term transme-dial remediation is actually more accurate for denoting the concept that I seek to circumscribe. For the sake of simplicity, I prefer the brief term transmediation.1 Nonetheless, this term should be understood to refer to notions of both ‘re’ and ‘trans’ – repeated mediation through another technical medium.

All transmediation involves some degree of transformation: the trans-mediated equivalent sensory configurations and the corresponding


DOI: 10.1057/9781137474254.0005

representations that they trigger may be only slightly different and clearly recognizable, but they may also be profoundly transformed; for instance, musical narratives based on literature evidently differ signifi-cantly from their sources. By the same token, an oral news report based on filmed material may transform the essence of the films in more or less radical ways – not only because of a lack of contextual evidence that limits the compass of the report but also because of the fundamental differences between the technical media involved in the transfer of facts. Sound waves partly – but only partly – mediate sensory configurations that are equivalent to those of visual, two-dimensional, moving surfaces; furthermore, descriptions partly – but only partly – represent the same objects as depictions.

Transmediation is the first type of media transformation, and the second type is media representation. Media representation involves the notion of one medium representing another medium. Media representation is at hand whenever a medium presents another medium to the mind. A medium, which is something that represents, becomes itself represented. Clearly, such a representation may be more or less accurate and more or less fragmentary or complete; similar to transmediation, media repre-sentation involves lower or higher degrees of alteration. Because media representation is defined as a kind of media transformation, it is under-stood to involve different types of media. Thus, if one wishes, terms such as transmedial media representation, transmedial representation, or simply transrepresentation may be used to denote the concept that I prefer to call media representation.

Media representation, such as a news article describing a photograph, or a photograph depicting a dance performance, may involve the representa-tion of both media form and media content; a description of a photograph may focus on both its overall composition and its discrete parts, and a photograph of a dance performance may capture both vital interrelations between different dancers and individual dancers and movements.

The notion of one medium representing the characteristics of another medium is obviously one particular instance of representation at large. Media products can represent almost anything. The reason for high-lighting media representing other media is that doing so is part of the diachronic perspective on media interrelations; the representation of media by other media is a kind of transfer of media characteristics.

Although the two types of media transformation – transmediation and media representation – are possible to distinguish theoretically in


DOI: 10.1057/9781137474254.0005

a rather clear-cut way, they are evidently intricately interrelated. The diagrams in Figure 2.1 are attempts to depict, in a general and somehow simplified manner, the distinction between the two.

In the case of transmediation, the target medium (M2) represents the same media characteristics (C1) as the source medium (M1); in the case of media representation, the target medium (M2) represents the source medium (M1), implying that the source medium becomes the media characteristic of the target medium (C2 = M1). As a represented charac-teristic of the target medium (M2), the source medium (M1) still has its own features and should be understood to still represent the same char-acteristics (C1). In other words, in the first case of media transformation, the target medium (M2) transmediates (represents again) the media characteristics of the source medium (M1) or, briefly put, the target medium transmediates the source medium. In the second case of media transformation, the target medium (M2) represents the source medium (M1). In both cases, the media characteristics of the source medium (C1) must, in principle, be understood as recognizable after the transfer from source medium (M1) to target medium (M2).

Of course, the diagrams in Figure 2.1 are schematic. If one applies the notion of sign in an equally schematic manner, and assuming that thinking of a medium as a single sign is accurate, the two types of media transformations as depicted in the diagrams are explained as follows in Peircean terms: transmediation means that the representamen of the

T

Media representation

C1

M1

C1

C2=M1

M2

T

Transmediation

C1

M1

C1

M2

figure 2.1 Transmediation and media representationNotes: M = Medium; C = represented media Characteristics; T = Transfer.


DOI: 10.1057/9781137474254.0005

target medium (its outward appearance) conjures up in the mind of the perceiver approximately the same object as the representamen of the source medium conjures up. Media representation means that the repre-sentamen of the target medium conjures up in the mind of the perceiver both the representamen and the object of the source medium; both the source medium’s outward appearance and its represented media char-acteristics are, at least to some extent, made present through depiction, deiction, or description.

Consequently, media representation may also be frequently under-stood as transmediation. Media representation involves transmediation if it includes, to some degree, repeated mediation of equivalent sensory configurations. No contradiction exists between a target medium on the one hand representing a source medium and on the other hand mediat-ing sensory configurations that are equivalent to the sensory configura-tions of the source medium. This concept might be inferred from the diagrams in Figure 2.1. For instance, whereas a photograph representing a drawing of two garden gnomes is obviously a medium representing another medium, it also clearly includes repeated mediation of equiva-lent and actually very similar (visual) sensory configurations by another technical medium. An auditory, verbal description of a drawing such as, ‘I bought a drawing of two ugly garden gnomes’, is also a case of media representation; however, because it includes repeated mediation of equivalent sensory configurations by another technical medium (the voice has the capacity to produce symbolic signs that represent substan-tial parts of the objects represented by the iconic signs on the paper: the notion of two garden gnomes), it also includes transmediation.

Both examples may be understood as comparatively complex instances of media representation. If a source medium (M1) is represented in some detail, the represented media characteristics of the source medium (C1) become transmediated by the target medium (M2). However, a very simple verbal representation such as, ‘I bought a drawing’, is a media representation but not a transmediation; the represented media characteristics of the source medium (C1, or the garden gnomes in this case) are not represented again by the target medium (M2) – only the representamen (‘a drawing’) and not the object (‘two garden gnomes’) of the source medium (M1) is represented by the target medium (M2). Hence, in pure media representation, only the representamen – the empty shell of the source medium (M1) so to speak – is transferred to the target medium (M2). In pure transmediation, only the represented


DOI: 10.1057/9781137474254.0005

object of the source medium (C1) is transferred to the target medium (M2). Frequently, both representamen and object are transferred, which is to say that both media representation and transmediation are at hand. Then, to be strict, the diagram representing media representation in Figure 2.1 actually depicts media representation including transmedia-tion. Pure media representation should be depicted as in Figure 2.2.

I emphasize that making absolute divisions between the types of media transfers modeled here is not always practically possible; we are dealing with theoretical perspectives that are supposed to be helpful in analyz-ing a complex reality rather than dividing it into categories. Models are schemes that highlight the most important features of something and neglect others, indicating that they are bound to break down if inspected closely. Thus, arguably, transmediation always actually includes some amount of media representation: the moment a perceiver believes that some media characteristics are represented again, these characteristics are recognized as belonging to a source medium, indicating that the source medium becomes present to the mind. To be firm, then, trans-mediation may be viewed as always involving media representation – at least in a secondary manner.

Nevertheless, I believe that the distinction is useful. In the following sections, I first provide an overview of transmediation and then of media representation (cf. Elleström, 2013a: 120–9). I methodically discuss and exemplify the two notions, keeping in mind that they are parts of a diachronic perspective on media interrelations. Thus, the examples should be understood as illustrations of analytical angles rather than as instances of different categories of media.

The outline is structured on the basis of a few distinctions that should be kept in mind. Both transmediation and media representation involve, on the one hand, specific media products (which, for the sake of

T


C1

M1

C2=M1

M2

figure 2.2 Pure media representationNotes: M = Medium; C = represented media Characteristics; T = Transfer.


DOI: 10.1057/9781137474254.0005

simplicity, were assumed so far) and, on the other hand, qualified media. Qualified media is a term that I use to denote media categories – artistic and nonartistic – that are historically and communicatively situated, indicating that their properties differ depending on parameters such as time, culture, aesthetic preferences, and available technologies. Qualified media include classes such as music, painting, television programs, and news articles. A qualified medium is constituted by a cluster of individual media products (Elleström, 2010a: 24–7).

Media products represent and transmediate both other media prod-ucts and qualified media.2 Whereas a novel may describe a particular piece of music, it may also discuss and, hence, represent music in general. A set of gestures may transmediate characteristics of a specific musical piece (as when hands rise and fall in symmetry with a certain melody, resulting in a new representation of the same motional structure) but it may also transmediate general musical characteristics (as when hands move in a rhythmical manner suggestive of typical musical structures). Similarly, qualified media may represent and transmediate both other qualified media and media products. Qualified media, consisting of clusters of related media products, must be understood as accumulat-ing phenomena: if a qualified medium exists that consists of a certain amount of media products that considerably transmediate certain media characteristics, such as motion pictures that use narrative patterns asso-ciated with written literature, the qualified target medium may be said to transmediate characteristics of the qualified source medium.

Hence, the diagrams in Figure 2.1 may be extended to illustrate trans-mediation and media representation that also involves qualified media, in which case M may be understood as either a media product or a quali-fied medium. Subsequent sections of this chapter are divided according to the character of the source medium. For example, ‘transmediation of qualified media’ indicates that examples of transmediation including qualified media as a source medium are discussed.

The next distinction to be employed is between simple and complex transmediations and media representations.3 This division is very coarse indeed and should only be understood as a tool to discern the obvious fact that different degrees of complexity exist in transmediation and media representation. Hence, the distinction between simple and complex is pragmatic rather than essential; the broad extension of the field of media transformations and the character of existing terminology make upholding some measure of differentiation practical.


DOI: 10.1057/9781137474254.0005

In subsequent discussions, I further discriminate between known media and unknown or fictitious media; this difference is sometimes useful but also far from always applicable.4 Finally, I roughly divide media form and media content, which is also understood as a pragmatic way to discriminate in an ad hoc manner between different levels of media char-acteristics that are transmediated or represented. Although useful and difficult to avoid, this manner of differing between media characteristics will be continuously challenged but also never abandoned; the division into media form and media content is an explanatory perspective and certainly not an attempt to objectively categorize media traits.5

Transmediation

As I previously stated, transmediation is at hand when equivalent sensory configurations (sensory configurations with the capacity to trig-ger representations that correspond to those of a source medium) are mediated for a second (or third or fourth) time and by another type of technical medium. Obviously, transmediation occurs all the time in all varieties of communication. Given that we live in a world with many types of technical media and because we use many different senses and types of signs when interacting with one another and the surrounding world, transmediation is essential for our capacity to productively and efficiently disseminate cognitive import. However, finding the definite borders between transmediation and simply mediation in actual cases of communication surrounding us is very difficult and perhaps impossible, and certainly pointless. Instead, transmediation should be understood as an analytical notion designed to understand media processes that are sometimes clearly apprehensible and vital, sometimes indistinct but still crucial, and sometimes only vaguely discernible and of marginal magnitude.

Simple transmediation could be understood as a standard proce-dure of all communication that is so common, unavoidable, and self-evidently present that putting it into a theoretical frame may hardly seem worthwhile. Although most instances of simple transmediation are likely of marginal magnitude, I believe that recognizing their existence is important. In the end, these instances are part of the field of media transformation in their own right; furthermore, they form the basis of and material for more complex transmediations. If achieving simple


DOI: 10.1057/9781137474254.0005

transmediation was not possible, communication would be divided into a number of completely isolated strata confined to insulated media types and nonconnecting cognitive capacities (such as reading visual verbal texts, comprehending speech, understanding gestures, and identifying the visual objects of images).

Simple transmediation of qualified media

Simple transmediation means that a target medium presents sensory configurations equivalent to those of a source medium in that they trig-ger representations of similar, elementary media traits. However, when either the source medium or the target medium, or both, are a qualified medium rather than a single media product, one should think in terms of clusters of media products that tend to present a certain type of sensory configuration. Hence, it is often far from obvious whether talking about transmediation among certain qualified media is meaningful or not.

Simple transmediation of qualified media may be understood in terms of elementary form being transmediated from one qualified medium to another (or to a particular media product). For instance, one may notice that mood contrasts occur in many musical pieces, which is also the case in many motion pictures. Without a doubt, general resemblances between qualified media such as these two can be understood as trans-mediations of simple formal traits. Although one may distrust the point of calling one or the other medium a ‘source’, simple transmediations such as this one are part of the broad field of media transformation.

Simple transmediation of qualified media may also be perceived as elementary content being transmediated from one qualified medium to another (or to a single media product). Many photographs of apples exist, and many news articles about apples exist, implying that the rela-tion between the two qualified media may be understood as transme-diation of simple content. Although the apple factor does not constitute anything essential in either photography or news articles (and nobody would argue that apple photographs are actually the sources of all news articles about apples), banal media interrelations such as this one must also have their place in a theoretical framework that professes to cover the field of media transformation. From a slightly different point of view, one might argue that trivial examples such as this one are important in highlighting the essential fact that different media types have the capac-ity to represent the same or similar objects to a certain extent; basic


DOI: 10.1057/9781137474254.0005

transmedial capacities, such as being able to represent apples, permeate the media landscape and make possible more complex transmediations.

Simple transmediation of media products

Simple transmediation of media products means that a target medium presents sensory configurations that are equivalent to those of a source media product in that they trigger representations of comparable, rudi-mentary media traits. Whereas the target, in principle, may be a qualified medium, I provide only a few examples of media products transmediat-ing simple traits of other media products. Again, the notion of a source and a target medium is noted to become fuzzier as the transmediated media characteristics become simpler.

When watching a drama about a lucky person who turns unlucky, one may notice that the same elementary narrative scheme could be identi-fied in an email that one received the day before. Indeed, the narrative outline is a very simple one that has been repeated millions of times in various media products of different types. Although insignificant, the narrative outline should be understood as simple transmediation of form among media products. Similarly, but less haphazardly, a character from a television commercial may be seen briefly in a graphic novel. This appearance may be said to be a simple transmediation of content between media products if it has no further implications. However, I believe that this example already puts us on the edge of more complex transmediations. As stated, the differentiation between simple and complex transmediation is coarse, to say the least.

Complex transmediation of qualified media

In complex transmediation, a target medium triggers representations of multifaceted media traits similar to those of a source medium. Compared with simple transmediation, complex transmediation is more significant for an in-depth understanding of particular media interrelations. Again, when either the source medium or the target medium, or both, are a quali-fied medium, transmediation is about groups of media products that tend to trigger similar representations. Complex transmediation of qualified media includes transmediation of multifaceted media traits from a quali-fied medium to either another qualified medium or a media product.

I argue that many media products constantly transmediate characteris-tics of qualified media that are not observed by recipients. To learn about


DOI: 10.1057/9781137474254.0005

diachronic media similarities is to become aware of transmediation processes. In principle, although seemingly farfetched, characteristics of fictitious qualified media may also be transmediated. For instance, a novel might describe the structural traits of an invented musical genre (a case of media representation) and actually use these traits as structural elements in its own narration (transmediation of the formal traits of a fictitious qualified medium). Of course, the notion of diachrony is then part of the represented world rather than the real world: one interprets the media interrelation to include a previously existing media type when, in fact, the qualified medium in question – the ‘source’ – is only a crea-tion of the novel – the ‘target’.

Although examples such as this one are interesting to consider and important to note, I see little gain in pushing them too far. Arguably, transmediation of an invented musical genre is not really transmedia-tion. However, it must be remembered that transmediation is part of a diachronic, analytical perspective, and that I do not wish to catego-rize objects and phenomena as such. Media interrelations might be immensely complex, and I settle with the notion that transmediation of fictitious media should also be included in a mapping of diachronic perspectives on media relations.

Media characteristics that can be transmediated comprise all types of features, including aspects of both form and content. For instance, two qualified media or submedia may share narrative characteristics. Many novels, newspaper articles, and motion pictures tell stories about, say, people who are mistreated but struggle to finally obtain redress, which is a fairly complex narrative. Qualified media may also share structural elements of an even more general type. For example, in both music and dance, repeating significant elements of form is common to create distinctive types of structures. Also common are qualified media repre-senting similar motifs. For instance, photographs, poems, and cartoons may all represent sleeping babies wearing pixie caps. Obviously, transme-diation of qualified media characteristics may be exemplified infinitely. Furthermore, because qualified media are largely historical and cultural constructions, their interrelations are disordered, unstable, and subject to constant change. Nevertheless, a multitude of transmedial characteristics may be realized in many ways through various qualified media.

Thus, some media characteristics circulate among several qualified media in a ‘media circuit’ (Punzi, 2007); transmedial traits may be trans-ferred from one qualified medium to another one in more or less evident


DOI: 10.1057/9781137474254.0005

ways. In some cases more than in others, emphasizing the source–target relation between qualified media makes sense, such as when narrative traits of graphic novels or computer games are compared with narration in traditional printed novels. However, transmediations of qualified media characteristics may also include a specific media product as a target. For instance, a media product’s structure may be similar to a conventional form associated with a qualified medium to which the media product does not belong. Examples are an acrylic painting with properties of detail, perspective, and color that make it look like a photo-graph, or a poem structured to make it resemble – in certain aspects – a musical fugue (so-called ‘musicalization’ [Wolf, 1999]).

Complex transmediation of media products

Complex transmediation of media products should be understood as target media presenting sensory configurations that trigger representa-tions of complex media traits comparable to those of source media products. The target media may be either qualified media or other media products.

Complex transmediation of media products is a process of change that leaves a distinct core intact: vital characteristics of a specific media product are transmediated into (as a rule) another media product. This new media product is markedly different, but is also based on, previously existing traits of another media product. In practice, of course, decid-ing on whether a trait or a set of traits originates from another specific media product or is part of a more general pattern in one or more quali-fied media is not always possible or even meaningful. Yet, the difference must certainly be noted, not the least because it may have hermeneutical consequences. For instance, to say that a specific motion picture shares general structural and narrative traits with several qualified media may be true but perhaps not essential for its import. To state that the motion picture has qualities that are actually normally more associated with, say, music than any other qualified medium is to be open to certain inter-pretive possibilities. To conclude that the motion picture is essentially a variant of a specific painting or short story is to push the interpretive impulse even further because the source media product, for good or for worse, inevitably becomes part of the relevant interpretive context once its existence has been acknowledged. That I stick to a general abstract discussion of transmediation and do not enter into the issue of active


DOI: 10.1057/9781137474254.0005

interpretation does not mean that my theoretical perspectives do not have implications for hermeneutical approaches to transmediation.

Usually, saying that a media product transmediates the traits of a fictitious media product makes little sense. Nonetheless, for instance, a novel may be a detailed representation of a fabricated motion picture (a fictitious media representation creates a virtual motion picture); if the representing novel and the represented movie turn out to have clear parallels, then arguably the novel transmediates the fictitious motion picture. However, one must conclude that this transmediation is in itself fictitious and, furthermore, only comes into existence on the basis of an initial media representation: if the fictitious medium is not represented in some detail, perceiving a transmediation of its characteristics is impossible.

Again, transmediation of fictitious media must be viewed as a thought-provoking but marginal phenomenon (at least outside the artistic domains). In contrast, transmediation of actual media products is certainly a marked and widespread phenomenon. All types of form and content may be transmediated from one media product to another or to a qualified medium. Normally, complex transmediation of media products involves the transfer of characteristics from one media product to another. However, sometimes stating that one or a few media products are actually transmediated to a new qualified medium or submedium also makes sense. Sergei Eisenstein (1949 [1944]) argued that the narra-tive traits of Charles Dickens’s novels were seminal for the formation of early American film and, in particular, the influential work of David Wark Griffith. Additionally, a complex of media products, such as Ian Fleming’s James Bond novels, may be transmediated to a new media product, such as a motion picture that does not correspond to a single novel but includes characters and narrative traits common to several novels.

Transmediation from one media product to another might be exem-plified when a media product represents more or less the same event as another, already known, media product. A scene from a motion picture representing a man picking red flowers in the sunset when crying may be understood as a variation of a previously existing oil painting depicting a similar scene. Thus, the represented media content (a man picking red flowers in the sunset when crying) may be said to have been transferred from one media product to another. The painting need not be repre-sented in the motion picture – not even rudimentarily – to stage the


DOI: 10.1057/9781137474254.0005

transmediation. Although the motion picture may actually represent the painting, and additionally transmediate its characteristics, transmediating a specific media product is nevertheless distinct from representing it.

Transmediation from one media product to another may also involve, for example, the transfer of narrative structure. For instance, a novel might represent a pig that is transported to the abattoir, manages to escape, and then meets a rabbit that reveals that the pig is actually a space traveler from Mars. This story may be viewed as a variation of a previously known comic strip about a woman who is sentenced to death, manages to escape from the prison, and then meets a man who claims that her parents were actually monkeys. Thus, the narrative structure may be said to have been transferred from one media product to another.

Transmediated media products and their targets belong to all types of qualified media and submedia, including old and new, artistic and nonartistic; they may be motion pictures, documentary films, novels, essays, computer games, graphic novels, theatre pieces, comics, televi-sion series, and so forth. The existence of various submedia designed especially to be transmediated is evidence of the significant importance of transmediation processes: the libretto is a qualified submedium with which few people engage unless transmediated to an opera, operetta, or musical performance; specialists are able to enjoy the musical score but it is supposed to be transmediated to sounding music; many written dramas require theatrical performances to be properly appreciated; film scripts, screenplays, and the like crave transmediation; and some argue that written poetry is not true poetry until it has been transmediated by the technical medium of voice.

The general term for transmediation of media products to other media products, normally excluding transmediation of libretti, scores, scripts, and so forth, is adaptation. The archetypical adaptation is a novel-to-film transmediation; however, the term has not been reserved exclusively for this specific type of transfer. In A Theory of Adaptation (2006), Linda Hutcheon offers numerous examples of adaptations, including novels, operas, films, and computer games. However, Hutcheon addresses only qualified media with developed narrativity, which is a common way of delimiting the notion of adaptation. In a collection of essays edited by David Francis Urrows (2008), the writers focused on processes of adap-tation from drama to symphony, from drama to opera, from novel to song, from novel to film to opera, from music to novel, and so forth.


DOI: 10.1057/9781137474254.0005

In spite of vital research in the area of adaptation for the past few decades, it is not clear how adaptation is supposed to fit into the broader field of media transmediation. Yet, Kamilla Elliott increased our understanding of the problematic border zones of adaptation. In general, the term adaptation is reserved for transmediation of one media product to another; however, when discussing individual examples of transmediation, distinguishing clearly among general characteristics of qualified media and properties that belong to a certain media product is obviously not possible. Elliott pointed out that common media traits are not always properly treated as parts of the field that she prefers to refer to as adaptation (2003: 113–32), which means that, using my terminol-ogy, Elliott argued that transmediation of qualified media and not only transmediation of individual media products should be included in the notion of adaptation.

Nevertheless, not even all types of transmediation of specific media products tend to be called adaptations. For instance, transmediations from written, visual, and symbolic (verbal) text to oral, auditory, and symbolic (verbal) text – that is to say, reading aloud of texts – or the other way around, are very seldom referred to as adaptation (however, see Groensteen, 1998: 276–7), and similarly for transmediations from nontemporal to temporal images (as in Dalle Vacche, 1996). Sometimes, not even transmediation from film to literature and from literature to film is called adaptation (Paech, 1997). Overall, adaptation researchers do not seem to agree on the proper delimitation of adaptation and, regardless of how adaptation has been delimited, it has only covered bits and pieces of the area of transmediation. Therefore, applying the general notion of media transformation with its two main types of transmediation and media representation, including several analytical subdistinctions, has good reason.


As stated, media representation, which is the notion of one medium representing a different type of medium, occurs whenever a medium presents another signifying unit to the mind. In other words, media representation should be understood as a medium – a representing entity – being represented by another type of medium.

Media representations may be more or less accurate, complete, or elaborated; they may be of marginal interest but also of crucial


DOI: 10.1057/9781137474254.0005

importance to understand the kernel of the representing medium. In line with the rough division into simple and complex transmediation, I propose a coarse split between simple and complex media representa-tion to distinguish among representations that have no major bearing on the overall significance of the representing medium and those that are more multifaceted. Although this division is not the same as the differ-entiation between ‘pure’ media representation and media representation that includes some amount of transmediation, that a correlation exists is clear: a media representation involving only the empty shell of the source medium is liable to be rather simple and a media representation embrac-ing transmediated media characteristics is likely to be fairly complex.

Numerous terms may refer to what I call simple representation. Because thoroughly relating these terms and the many notions that they denote to one another would lead much too far, I only mention some illustrative examples. A media product may refer to another medium, it may suggest, allude to, or hint at another medium, it may mention or name another medium, and it may quote, cite, or comment on another medium. References, allusions, and hints are associated with iconic, indexical, and symbolic representation, whereas primarily symbolic representations – descriptions – are said to mention or name other media. Additionally, quotation, citation, and commentary are chiefly associated with descrip-tion: a quote or a citation is primarily a verbatim reference. Recently, however, quotation, citation, and commentary have become associated with depiction as well, as when various types of visual images are said to ‘quote’ other media products of the same or other types.

Of course, all of these terms may also be used to denote more complex media representations; no one-to-one correspondence exists between common term usage and theoretical distinctions of the type that I promote in this research. However, I believe that the listed terms tend to stand for rather simple types of representation.

Simple representation of qualified media

Examples of media products that represent, in a simple manner, qualified media and general characteristics of qualified media include a novel or a newspaper report that mentions ‘music’, ‘melody’, ‘sonata form’, or ‘rock-’n’-roll’; refers to the style of Renaissance painting or comments on the narrative form of garden programs on television and the typical person-ality types of their main characters; a pop song that mentions phone


DOI: 10.1057/9781137474254.0005

calls or motion pictures or momentarily sounds like a piece of Baroque music; and a photograph of people in a living room that vaguely displays a few posters on the wall. These types of representations may present known, unknown, and possibly fictitious (or yet-to-be-invented) quali-fied media; the latter may be exemplified through a mock documentary film commenting on olfactory telecommunication.

Simple representation of media products

Examples of media products that represent other specific and known media products in a simple manner include an obituary notice in a newspaper that briefly refers to a particular song by Patti Smith, to the national anthem of the Soviet Union, or to Michelangelo’s statue of David; a symphony that quotes the melody of the children’s song ‘Frère Jacques’, thus invoking also the sound of its words; and a soap opera in which the viewer hears fragments of radio jingles and momentarily sees well-known commercial prints.

Cases of media products that represent other unknown or fictitious media products in a simple manner include a short story that quotes a few words from a song that cannot be identified by the reader or that briefly describes the composition of a photograph taken by one of the characters of the story; a scene in a computer game depicting a televi-sion screen mediating a short sequence from an unidentifiable motion picture; and a movie cut that visually and auditorily depicts small pieces of an unknown song performed within the diegesis.

Occasionally, for some perceivers, media representations such as these are not at all simple because they trigger far-reaching associations and interpretations. Sometimes, details are important; at other times, they are only details. Again, we must remember that the concepts and distinctions that I launch are theoretical instruments and methodical perspectives rather than attempts to classify the world of media objects and phenomena out there.

Complex representation of qualified media

Complex media representation occurs when a media product or, indeed, a cluster of media products forming a qualified medium is said to convey representations that are more composite than simple references or comments. Whereas simple representations may certainly have qualified media as sources, complex representations of media tend to call forth


DOI: 10.1057/9781137474254.0005

specific media products in the mind of the perceiver. The more details that are conveyed, the more likely that both an abstract media category and an actual media product are represented.

However, certain detailed descriptions may be the exception, such as comprehensive accounts of the general characteristics of the gothic cathedral or the traditional blues song. Detailed descriptions of qualified media such as these that are not also descriptions of particular media products are only possible if the represented qualified media or subme-dia are formed by a set of truly rule-governed media products; if they are, complex descriptions of form, structure, theme, and motifs may be provided without ever leaving the level of common media characteristics. This scenario is also the case for complex representations of fictitious or unknown qualified media; if one has never been confronted with an actual media product belonging to the represented qualified medium in question, the representation remains linked to the abstract category of a qualified medium.

I believe that comprehensive iconic and indexical media representa-tions must necessarily have specific (existing or fictitious) media products as objects. For instance, whereas a detailed photograph of a comic strip represents a certain media product, imagining a detailed photograph of only the general characteristics of a qualified medium is quite difficult.

If one thinks in terms of a qualified medium representing another qualified medium in a complex manner, it must be acknowledged that qualified media and submedia exist that are characterized by the feature of representing other qualified media in a complex manner. In its most confined form, the genre of ekphrasis consists of poems representing real or fictitious paintings. At least to a certain extent, a qualified medium such as television news may be defined in terms of its multitude of complex verbal representations of (hopefully nonfictitious) media types such as political speeches, written law text, and documentary footage.

Complex representation of media products

I noted that when media products depict, deict, or describe other media in a complex manner, they tend to represent specific media products. These media products may be fictitious, unknown, or known. One exam-ple of an unknown or fictitious media product represented in detail is an unrecognized painting clearly seen in a movie. Arguably, if the movie is based on photography, the painting cannot be completely fictitious (the


DOI: 10.1057/9781137474254.0005

film is an index of an existing painting – even though it may not be a Rembrandt, as claimed in the virtual world of the movie); however, if the movie is, say, computer-generated, the depicted painting may clearly be either wholly fictitious or simply unknown to the beholder (other exam-ples of fictitious media representations are provided in Weber, 2014). Similarly, detailed descriptions of pieces of music in written literature, so-called ‘verbal music’ (Scher, 1968), or the oral accounts of reporters on the radio, represent both fictitious and unknown media products.

Complex representations of media products may be focused on a variety of characteristics: from formal and more abstract traits to features that one tends to relate to content. For instance, whereas form and content of a specific motion picture are indissolubly interconnected in practice, a representation may nevertheless focus on the one or the other (cf. Elliott, 2003: 133–6): when talking about a movie, the plot and dramatic turning points may be described in detail without referring to specific characters or places; however, the peculiarities of all of the characters may also be carefully described without paying much notice to the structure of the narrative.

Complex representations of specific, known media products are of many types. A verbal, visual text such as a description on a website or a short story printed in a magazine may represent in detail both temporal media products such as Handel’s hallelujah chorus from Messiah and nontemporal media products such as the label on a can of Heinz baked beans. Both a monologue in a drama and the voice and gestures of a news correspondent on television may describe the content of Virginia Woolf ’s diary from 13 September 1925. A still photograph may depict and deict the same text from Woolf ’s diary but also may represent Vanessa Bell’s 1912 portrait of the author. A piece of music may deict and depict Martin Luther King’s speech, ‘I Have a Dream’, delivered on 28 August 1963, by including its recorded sound and elaborating its rhythm and sound qualities.

That virtually no limits exist for how media products may be repre-sented by different types of media in various ways is safe to say and does not mean that having a methodical approach to the diversity of complex media representation is impossible. Nevertheless, existing research is generally narrowly focused on specific media types. For instance, in the edited volume, Picture This: Media Representations of Visual Art and Artists (Hayward, 1988), the authors wrote about how audiovisual media such as cinema, video, and television represent visual media such as painting,


DOI: 10.1057/9781137474254.0005

photography, sculpture, and architecture (cf. Kirchmann and Ruchatz, 2014).

Certain types of complex representations of media products generally called ekphrases constitute what Lydia Goehr labeled the modern ‘work-to-work ekphrasis’, to distinguish it from the understanding of the notion in classical rhetoric as any vivid description (Goehr, 2010). Originally, the term ekphrasis referred to ‘a verbal description of something, almost anything, in life or art’ (Krieger, 1992: 7). During the last century, ekph-rasis was primarily understood as a symbolic (verbal) representation of an iconic representation – typically a poem representing a painting – but was also circumscribed in terms of other traditional art forms and, indeed, beyond the artistic domain.

Thus, the notion of ekphrasis has a long history and was delimited in different ways, also within the modern outline of the work-to-work relation, and scholars debated its proper delimitations. Because the various suggested borders of ekphrasis were constructed largely within but also partly outside the limits of media representation, I find that a brief discussion of the ways that ekphrasis has been defined is motivated to highlight the general notion of media products that represent other media products in complex ways.

The modern notion of ekphrasis stems from Leo Spitzer’s idea that ekphrasis is ‘the poetic description of a pictorial or sculptural work of art, which description implies, in the words of Théophile Gautier, “une transposition d’art”, the reproduction through the medium of words of sensuously perceptible objets d’art (ut pictura poesis)’ (Spitzer, 1955: 207). This notion is generalized in James A. W. Heffernan’s oft-cited formula, ‘ekphrasis is the verbal representation of visual representation’ (1993: 3); note that ‘visual’ must be understood as ‘visual and iconic’ to make sense (ekphrasis is not understood to include, for instance, verbal representations of written literature, which is evidently visual). Claus Clüver further widened the notion of ekphrasis when he defined it as ‘the verbal representation of a real or fictitious text composed in a nonverbal sign system’ (1997: 26; the definition is slightly modified in 1998: 49). Clüver emphasized that both the representing and the represented text may be nonartistic, that the represented text may belong to an extensive range of media types, and that the represented text may be fictitious (cf. the notion of ekphrasis in Persin, 1997).

At this point, the target medium is still considered to be verbal (symbolic in semiotic terms), whereas the source medium may be


DOI: 10.1057/9781137474254.0005

any type except verbal.6 This view was contested when Siglind Bruhn radically expanded the notion of ekphrasis to ‘a representation in one medium of a text composed in another medium’ (2000: 8), indicating that both source and target medium may be of any type as long as they are not the same. In principle, then, Bruhn’s notion of ekphrasis covers the entire field of (complex) media representation as I circumscribe it, although we conceptualize this field in dissimilar ways. Bruhn presented her ideas in a major work on musical ekphrasis that she defined as ‘the musical representation of verbal or visual representation’ (2000: 28; cf. 2000: 27, 47) or, more consistently, as ‘the musical representation of a text created in a non-musical sign system’ (2008: 8). After Bruhn, several studies on ekphrasis with nonverbal target media appeared. For instance, Laura M. Sager Eidt (2008) explored photographic and filmic ekphrases of paintings and Ágnes Pethő (2010) wrote about cinematic ekphrasis, or films representing works of art.

Then, in its broadest sense, ekphrasis is said to include a variety of complex depictions, deictions, and descriptions of all types of media prod-ucts. Even if one does not want the notion of ekphrasis to be extended that far for historical or other reasons, it must be acknowledged that complex representations of media products are possible, common, and worthwhile to theorize about far beyond the conventional borders of ekphrasis. In all circumstances, poems representing paintings constitute only a tiny frag-ment of the broad field of complex media representation. As a curiosity, also note that traditional ekphrasis may provide us with examples of the odd phenomenon of qualified media representing media products; a painting such as Pieter Bruegel the Elder’s Landscape with the Fall of Icarus (ca. 1590–95) is represented by such a significant number of poems that they might be understood to constitute a qualified submedium primarily defined by the complex representation of a single media product.

However, ekphrasis is not exclusively understood as media repre-sentation. Theoretical discussions of ekphrasis and, first and foremost, analytical practice tend to include not only media representation but also transmediation. This practice is well in line with my theoretical consid-erations regarding media transformation. Previously, I concluded that media representation comprises transmediation if it includes, to some degree, repeated mediation of equivalent sensory configurations, which is likely the case in complex media representations. As the diagrams in Figure 2.1 clearly indicate, no conflict exists between a target medium both representing and transmediating a source medium.


DOI: 10.1057/9781137474254.0005

Consequently, Clüver was not inconsistent when he marked out ekph-rasis not only in terms of representation but also in terms of ‘intersemi-otic transposition’ and ‘translation [ ... ] from one sign system into another that is based on a different medium’ (1997: 21). Similarly, Bruhn frequently uses the word transformation and coined the term transmediali-zation (2000: 51). When scrutinizing musical ekphrasis, she investigated how composers ‘transform the essence of [an] art work’s features and message, including their personal reaction to it, into their own medium: the musical language’ (2000: xix). In actuality, musical ekphrasis is about ‘a rendering of that representation in musical language’ (2000: 8) rather than about music representing other types of media products; musical ekphrasis is about transmediations of nonmusical media characteristics into musical pieces rather than musical representations of nonmusical media products.

Although including the notion of transmediation might be necessary in all comprehensive investigations of complex media representation and, consequently, in explorations of ekphrasis, recognizing the ekphrastic principle is difficult if the emphasis on transmediation is increased to the point at which media representation totally disappears. This situation occurred in an article by Krieger that demarcated ekphrasis not as repre-sentation but in terms of imitation and equivalence (Krieger, 1998: 4).7

However, arguing for or against different ways of defining ekphrasis (which include more traits than those discussed here) is not my business, except to state the necessity of positioning such arguments within the general framework of media transformation to which the discrimination between transmediation and media representation is vital. Without such positioning, comparing and evaluating notions such as adaptation, ekph-rasis, and others that lack established terms with which to be identified is difficult. Hopefully, this overview clarified that methodical studies of the field must be performed bottom-up, starting with the general theo-retical principles that permeate media transformations at large and then continuing to the immense variety of more or less distinguishable types.

Notes

Charles Suhor used the term transmediation to denote ‘the student’s translation of content from one sign system into another’ (1984: 250). Although such a definition seemingly comes close to my concept of transmediation, Suhor’s


DOI: 10.1057/9781137474254.0005

discussions of transmediation include examples of both transmediation and what I distinguish as media representation.My distinction between transmediating and representing media products and qualified media, respectively, approximately corresponds to Tamar Yacobi’s distinction within the narrower field of ekphrasis: in addition to ekphrases of unique artworks, Yacobi argued that ekphrases exist that represent ‘a visual common denominator (as theme, style, and so on)’ (1998: 33; cf. 1995). Similarly, Emily Petermann (2014) distinguished among novels based on a single musical work and novels based on a musical genre.Again, this distinction between simple and complex transmediation and media representation approximately corresponds to divisions made by Tamar Yacobi within the field of ekphrasis. Yacobi paid attention to the broad field of possible representations in ekphrasis, including both ‘shorter and longer re-presentational stretches of texts’ and ‘description and narration’ (1998: 23).In the field of ekphrasis, distinguishing between representations of real and fictitious works of art, respectively, is common (cf. Clüver, 1997: 26).The subsequent sections ‘Transmediation’ and ‘Media Representation’ are substantially expanded and rewritten accounts of ideas first presented in my article, ‘Adaptation Within the Field of Media Transformations’ (Elleström, 2013a: 120–9).Interestingly, Claire Barbetti argued that in ekphrasis, ‘it is not the visual work of art, the visual representation being represented; it is the perception of the visual representation that is interpreted and translated into a verbal form’ (2011: 11). Consequently, Barbetti held that ekphrasis might also include representations of dreams and visions. Although my task is not to set the limits of ekphrasis, I believe that Barbetti’s point is important. However, it is an open question whether a presumably ekphrastic verbal text is perceived to represent another medium or the perception of another medium; the indisputable fact that the writer in a creative act ‘translates’ her vision – and not the work itself – ‘into a verbal form’ does not necessarily imply that the reader perceives the verbal medium to represent the perception and not the perceived object. Representation is a result of interpretation, and representation of a work and representation of the perception of the same work are not mutually exclusive.The German term Bildgedicht, which commonly refers to poems that depict visual objects (generally called carmina figurata) and poems that describe visual and iconic art products (generally called ekphrases), creates further confusion. In the standard work Das Bildgedicht, Gisbert Kranz distinguished among numerous types of Bildgedichte under headings such as, on the one hand, ‘Transposition’ (1981: 27–46) and, on the other hand, ‘Interpretation’ (1981: 100–21).

DOI: 10.1057/9781137474254.0006

3The Transmedial Basis

Abstract: In this chapter, I explore the fundamental media properties that make media transformation possible: the transmedial basis. The most elementary but profoundly essential transmedial basis consists of the four modalities of media (categories of related media characteristics): the material, the sensorial, the spatiotemporal, and the semiotic modalities. I argue that the observation of differing modes of the modalities is necessary to pinpoint media similarities and differences and demonstrate that modal differences are essential for circumscribing processes of transmediation and media representation. The entities that are understood to be transferred across modal borders are termed compound media characteristics; although based on material mediation, these characteristics are ultimately cognitive entities. In this context, I scrutinize the crucial but problematic distinction between media form and media content.

Keywords: media modalities; media materiality; media perception; space and time in media; media characteristics


The Transmedial Basis

DOI: 10.1057/9781137474254.0006

In the previous chapters, transmediation and media representation have been investigated largely without actually asking how these phenomena are possible. Which media features are involved in the transformational processes encompassing several media types and how are they related? Initially it must be restated that no medium can fully transmediate or represent all media types. Whether one considers qualified media or individual media products, different media have dissimilar basic proper-ties and these divergences set the limits for what can be transmediated or represented. In addition, transmediated and represented media char-acteristics are more or less transmedial; whereas certain traits are almost universally present in the media landscape, other are only transformed to fit dissimilar media to a very limited degree.

I refer to this broad range of media properties and characteristics as the transmedial basis. The transmedial basis provides a setting within which both transmediation and media representation are investigated in further detail – which media characteristics can or cannot be represented or transmediated by various media, and why? These questions may be treated systematically only to a certain extent because the media characteristics that we perceive as vital are frequently the result of complex interpretive practices and advanced contextualization. Inevitably, all analytic approaches to media representation and transmediation are infiltrated using interpretive maneuvers that may turn into pure subjectivity. Yet, this does not mean that constructing a theoretical foundation – with a consistent transmedial terminology attached to it – that facilitates methodical comparison is impossible.1

The four media modalities

The most elementary but profoundly essential transmedial basis consists of what I have called elsewhere the four modalities of media: the mate-rial, the sensorial, the spatiotemporal, and the semiotic modalities (Elleström, 2010a: 17–24). A modality should be understood as a category of related media characteristics that are basic in the sense that all media can be described in terms of all four modalities. All individual media products, and all conceptions of qualified media, may be understood as specific combinations or types of combinations of modes of the four modalities, in other words, unions of particular traits belonging to the


DOI: 10.1057/9781137474254.0006

four modalities. The modes of the modalities far from cover all media characteristics but form a type of skeleton on which all media are built.

Thus, the four modalities of media and, more specifically, the modes of the four modalities, constitute a transmedial foundation that cannot be overlooked if one wants to understand the ways in which more complex media characteristics are represented and transmediated. The flat surface, as a mode of the material modality, is an aspect of computer games, acrylic paintings, posters, printed literature, and so forth, and is a prerequisite for a comprehensive and close transmediation of, say, a graphic novel to an animated cartoon. The audible, a mode of the sensorial modality, is an aspect of instrumental music, recited poetry, motion pictures, weather forecasts on the radio, theatre, and many other qualified media. The easiest way to represent the characteristics of sound media is to produce similar sounds.

Temporality, a mode of the spatiotemporal modality, is an aspect of songs, speeches, gestures, and dance, but not of stills and sculptures. All media are perceived and decoded in time, which is not the same as being temporal in itself. Often, transmediations involve media that are either temporal or nontemporal. Paintings are easily and faithfully transmedi-ated through still photographs, whereas stills only partially transmediate performances. However, some qualified media, such as most printed visual literature, are conventionally decoded in a fixed sequence, which makes them second-order temporal, so to speak, and hence well-suited for transmediation into temporal media such as motion pictures.

Iconicity, a mode of the semiotic modality, is a vital aspect of the production of cognitive import in media such as children’s books, statistical graphs, music, and web sites. Iconic structures create meaning through resemblance. Although similarities are most clearly perceived among visual and auditory phenomena, respectively (a photograph of a boat clearly looks like a boat and a skilled whistler is able to sound just like a blackbird), similarities can be established across both senso-rial and spatiotemporal borders. For example, visual traits may depict auditory or cognitive phenomena, and static structures may depict temporal phenomena; a graph may depict both changing pitch and altering financial status (cf. Elleström, 2010b, 2013c). Transmediation and representation of iconic traits are potentially very complex. In general, iconicity interacts with the two other main modes of semiotic modality: indexicality (based on contiguity) and symbolicity (based on habits).


DOI: 10.1057/9781137474254.0006

Hence, the three modes of semiotic modality are identical to Peirce’s three basic types of representation that were previously observed. As noted, depiction, deiction, and description are not in any way mutually exclusive; as modes of the other modalities, they are often combined to create multimodal media: media that are both visual and auditory, spatial and temporal, iconic and indexical, and so forth.

The observation of differing modes of the modalities is necessary for pinpointing media similarities and differences and, consequently, is essential for circumscribing the processes of transmediation and media representation. Whereas the material, sensorial, and spatiotemporal modalities form the framework for explaining presemiotic processes of mediation, the semiotic modality is the frame for understanding representation. However, differing modality modes do not in themselves determine the limits of media transformation. Shared modality modes may facilitate detailed and extensive media representation and transme-diation, and some media are very difficult to represent or transmediate if the representing or transmediating medium does not include vital modality modes. Nonetheless, given the brain’s cross-modal capacities, media representation and transmediation over the borders of modality modes are, to a certain extent, possible, common, and indeed productive. Works of art may gain value by creating differences instead of maximal accuracy.

In brief, the material, sensorial, spatiotemporal, and semiotic differ-ences between source medium and target medium allow for inventive alterations of media products into new creations. By the same token, modal differences make transferring vital information impossible with-out transforming it, as in news reports that include chains of intercon-nected media.

Compound media characteristics

Whereas the modes of the four media modalities are basic and universal transmedial characteristics, the modality modes are not transmediated or represented in media transformation processes; rather, they are deeply integrated fundamentals that are required to form what I call compound media characteristics: cognitive entities represented by media (cf. Elleström, 2013b: 101–4). These compound media characteristics may be perceived as being transferred among media and, indeed, across modal


DOI: 10.1057/9781137474254.0006

borders; for instance, a certain type of visual composition may be recog-nized as the source of a similar auditory composition.

Whereas a limited amount of media modality modes exists, virtu-ally endless numbers of compound media characteristics are present. To a significant extent, media share modality modes; however, each individual media product has a rather distinct set of compound media characteristics that is created by arranging all details in the full medial expression, as discerned and construed by the perceiver. For example, the songs ‘Stairway to Heaven’ by Led Zeppelin and ‘Non, je ne regrette rien’ sung by Edith Piaf are founded on the same modality modes; in other words, they belong to the same basic medium (Elleström, 2010a: 27–35). They are realized by vibrating air perceived by the faculty of hearing, they unfold temporally, and their meaning is created through both symbolic and iconic signs. On the one hand, sound qualities and relations are conventionally decoded as parts of verbal language and, on the other hand, sound qualities and relations are decoded as resembling, for instance, patterns of emotions and body movements. Yet, the two songs differ markedly because their compound media characteristics are simply not the same: their overall structures, rhythms, melodies, and sound qualifies differ, as do the thoughts, feelings, and events to which their symbolic signs refer.

Compound media characteristics should be understood as features of media products that are apprehended and formed when a structuring and interpreting mind makes sense of the mediated sensory configura-tions; thus, although triggered by material input, compound media characteristics are ultimately cognitive entities. They may be widely spread or strongly linked to a certain modality mode or specific media product. Given that qualified media are groupings of media products, certain compound media characteristics tend to be associated with certain qualified media. Furthermore, their degrees of transmediality differ, which is to say that they may be transferred among different media more or less successfully. However, compound media characteristics are never fully transmedial: the modal differences between dissimilar media always have an effect. As Regina Schober reminds us, ‘There can be no “true“ transmediality, if we assume that every medium operates within a closely knit set of connections between interdependent variables’; in one important respect, ‘media transformations can only highlight the differ-ences between media’ (Schober, 2013: 98). Nevertheless, our minds have the crucial capacity to recognize vital similarities that overbridge media


DOI: 10.1057/9781137474254.0006

variances. Whereas these perceived similarities highlight the differences, similarities with a difference are nonetheless similarities.

Additionally, note that compound media characteristics may originate in the real world. The various details and dramatic structure of the bombing of Pearl Harbor on 7 December 1941 can and have been mediated in many ways: by voices and other sounds, by journal and book pages, by television, cinema and computer screens, and so forth. Nevertheless, these different types of mediation may trigger representa-tions of compound media characteristics that are understood to be the same, albeit not perfectly identical.

The notion of compound media characteristics must not be confused with the notions of complex transmediation or complex media repre-sentation. As I emphasized, the distinction between simple and complex media transformation is rough and pragmatic, and both simple and complex transfer processes involve compound media characteristics; everything that is understood to be transferred among media should be conceptualized in terms of compound media characteristics. Even the least intricate media characteristics that are transferred are compound in the sense that they result from an interpreting mind that forms concep-tions on the basis of a set of modality modes.

I now present an elementary overview of compound media char-acteristics. As continuously observed in the previous chapter, media characteristics that can be transferred among media may be crudely divided into aspects of form and content – which should be understood as a pragmatic, and not an essential, division into different facets of transmedial characteristics. Again, this discrimination between form and content characteristics must not be confused with the distinction between simple and complex media transformation; as already observed, both simple and complex transformations may involve both form and content.

The form and content of media may be identified on many levels. Indeed, what is perceived as content in relation to one type of form may simultaneously be perceived as form in relation to some other content. For instance, the form of a classical symphony may be said to consist of a structure of four movements with different character, which is to say that the symphony contains four movements. In turn, each movement has a certain form – perhaps a more individual outline or a structure that can be captured in a designation such as ‘sonata form’ characterized by the repetition of musical content after a contrasting section. Of course, this


DOI: 10.1057/9781137474254.0006

repeated content might also be perceived as form containing, say, sound qualities, dynamic relations, and melodies. However, every melody has its form, the content of which is the individual notes, and each of these many levels of form and content is transmedial to a certain extent and can appear, more or less clearly, in other media types.

Although thinking in terms of form and content is sometimes prob-lematic, it is unavoidable. The dichotomy is also fundamental for our minds when we make sense of all types of nonmaterial phenomena and when we form ideas and concepts, which is central to our perception and conception of the external world. Content may be understood as ‘entities’ and form may be understood as ‘relations between entities’. As a case in point, the relations among characters, events, and so forth shape the form of a story; similarly, the form of a gesture consists of the overall relations between the various positions of the hands and arms during a certain time span. Whereas form is sometimes indisputably based on exterior qualities, it is also always a creation of the interpreting mind and its inclination to perceive coherent shapes.

Despite their sometimes inaccessible nature, notions such as form and structure prove to have extraordinary longevity, which must mean that they correspond to certain basic perceptual inclinations and fulfill vital cognitive needs. I hence argue that we simply cannot avoid thinking in terms of form. In its most fundamental aspect, form is created from relations among entities in material or mental space, and because spatial thinking is vital for cognition in general, the notion of form is indis-pensable to any effort to describe and interpret media. Consequently, form is an essential general compound media characteristic that covers a complex web of more specific structures of all types of sensorial inputs and cognitive entities. One might say that form is the master trait of media – it is the most broad ranging of all compound media charac-teristics. A visual image has form, audible speech has form, the line of argumentation in a scholarly text has form, and an imagined trip to another galaxy has form. However, when we think of transfer of media characteristics, specific variations of form are understood to be trans-ferred and not form as such, of course.

Thus, form includes a variety of structural relations, manifested sensorially in what can be seen, heard, or otherwise perceived, or as cognitive configurations. A range of notions exists that are derivations of or specific aspects of form: pattern, rhythm, balance, proportion, repetition, and contrast, to name only a few. Other compound media


DOI: 10.1057/9781137474254.0006

characteristics that are said to have a formal nature are style and perspec-tive. The notion that form and its many derivations are transmedial features has long been acknowledged (cf. Greene, 1940: 213–26). Formal transmedial characteristics that were recently scrutinized include irony (Elleström, 2002), framing (Wolf and Bernhart, 2006), description (Wolf and Bernhart, 2007; Wolf, 2008 – of course, the term here refers to the traditional notion), and metareference (Wolf, 2009, 2011; Bernhart and Wolf, 2010). In line with this notion, Jens Schröter noted in a recent arti-cle that, ‘there are forms which appear in identifiable guise in artefacts of varying media provenance’ (2013: 36) and exemplified with the central perspective; he also accurately observed that not all forms are ‘equally available to all media’ (2013: 37). In fact, the degree and exact nature of possible structural similarities among different media have been debated in various contexts for as long as they have been acknowledged. This issue will never be solved at a general level for the simple reason that an abundance of structural relations exists among various types of media entities in a very wide range of media types. However, despite these obstacles, denying that formal media characteristics can actually be transferred, albeit in diverse ways and in different grades, is odd.

Because the quality of being a piece of content is a feature resulting from an interpretive perspective rather than a quality as such, attempting to circumscribe a more or less fixed set of transmedial content entities is pointless. Nevertheless, it must be emphasized that compound media characteristics consist of both entities and structural relations. These entities may be relatively closely connected to the sensation of the mate-rial interface of the media product, such as the appearance of certain symbolic and iconic microstructures, including visual or auditory words and sentences and visual or auditory iconic details. Other compound media characteristics are products of more elaborated cognitive opera-tions, including what we take as representations of situations, spaces, places, persons, animals, objects, and motifs. All of these characteristics are, to a certain extent, transmedial and may all be understood as form from a different point of view.

Allow me to briefly illuminate these discussions through the example of narration. Narration, which clearly includes aspects of both form and content, provides one of the most important sets of compound media characteristics. Whereas narration is traditionally associated with literature and motion pictures, it has increasingly gained the status of a fundamental cognitive notion during the last few decades. Narration is


DOI: 10.1057/9781137474254.0006

an offspring of structured perception and spatial thinking. To narrate and to interpret in terms of narration is to make sense and create cognitive import in sequential form. Generally, narration is built on deiction. No narratives would exist if we were not so prone to apprehend relations of proximity as meaningful; actually, survival would be difficult if humans did not understand that ‘before’ and ‘after’ must often be understood as ‘cause’ and ‘effect’. Narration is based on actions and reactions that form indexical chains and webs.

Narration is not limited to specific material or sensorial modes. Our two most cognitively developed senses – sight and hearing – are both well suited to narration, and all types of spatiotemporal configurations may display traits that are connected to complex narrative sequences. Naturally, media that are temporal on the material level, such as motion pictures and pieces of music, and media based on conventionally sequen-tial sign systems, such as oral and printed literature, have an advantage with respect to forming developed narratives. In addition, media that rely on advanced, symbolic sign systems (in principle, equivalent to verbal language), such as literature and motion pictures, are well suited to outlining complex narratives. Consequently and indisputably, novels and motion pictures generally form more multifaceted narratives than still images, which are nontemporal on the material level and rely prima-rily on iconic sign functions rather than sequentially ordered symbolical signs. Regardless, still images can tell stories.

Accordingly, transferring narratives and narrative traits among different media types is also possible. Marie-Laure Ryan, among others, investigated this phenomenon and explored what she calls ‘transmedial narratology’ (2004: 35). In an elementary manner, the form-content dichotomy may help us understand what is actually transferred when narratives are transformed to other media types. On the one hand, plots and stories (the series of events as they unfold in the temporal decoding of the medium, and the series of events as they are perceived to be related to each other in the represented, virtual world) are two types of narrative sequential structures that are transferred among media. Additionally, the storyworld, which should count as form and includes an elaborate virtual space, may be at least partly transferred among different media. In addi-tion, parts of narratives, such as relations among particular characters or other figurations, may be transferred; clearly, relations among entities of this type are formal narrative features. On the other hand, many features in narratives are perceived as entities. Linda Hutcheon listed a number of


DOI: 10.1057/9781137474254.0006

elements of a story that are certainly compound media characteristics and may be understood as transmedial narrative content, such as characters, motivations, consequences, events, symbols, and themes (2006: 10).

Creating an exhaustive list of transmedial compound media char-acteristics is clearly impossible. Furthermore, the complexity of media characteristics makes neat classification very difficult. Yet, once trans-medial characteristics are identified, their transfer among media may be investigated with some accuracy, as I attempt to demonstrate in the next chapter.

Note

Chapter 3 is partly based on rewritten material from my article, ‘A Theoretical Approach to Media Transformations’ (Elleström, 2013b: 100–4).

DOI: 10.1057/9781137474254.0007

4A Model for Media Transformation

Abstract: In this chapter, I sketch an elementary model for analyzing media transformation intended to work for all conceivable media types and to simplify a methodical approach to media transformation. Whereas the proposed formula, ‘Compound media characteristics are Transferred from a source Medium to a target Medium’, is indeed exceedingly simple in itself, it is framed by developed explanations of notions such as modality modes, technical medium, and mediation, and a series of visual diagrams. The aim of this chapter is to summarize and specify various aspects of the process of transferring compound media characteristics and to point to the complications that arise as soon as one leaves the most clear-cut examples of media transformation.

Keywords: media transformation, media transfer, media modalities, technical medium, mediation


A Model for Media Transformation

DOI: 10.1057/9781137474254.0007

Expressing what I view as the core subject of this investigation may be done in many ways. I primarily used two types of phrasing to frame the same phenomenon. First, I discussed in terms of media transformation: a source medium is transformed into a target medium. Second, I argued that media traits are transferred among media: a source medium’s compound media characteristics are transferred to a target medium. Both conceptions are schemes for understanding media interrelations involving contiguity and continuity; furthermore, they are ways to make sense of media relations that ultimately result from a certain analytical view: a diachronic perspective. Strictly, of course, the source medium is not at all materially transformed; it remains intact after the appearance of a target medium, even though it may certainly be understood differently. Likewise, the compound media characteristics of the source medium cannot be materially transferred to a target medium; because they remain in the source medium, they are cloned onto a target medium.

Nevertheless, our interpreting minds perceive the relation between source and target in this manner: the source medium or the cognitive import of the source medium is apprehended as a mental configuration that is actually transformed – an arrangement consisting of mental entities that can actually be transferred. However, an investigation into the phenomena of transfer and transformation in their actual media environment show that they are far from only mental phenomena but utterly depend on mediation involving diverse material, sensorial, and spatiotemporal capacities that are only partly shared by different media. To properly understand the possibilities and limitations of media trans-formation, and to create a lucid model of the process, we must return with renewed force to the issue of modality modes.

Technical media and modality modes

Why are compound media characteristics more or less transmedial? To the best of my knowledge, this question has not been thoroughly investigated but has not passed unnoticed. Brian McFarlane approached the issue when he distinguished ‘transfer’ from ‘adaptation proper’ and argued that, although transfer is ‘the process whereby certain narrative elements of novels are revealed as amenable to display in film’, adaptation proper is ‘the process by which other novelistic elements must find quite different equivalences in the film medium, when such equivalences are


DOI: 10.1057/9781137474254.0007

sought or are available at all’ (1996: 13). Elements that are ‘not tied to one or other semiotic system’ are transferable; elements that are ‘closely tied to the semiotic system in which they are manifested’ must be adapted (1996: 20). This distinction has its merits to the extent that it highlights the difference between more and less transmedial characteristics or elements; furthermore, it emphasizes that adapted elements are ‘closely tied’ to a specific ‘semiotic system’. Although this statement is to the point, the properties of semiotic systems must be investigated in some detail, and beyond McFarlane’s linguistically colored notion of ‘enuncia-tion’, to reach an understanding of the ‘equivalences’ between semiotic systems. This concept is true for all types of media transformations.

Other researchers who moved toward the decisive differences between media types are Thierry Groensteen, who argued that a media change necessarily affects the manner in which the new medium represents (1998: 275), and André Gaudreault and Philippe Marion, who claimed that ‘any process of adaptation has to take into account the kinds of “incarnations” inherent in this encounter in terms of the materiality of media’ (2004: 61). I fully agree and, moreover, claim that the statement is valid for the field of media transformation at large.

In the subsequent text, I emphasize vital material media differences that cannot be disregarded in an effort to explain the media transforma-tion processes. Yet, there is also a mental aspect to be kept in mind. Claire Barbetti wrote about ekphrasis and noted accurately that, strictly, ‘it is the perception of the visual representation that is interpreted and translated into a verbal form’ (2011: 11). This statement is certainly true whether one thinks in terms of production or reception (although Barbetti argued in terms of production); all semiosis is essentially mental, also when it is generated by sensory input. In Peircean terms, both creation and interpretation of media transformations could be said to be about forming or finding new types of representamens in a different medium that to some extent refer to the same objects as the representamens of the source medium. Whereas this process is very much mental and semiotic, it fundamentally depends on presemiotic factors. The percep-tion of the visual representation is actually interpreted and translated, as Barbetti argued; however, the perception is nevertheless triggered by a material object. Additionally, many different kinds of transformational processes exist; some, such as the description of a visual, iconic medium, are essentially but far from exclusively mental (the two media types have dissimilar material traits that bear on the mental process), whereas


DOI: 10.1057/9781137474254.0007

others, such as the filming of a sketch, are primarily but not solely material (the photographic equipment and its qualities of operation are formed, effectuated, and handled by humans using their inventive minds). In the end, I simply argue that both material and mental aspects must be included when theorizing about media transformation: whereas transformations of both artistic and nonartistic media types is always to some degree effectuated by a creative act, the material differences among media afford certain frames that cannot easily be cut across (see Elleström, 2014).

Thus, clearly answering the question why compound media charac-teristics are more or less transmedial requires scrutiny of the notions of technical medium and media modes. As previously stated, my notion of technical medium does not include aspects of production or storage but is about distribution in the sense of disseminating sensory configura-tions. Whereas technical media necessarily mediate all media products, some technical media may exist independent of specific media products, although in an inactive manner; for example, a television screen exists on its own in disregard to the numerous media products that it mediates. In any case, the technical medium is the entity that mediates; it is the mate-rial presence that is actually perceived. The point is that technical media of distribution are able to mediate only certain modes of the modalities and to different degrees. Strictly, only the material, sensorial, and spatio-temporal modalities concern mediation, although the semiotic modality concerning representation depends on the three presemiotic modalities. Subsequently, I relate all four modalities to the capacities of dissimilar technical media.

The modality modes are at the heart of media transformation. When the transfer of compound media characteristics is constrained by the modal capacities of technical media, or when the technical media allow modal expansion – when the transfer is based on more or less radical (or even minimal) modal changes – it includes a certain degree of transfor-mation. A sweeping gesture representing a large size through similarity transferred to a radio program is drastically transformed because the technical medium of sound waves emerging from loudspeakers is very different from the technical medium of hands and arms. Such a transfer may be performed in at least two ways, with the gesture being trans-formed to either auditory verbal text or organized nonverbal sound such as music. In both cases, corporeal materiality is substituted by a fluid mode of materiality, a visual mode is substituted by an auditory mode,


DOI: 10.1057/9781137474254.0007

and the spatiotemporal mode of a gesture (extending in space and evolv-ing in time) is replaced by a predominantly temporal mode (normally, the spatial character of radio sound is rather limited). Finally, a change of semiotic modes may or may not occur. If the target medium is auditory verbal text, the iconic mode is changed to a primarily symbolic mode, in which case the words may either represent the source medium (describe the gesture) or transmediate it (represent largeness, again with the aid of dissimilar modality modes). If the target medium consists of nonverbal sound, such as music, the iconic mode of the source medium is not changed: both gestures and music are arguably predominantly iconic and both media types can to a certain extent depict the same phenom-ena. A visual gesture may depict largeness through the magnitude of the area that the hands circumscribe and a musical sound may depict largeness through, for instance, a broad range of pitches kept together in melodic form – suggesting expansion. However, sound iconically repre-senting largeness is clearly not exactly the same as visual impressions iconically representing largeness; whereas the two notions of largeness are similar enough to be regarded as equivalent, they are not identical. Consequently, the conclusion is reached that the semiotic options in this example are not restricted as such by the mediating capacities of the technical medium: both source medium and target medium may be iconic. Nevertheless, the iconic aptitude of the target medium is modi-fied by the alteration of material, sensorial, and spatiotemporal modes.

A silent movie depicting a riot transformed to a pencil drawing on paper of the same event is an example of a transfer of media charac-teristics that is less radical regarding technical media dissimilarities but nevertheless clearly affected by the limited modal differences. Both screen and paper are materially solid entities that primarily mediate visual sensory configurations. However, although they are both two dimensionally spatial, the screen is capable of mediating a temporal flow of altering visual formations, whereas the paper only mediates nontem-poral, static visual shapes (once the drawing is completed). Although the movie and the drawing share the same semiotic mode – they are both iconic – this difference in spatiotemporal mode is decisive for the trans-fer of compound media characteristics such as the narrative structure that forms the notion of a riot. Indeed, the transfer of a narration from temporal to nontemporal mode must entail substantial transformation. Although several compound media characteristics associated prima-rily with narrative content may be preserved relatively intact (such as


DOI: 10.1057/9781137474254.0007

the place of the event, the time of day, the look of many of the people involved, and so forth), the narrative form consisting of a series of episodes linked temporally to one another must be severely simplified through the transfer between different spatiotemporal modes. Whereas a temporal depiction of evolving events may be immensely detailed, the static depiction of temporality only amounts to a certain degree of complexity. However, by creating a series of static drawings, the conven-tion of sequential decoding may bridge the modal difference between a nontemporal and a temporal iconic medium.

Transmediation from printed novel to movie is the classical type of media transformation in adaptation studies. Because the book and the cinema or television screen are entirely different technical media, the transmediation of compound characteristics certainly involves several modal changes: the auditory mode is added, the conventional sequentiality of decoding the symbolic signs of the novel (reading) is transformed to the material temporality of the movie (the constantly changing sensory configurations), the degree of iconicity of the visual surface is dramatically increased, and so forth. This addition forces severe transformations. Yet, vital aspects of the narrative form may survive the transmediation from conventional sequentiality to material temporality, many of the features of the characters in the novel may also be expressed in the movie because verbal language and visual depictions significantly overlap with respect to what they are able to represent, and verbal micro structures are easy to transmediate from the visual to the auditory.

Again, the conclusion can be made that compound media characteris-tics are more or less transmedial because they depend on certain modal-ity modes. This conclusion is relevant for both transmediation and media representation: the modality modes of the target medium set the limits for compound media characteristics being represented again by another type of medium, and they set the limits for media being represented by other media; indeed, the represented medium must itself be under-stood as a compound media characteristic and, hence, subject to the same modal constraints as other media characteristics. Furthermore, a medium representing another medium may become the source medium of a transmediation.

Similar to other compound media characteristics, a medium may be represented in numerous ways and a variety of modality modes may be involved even though all media types and media aspects obvi-ously cannot be represented in just any way. One variation of media


DOI: 10.1057/9781137474254.0007

representation is the representation of technical media, which I discuss briefly before moving on to the issue of how technical media and their capacity to mediate only a selection of modality modes is integrated into my visual diagrams of media transformation.

Examples of representations of technical media are a photograph of a painting in a magazine and depictions of a book or a dancing body on a screen. The represented technical medium is sometimes indistinguish-able from what its mediated sensory configurations represent; seeing a photograph of a painting includes seeing what it depicts, and seeing a dancing body includes seeing the dance and perceiving at least parts of its significance. Another example is the depiction of a computer screen in a leaflet. The computer screen is the technical medium of the source medium and the leaflet is the technical medium of the target medium. The latter’s specific capacity of mediating certain modality modes defines the limits of how the technical medium of the source medium is repre-sented: although the screen’s solid materiality and its two-dimensional surface may be depicted visually, its ability to mediate temporally evolv-ing sensory configurations cannot be smoothly represented by the leaflet unless it presents, for instance, a series of spatially distinct representa-tions to be decoded as a succession of moments.

Obviously, some combinations of media modes have limited capaci-ties for representing certain technical media. Imagining, say, detailed and realistic, nontemporal visual iconic representations of the technical medium of vibrating air mediating music or speaking voices is difficult. In contrast, virtually no limits exist for the representation of technical media by either visual or auditory symbolic media; both written and spoken language are very well developed for pinpointing this type of tangible compound media characteristic. In other words, language describes virtually all types of technical media, but certainly not all types of compound media characteristics.

Overall, the importance of technical media and their modal capacities for the transfer of media characteristics is indisputable. However, tech-nical media and the process of mediation are virtually invisible in the diagrams presented in Figures 2.1 and 2.2, in which transmediation and media representation were first diagrammatically depicted. Hence, to achieve greater detail in modeling media transformation, the technical medium and mediation must be included. Therefore, I present a series of visual diagrams with the aim of finally returning to the initial figures with a more thorough understanding of their parts.


DOI: 10.1057/9781137474254.0007

Figure 4.1 presents a diagram that highlights the processes of media-tion and representation: the presemiotic and semiotic fundamentals of how media work. A technical medium mediates sensory configurations that form what is viewed as a media product. This media product has material, sensorial, and spatiotemporal properties determined by the technical medium. A person perceives the sensory configurations of the media product, which triggers semiotic activity, resulting in representa-tions of compound media characteristics. Some of those characteristics are transmedial, which is to say that they can be transferred among media. In Peircean terms, the media product is the representamen that represents (iconically, indexically, or symbolically) media characteristics that constitute the object of representation.

Importantly, note that the diagram in Figure 4.1 should be understood to represent the same as M1 containing C1 as depicted in Figure 2.1. Thus, the M circle in Figure 2.1 must be construed as a conflation of the technical medium and the media product in Figure 4.1: M = TM + MP. Consequently, the notion of a medium containing compound media char-acteristics (Figure 2.1) should be understood as the same as a medium (a media product’s sensory configurations mediated by a technical medium) representing compound media characteristics (Figure 4.1).

The next diagram shows an attempt to once again depict the basics of transmediation but in a slightly different manner. Whereas Figure 4.2 significantly resembles Figure 4.1, an important difference highlights the essence of transmediation: the diagram in Figure 4.2 comprehends TM2 and MP2 and not TM1 and MP1. Transmediation involves a new medium that represents compound media characteristics that have been

figure 4.1 Mediation and representationNotes: TM = Technical Medium; MP = Media Product; C = represented compound media Characteristics; Mn = Mediation; Rn = Representation.

TM1Mn

MP1 C1Rn

Mediation and representation

TM2Mn

MP2 C1Rn

Transmediation

figure 4.2 TransmediationNotes: TM2 = a new Technical Medium; MP2 = a new Media Product; C1 = compound media Characteristics previously represented; Mn = Mediation; Rn = Representation.


DOI: 10.1057/9781137474254.0007

represented before (C1) by another, different type of medium. The object of representation (C1) consists of compound media characteristics that either belong to a single media product or are associated with a qualified medium. If the diagram is to cover the entire field of transmediation, TM2 and MP2 may be further understood as either an individual media product or a cluster of related media products; in other words, a quali-fied medium.Observe that the diagram in Figure 4.2 is intended to depict the same as M2 containing C1, as shown in the diagram of transmediation in Figure 2.1. Again, the M circle in Figure 2.1 is supposed to represent both the technical medium and the media product in Figure 4.2.

Figure 4.3 is an alternative depiction of the fundamental structure of media representation. As is shown, the difference between Figures 4.2 and 4.3 is located to the right in the two diagrams. Similar to transmediation and, indeed, all cognitive import created by media, media representa-tion depends on a technical medium mediating sensory configurations; however, media representation involves double representation: the repre-sentation of a previous representation (C2 = M1). Thus, C2 (the repre-sented medium, M1) and C1 (what the represented medium represents) are two layers of compound media characteristics, so to speak.

In accordance with the two prior illustrations, the diagram in Figure 4.3 should be understood to represent the same as M2 containing C2 = M1 containing C1 as depicted in the diagram of media represen-tation in Figure 2.1. An even stricter diagram of media representation substitutes M1 with TM1 mediating MP1 (in some media representations, this differentiation may actually be discernible). A diagram depicting pure media representation brackets C1.

According to Figure 4.3, as designed, the object of representation (C2 = M1) should primarily be understood as a single media product representing something specific; however, M1 (the source medium) may also be construed as a qualified medium formed by a constellation of similar media products representing more or less diverse things.

TM2Mn

MP2Rn

C2 = M1Rn

C1Media representation

figure 4.3 Media representationNotes: TM2 = a new Technical Medium; MP2 = a new Media Product; C2 = new media Characteristics consisting of M1= another already existing medium representing C1 = media Characteristics previously represented; Mn = Mediation; Rn = Representation.


DOI: 10.1057/9781137474254.0007

Moreover, TM2 and MP2 (the target medium) are understood as either an individual media product or a qualified medium.

Figure 4.4 shows a more complete diagram of transmediation. The lower part of the figure is identical to the diagram in Figure 4.2. The upper part, identical to the diagram in Figure 4.1, was added to clearly stress the fact that the core of transmediation consists of a relation between a source medium and a target medium.

In transmediation, the compound media characteristics of a source medium are represented again. The represented media characteristics (C1) remain the same but are not identical after the transmediation; they are necessarily transformed to some extent by the transfer. The advantage of Figure 4.4 is that the stages of mediation and representation of the two media are depicted in parallel, which makes it clearly visible that the target medium consists of another technical medium mediating different sensory configurations that form another media product representing the same media characteristics in a different way. These exact elements, captured in the four media modalities, constitute the limits – and possibilities – for trans-mediation. Different types of technical media are able to mediate different types of sensory configurations, forming dissimilar types of media products. Consequently, different media products obtain dissimilar material, senso-rial, and spatiotemporal qualities; furthermore, different media products acquire more or less dissimilar semiotic qualities. These are the conditions for transmediating compound media characteristics; some may only be realized through certain material, sensorial, spatiotemporal, and semiotic modes, whereas others are able to bridge modal differences in various ways.

Finally, Figure 4.5 shows a more complete diagram of media represen-tation. The lower part of the diagram is identical to the diagram in Figure

TM2Mn

MP2 C1Rn

TM1Mn

MP1 C1Rn

Transmediation

Source

Target

T

figure 4.4 TransmediationNotes: TM1 = Technical Medium; MP1 = Media Product; C1 = represented compound media Characteristics; TM2 = a new Technical Medium; MP2 = a new Media Product Mn = Mediation; Rn = Representation; T = Transfer.


DOI: 10.1057/9781137474254.0007

4.3. Again, an upper part – identical to Figure 4.1 – was added to stress that the core of media transformation, in this case media representation, consists of a relation between a source medium and a target medium.

Once again, mutatis mutandis, a diagram such as this one reveals that the target medium of a media representation, as the target medium of a transmediation, consists of another technical medium mediating differ-ent sensory configurations that form another media product. A compari-son of the diagrams in Figures 4.4 and 4.5 enables the inference that the elements that set up restrictions and possibilities for media representa-tion are the same as those that constitute the limits for transmediation. Media differences matter for all types of media transformations.

Formula for the transfer of media characteristics

We now return to Figure 2.1 with a fuller understanding of its details. Tilting Figure 4.5 (depicting media representation) to the left and adding two spheres does not change its substance but slightly changes its appear-ance (Figure 4.6). Figure 4.4 (depicting transmediation) may be similarly modified, clearly showing that Figures 4.4 and 4.5 in their new shapes are virtually identical to the transmediation and media representation diagrams in Figure 2.1. The only essential difference is that Figure 2.1 is designed in a less detailed manner: the process of a technical medium mediating sensory configurations that form a media product is conflated to the simplified notion of a medium.

On the basis of the more multifaceted understanding of this initial figure that we gained during the last chapters, I now propose a simple

TM1Mn

MP1 C1Rn


Source

Target

T

TM2Mn

MP2Rn

C2 = M1Rn

C1

figure 4.5 Media representationNotes: TM1 = Technical Medium; MP1 = Media Product; C1 = represented compound media Characteristics; TM2 = a new Technical Medium; MP2 = a new Media Product; C2 = new media Characteristics consisting of M1= another already existing medium representing C1 = media Characteristics previously represented; Mn = Mediation; Rn = Representation; T = Transfer.


DOI: 10.1057/9781137474254.0007

and flexible formula for describing the fundamental traits of media transformation including both transmediation and media representation. ‘Trans’ means ‘across’ or ‘beyond’ and transfer as well as transformation must be understood as spatiotemporal notions in that compound media characteristics are transferred (and simultaneously transformed) from one physical or cognitive place to another; first they are ‘here’, and then they are ‘there’. First we play a computer game, and then we see a motion picture at the movies and recognize it as more or less the same story. First a sculpture is placed on the square, and then we see photos and read descriptions of it in the newspapers.

Hence, media transformation may be captured in the formula, ‘Compound media characteristics are Transferred from a source Medium to a target Medium’, or ‘C are T from M1 to M2’. This formula is deduced from Figure 2.1. The transfer is a transmediation, a media representation, or a combination of the two. In any case, transformation is involved.

Although the formula adds nothing new to the previous descriptions and depictions of media transformation in this investigation, it may work as a methodological tool for easily identifying the core aspects of transfer proc-esses. Thus, when thinking further about the variations and complexities of media transformation, one might consider what the component parts of the formula actually stand for and the flexibility of their interrelations. The formula in itself may be simple but also may reveal a rather messy world of transfers and transformations when applied to media reality. In my view,

Med

ia re

pres

enta

tion

Sour

ce

TM1

Mn

MP1

C1

Rn

Targ

et

TM2

Mn

MP2

RnC

2 =

M1

RnC

1

T

figure 4.6 Figure 4.5 tilted


DOI: 10.1057/9781137474254.0007

this point is exactly why theoretical models are created: they should be helpful to discern actual patterns – in the world around us and in how our minds work – for obtaining an indication of the sometimes elusive complexity of these patterns and, ultimately, to reveal their own shortcom-ings when confronted with an immensely complicated actuality.

Thus, what do C, T, M1, and M2 represent? In the subsequent pages, I repeat basic aspects of these components that were already either thor-oughly discussed or at least touched on. My aim is to sum up, to specify some facets of the process of transferring media characteristics, and to clearly indicate the complications that arise as soon as we leave the most clear-cut examples of media transformation.

C stands for Compound media characteristics, which should be understood as media features that are apprehended and formed when structuring and interpreting minds make sense of mediated sensory configurations. As explained rather recently, these transmedial char-acteristics, which are both transmediated and represented, may be understood in terms of form and content. The virtually infinite list of compound media characteristics includes overarching features such as form, structure, rhythm, narration, material, theme, motif, and so forth. I do not dwell on this issue again in this study.

M stands for Medium. The model of media transformation is built on the fact that media are ultimately large sets of single media products that work as communication tools in the broadest sense of the notion. Every media product is realized through a technical medium that mediates certain sensory configurations. However, media products tend to be clas-sified (with shifting degrees of accuracy). One way of classifying media products that I hardly address in this study is to group media products exclusively in terms of their modality modes. I call these clusters basic media. Another way to classify media products is to consider not only modality modes but also media traits that are historically and communi-catively situated, in the sense that they must be seen in light of parameters such as period, culture, society, and aesthetics. I call these clusters quali-fied media. Because qualified media are more complex media categories that clearly tend to be formed around basic media, settling with the inclusion of qualified media in discussions of media transformation is frequently sufficient, although the categories of basic media are actually more directly related to the mediating capacities of technical media.

More specifically, then, as demonstrated in some detail in Chapter 2, M stands for either Media Product (MP) or Qualified Medium (MQ);


DOI: 10.1057/9781137474254.0007

furthermore, the media may be known, unknown, or fictitious. M1 is the source medium, the ‘first place’ that harbors certain characteristics, and M2 is the target medium, the ‘second place’ perceived as holding the same characteristics, to a certain extent. Importantly, note that M1 and M2 must not necessarily be understood as complete media; considering only parts of the source medium, the target medium, or both may be relevant. When thinking of transfer of media characteristics in the most straightforward manner, M1 and M2 are two particular media products: ‘C are T from MP1 to MP2’. However, media transformation also occurs in cases in which M1, M2, or both are qualified media or submedia (genres) rather than particu-lar media products. The formula might then be developed to ‘C are T from MQ1 to MQ2’, ‘C are T from MP1 to MQ2’, or ‘C are T from MQ1 to MP2’.

Finally, T stands for Transfer, a notion no less complicated than the other ones in the formula. First, of course, T includes both transme-diation and media representation. Because this conceptual couple was already thoroughly analyzed, I instead scantily scrutinize three further aspects of the transfer of compound media characteristics.

The first aspect concerns the ‘thickness’ of the T arrow, which includes a cluster of interrelated factors. To start with, transfers are said to be stronger or weaker. We noted that, as a rule, transmedial characteristics are more or less modified by the modal changes involved in a media transformation; crossing the borders between material, sensorial, spatiotemporal, and semiotic modes certainly affects the perceived strength of the transfer.1

The perceived thickness of the T arrow also involves differences in the degree between what I call simple and complex media transformation: the greater the complexity of successfully transferred media character-istics, the thicker the T arrow. In a comparable manner, the transfer of compound media characteristics encompasses degrees of more or less complete transfers. A single media transformation may involve only one, a few, or several compound media characteristics; the more characteris-tics it involves, the more complete it is considered and the thicker the T arrow. One might also consider the issue in terms of more or less direct transfers. Walter Bernhart wrote about literature and music and empha-sized that thinking of adaptation ‘in terms of a direct transposition from one medium into another’ may be difficult (2008: 25). I claim that this concept is valid for all types of media transformation. Considering inter-mediate stages between what is construed as source and target media is sometimes necessary; these stages may be difficult to pinpoint because they involve networks of transitional mediations and representations. In


DOI: 10.1057/9781137474254.0007

any event, I claim that the experienced directness of the media transfor-mation is partly proportional to the thickness of the T arrow. Ultimately, if the T arrow is perceived as very thin, a valid question is whether the media relation should be treated as a transformation at all.

Although the transfer of media characteristics is basically theorized in terms of transfer among media products, the principles sketched here regarding the thickness of the T arrow are also valid for the transfer of media characteristics among qualified media.

The second aspect concerns the ‘direction’ of the T arrow. In a straight-forward standard transfer, the arrow points from M1 to M2 (as in Figure 2.1). If this type of arrow is not present in some way, simply no diachronic perspective is being applied and, hence, no real media transformation occurs as the concept is circumscribed in this study. In the case of media representation, the direction of the arrow is never in doubt: one particular media product (MP2) or qualified medium (MQ2) represents another media product (MP1) or another qualified medium (MQ1). No question arises regarding what represents and what is being represented. In this respect, transmediation is less clearly confined. When qualified media are involved in transmediation, the chronological relations are frequently vague, indicating that a diachronic perspective on media interrelations may be somewhat flexible. When both M1 and M2 are qualified media or submedia, there is simply not always a point to say that compound media characteristics are transferred in only one direction. Instead, saying that they go back and forth between MQ1 and MQ2, as in the development of forms and motifs in modern literature and film, might be better. Thus, the T arrow may sometimes be said to point in two directions rather than one.

The arrow may also be understood to point in both directions between two media in another way. Even if one identifies a more definite source media product that actually precedes the target medium, from a herme-neutical point of view, the source may be said to be affected by the target: we understand the old media product differently in light of the new media product (see Groensteen, 1998: 277; Bruhn, Gjelsvik, and Thune, 2011; Bruhn, 2013). Then, in a sense and as a consequence of media trans-formation being defined as an analytical diachronic perspective on media interrelations, the media characteristics from a new media product may be understood to be transferred to an old media product; the source may become the target. This phenomenon is true for both transmediation and media representation, and is valid for qualified media: new media types that build on old ones may change our view of the old ones.


DOI: 10.1057/9781137474254.0007

The third aspect concerns the ‘extension’ of the T arrow. Individual transfers may also be viewed as parts of more far-reaching and complex networks involving many specific media products (MP3, MP4 ...) or qualified media (MQ3, MQ4 ...). Several source media may exist that are transformed into one new medium and, indeed, one and the same source medium may be transformed into several target media.2 This is the case for both transmediation and media representation. Figure 4.7 illustrates one example: two qualified media that are transmediated to one media product, as when an advertisement borrows traits from both concrete poetry and comics, or when a photograph has the appearance of both a classicist painting and a scene from a theatre drama.

Furthermore, the T arrow may be part of chains of arrows. A medium that is the target in one specific case of media transformation may be viewed as the source of yet another target. In the end, most media prod-ucts and qualified media may well be knitted together in a giant and ever unfinished web of media transformations.

Notes

Similarly, iconicity crossing modal borders is perceived as stronger or weaker (see Elleström, 2013c: 105–9).This observation is preceded by Tamar Yacobi’s model of ekphrastic relations. She noted that both the source and the target in an ekphrasis might be ‘one’ or ‘many’ (1995: 601–3; 1998: 25; 2013).

T

T

C2

MQ2

MP3

C1

MQ1C1+2

figure 4.7 Example of extended transmediationNotes: MQ = Qualified medium; MP = Media product; C = media Characteristics; T = Transfer.

DOI: 10.1057/9781137474254.0008

5Three Analyses

Abstract: This chapter focuses entirely on three empirical examples that are analyzed briefly to demonstrate the applicability of the theoretical framework developed in the previous chapters of the treatise. The examples, analyzed as target media of media transformation, are three animated short films by Jan Švankmajer: J. S. Bach – Fantasy in G Minor (1965), using music as the source medium; Jabberwocky (1971), using literature as the source medium; and Dimensions of Dialogue (1982), using painting as the source medium. These three films are methodically and narrowly scrutinized using the concepts established in the study.

Keywords: media analysis, animated film, Jan Švankmajer


Three Analyses

DOI: 10.1057/9781137474254.0008

Formulas and visual diagrams are invaluable as tools for thinking. They help us to schematically isolate the most vital entities of complex phenomena and put them in relation to one another. However, the results may be absurd if they are developed too far; in the end, the immense complexity of cognitive operations escapes the sharpness of a visual diagram. The diagrams developed during this study and the formula for the transfer of media characteristics presented in the last chapter should therefore be understood as nothing more – and nothing less – than streamlined instruments to assist us in understanding and analyzing media transformation.

Because illustrating empirically more than fragments of the great web of media transformations is not possible, we settle for examples. I selected three short films by Jan Švankmajer that I use to demonstrate a few possible ways to apply the model presented in the last chapter. The advantage of analyzing a few relatively similar media products, rather than very dissimilar ones, is that a comparison made within a fairly limited frame makes detecting vital differences and nuances easier. Up to this point in this study, I hope to have already illustrated more rough and blatant media transformation differences.

Nevertheless, the three films are selected to demonstrate dissimilar types of media characteristics transfers; whereas the target media prod-ucts all share the same basic media traits and belong to the same qualified medium, the source media are fundamentally different regarding their modality modes. However, they are all artistic media qualified as music, literature, and painting. The choice of analytical examples is clearly colored by my scholarly training, and I wish to restate that artistic media are multifaceted creations that differ in degree rather than in kind from other media; given their complexity, they are well suited as examples for discerning general media transformation principles.

Jan Švankmajer (born 1934) is a Czech artist and filmmaker, primarily famous for his animated films that are generally created using a stop-motion technique that moves or changes objects in small steps between individually photographed frames to create the effect of objects trans-forming and being in motion as the frames are continuously displayed. Many of Švankmajer’s films may be said to focus on themes such as change, incorporation, and eating.1 Well worthy of in-depth attention, I must settle with brief, schematic, and rather strict analyses of the three selected films: J. S. Bach – Fantasy in G Minor (1965), Jabberwocky (1971), and Dimensions of Dialogue (1982) (Švankmajer, 2007). Because analysis


DOI: 10.1057/9781137474254.0008

cannot really be separated from interpretation (even the strictest analysis of objects and phenomena is based on conscious or unconscious inter-pretive maneuvers on fundamental levels of perceptions of the objects and phenomena in question), I do not hesitate to actively engage in rudimentary interpretation as well, although I refrain from unfolding the interpretations as soon as the most important analytical claims concerning media transformation have been made.

J. S. Bach – Fantasy in G Minor

The first media product to be scrutinized is Švankmajer’s J. S. Bach – Fantasy in G Minor, a black and white film from 1965, nine minutes long and originally titled J. S. Bach – Fantasia g moll. I demonstrate how the formula ‘Compound media characteristics are Transferred from a source Medium to a target Medium’, or ‘C are T from M1 to M2’, may be applied to this particular medium. I first generally describe M2, the target medium (Švankmajer’s film). In doing so, I suggest a division of the piece into seven sections. Such a division and a rather detailed overview of the parts of the film facilitate the forthcoming analysis.

In part 1 (0:00–1:45), the credits are initially displayed. More impor-tantly, we see a man wearing a coat and a hat walking toward and enter-ing an old stone building. He looks around a little and then ascends some stairs. Finally, he enters a room with an organ and with apples scattered in various places; he takes off his clothes, sits by the organ, puts an apple into his mouth, and raises his hands over the keyboard. In this opening part of the film, the point of view constantly shifts as the man is seen from different angles (mostly from behind); sometimes he is at a distance and sometimes the viewer is very close to parts of his body. The scenes also show close-ups of mosaic floors and other details in the house. Part 1 consists of live-action film and virtually the only things to be heard are the quiet but distinct sounds from the man’s steps and doors being opened and closed.

In part 2 (1:46–3:02), the music begins. We hear forceful and energetic organ music composed by Johann Sebastian Bach. From now on, the organist is no longer seen; instead, the field of vision is filled with close-ups of details of walls and locked doors. Rapid shifts occur between different visual frames and after a while the field of vision slides along some marks on a wall. Part 3 (3:03–3:56) is distinguished by a shift

Three Analyses

DOI: 10.1057/9781137474254.0008

in the character of the music. A calm melody in the upper register is introduced and varied. Furthermore, this part consists entirely of stop-motion animation, which up to this point was used only during a couple of seconds in the absolute beginning of part 2. We see holes, stains, and textures in different shapes that grow, shrink, and move on worn plas-tered brick walls (see Figure 5.1).

Part 4 (3:57–5:37) is characterized by the return of livelier music with a much denser sound texture and the disappearance of stop-motion animation. Holes, metal bars, wire netting, metal handles, and locks are displayed in a vigorous tempo. The use of frequent distinct cuts is alternated with horizontally and vertically sliding points of view and an occasional in zoom. Finally, the perceiver briefly glimpses the buildings outside through a little opening in the wall. Part 5 (5:38–6:41), again, is distinguished by a clear shift in the musical mood: the calm melody from part 3 is reintroduced and varied in a similar manner. Once more, stop-motion animation comparable to the visual material in part 3 is presented but now together with static shots of more or less obscure objects on the walls.

Part 6 (6:42–8:21) introduces shots of windows and doors being opened. The organ music slowly grows as it becomes louder and mightier, and the tonal figures strive upwards. Finally, as the intensity of the music increases, the visual point of view gradually slides out of the building; we see the dazzling light of the sky and, through a series of shots with rhythmically shifting length, we are led through the streets outdoors at increasing speed until the movement suddenly stops and we are faced with a close-up of a cracked wall as the movement of the organ

figure 5.1 From Švankmajer, J. S. Bach – Fantasy in G Minor (3:37)


DOI: 10.1057/9781137474254.0008

music halts on two heavy chords. Stop-motion animation is absent in part 6 and returns in the beginning of the concluding part 7 (8:22–9:27), during which holes in the outdoor walls are now growing. After a while, the visual field is distorted as if the walls are rapidly shaken and then the film concludes with a series of frozen shots of closed, broken, or bricked up windows. The music reaches its climax in a mighty stretched chord and then stops.

Identifying the first part (composed ca. 1720) of Bach’s Fantasia and Fugue in G Minor BWV 542 as a source medium (MP1) for Švankmajer’s film (MP2) is easy. Which compound media characteristics (C) are then transferred from MP1 to MP2? Although starting with a full account of the characteristics of MP1 is ideal, clearly no method exists for discern-ing and taking inventory of all of the compound media characteristics of Bach’s Fantasia because individual listeners find partly different traits and, in principle, the list of media characteristics may be extended infi-nitely. However, in practice, circumscribing a more limited amount of traits is possible if one settles with less clear-cut descriptions of them and accepts a share of subjectivity in such descriptions. Nevertheless, some media characteristics are more closely connected to the mediated sensory configurations and, hence, less subjectively construed. In this case, the sound of the Bach Fantasia may be said to represent – that is, to call forth in the mind of the listener – both characteristics that we tend to associate with form (structural relations of all types) and characteristics that we tend to associate with content (more isolated entities). Many of the perceived structural relations are intimately connected to how the music actually sounds and how the musical entities are related to one another in a quantifiable temporal flow, indicating that a strong tendency exists toward intersubjective perceptions and their basic interpretation.

Still, the compound media characteristics of music, as the media characteristics of any qualified medium, are clearly transmedial only to a limited extent. Therefore, some of them are virtually impossible to trans-fer to dissimilar media, whereas others may be transferred only partially. Because this study primarily consists of symbolic signs mediated by a nontemporal visual surface, in contrast to Bach’s Fantasia that primarily consists of iconic signs mediated by temporally evolving sound waves, fully describing (fully representing symbolically) many of the compound media characteristics of the Fantasia is actually impossible. Escaping the fundamental modal differences between written text and other media types cannot be done, which is a crucial problem for all media studies

Three Analyses

DOI: 10.1057/9781137474254.0008

made clearly visible through the example of music in relation to visually mediated language.

However, our minds have certain cross-modal capacities, meaning that even extensive modal differences among media types may be partly bridged. Furthermore, media are generally semiotically multimodal to some extent, which partly eliminates the differences. Whereas music, the source medium in this example, is certainly predominantly iconic, it also generates cognitive import through indices and symbols. Bach’s Fantasia, a piece for the organ, is part of a tradition of Western music associated with the Christian church. Most organs are actually in churches, entail-ing that their sound qualities for many listeners become indices for the interior church space and the religious activities therein. Moreover, this connection has become so habitual that the sound of organ music in many contexts may appear as a symbol for qualities such as solemnity, thoughts of life and death, and the transfiguration of Christ.

Thus, some compound media characteristics of MP1 are the results of signification based on contiguity and convention that the media product has in common with the submedium of organ music in general. However, signification based on similarity – iconicity – must be said to dominate most music. Of course, Bach’s Fantasia also has iconic proper-ties that are shared with other musical pieces; however, these properties are arranged and subsequently interact with one another in a manner that is unique for this very piece. Among the depicted (iconically repre-sented) compound media characteristics that are perhaps primarily associated with content, one may mention traits such as arousal and serenity, expansion and diminution, exploration and reflection, power, ascension, happiness, and even liberation. These musical traits are created on the basis of essential similarities among sensory perception of physical motions, experienced corporeal movements, and mental motions and emotions.

Some compound media characteristics more associated with form – and somehow more specific for this particular musical piece – are subsequently highlighted in the context of transmediation from organ piece to film. However, before discussing how media characteristics are transferred in terms of transmediation and media representation, note that M1 may also be perceived as the qualified medium of music in general. Although I briefly analyze the transfer in terms of ‘C are T from MP1 to MP2’, thinking in terms of ‘C are T from MQ1 to MP2’ is also prolific considering the many universal musical traits of the musical


DOI: 10.1057/9781137474254.0008

Fantasia (such as structures of repetition, variation, and contrast) that are actually transferred to the film.

Now, how is T, the transfer, construed? First, MP1 is represented in the title of MP2. The musical medium’s symbolic title (translated from German) is part of the filmic medium’s symbolic title, which is seen at the beginning of the film: J. S. Bach – Fantasia g moll. This representation is very simple and, clearly, no further complex visual and symbolic represen-tations of MP1 are seen (MP2 contains virtually no written language at all, apart from the credits). However, one may ask whether Švankmajer’s film represents Bach’s Fantasia in a complex manner with the aid of auditory and iconic signs. From the point of view of the complete film, the target medium does in fact extensively present the source medium to the mind of the perceiver to such an extent that it borders on completeness, and the transfer involves different types of qualified media. Thus, arguably, the visual symbolic signs initiate the representation of the musical Fantasia in a simple way and the auditory iconic signs then fully finalize the represen-tation of another medium, and we actually hear the entire musical piece.

One may feel slight resistance toward such reasoning. The complex, effectively complete representation of the music is actually performed by sounding sensory configurations in MP2 that are virtually the same as those in MP1; therefore, why say that the musical Fantasia is represented when it is essentially part of the film? However, the sounding sensory configurations of the two media products are not necessarily the same. If MP1 is taken as a live performance of Bach’s Fantasia, the sound in Švankmajer’s film differs slightly but significantly, given that the full spatiality of the source sound is clearly lost in the target sound, which is to say that a small modal difference exists (most sound recordings should actually be understood as iconic and indexical representations that differ slightly from their source media products). In this case, the film may be understood to comprise a very complex media representation involv-ing only little transformation. However, if MP1 is taken as a recording of the Fantasia, I argue that MP2 contains a new mediation but not a transmediation of the music. Then, applying a synchronic rather than a diachronic perspective on the sounding music in the film and simply stating that Bach’s Fantasia is an integrated auditory part of the film is more rewarding. These different perspectives are by no way mutually exclusive. The two perspectives – the film both represents a live perform-ance of the Fantasia in a complex manner and integrates the recorded sound of the Fantasia – are complementary rather than contradictory.

Three Analyses

DOI: 10.1057/9781137474254.0008

However, media transformation in Švankmajer’s film is not limited to media representation; complex transmediation also occurs from the auditory MP1, conceived as a live performance of Bach’s Fantasia, to the visual part of MP2, the audiovisual film. Because recorded music is actu-ally an integrated part of the film (depicting and deicting live music), this transmediation from live music to visual film is simultaneously perceived as an interaction between the cohesive auditory and visual parts of the film; this interaction might even be conceived as a type of counterpoint between the auditory and the visual. I highlight some of the most prominent visual iconic features of the film to illustrate the transmediation of compound media characteristics.

Compared with the film’s almost complete auditory representation of a live performance of Bach’s Fantasia, which entails an equally compre-hensive transmediation of compound media characteristics, the trans-mediation of media characteristics from the auditory music to the visual part of the film is very limited and partly elusive, particularly concerning aspects of content (represented emotions, feelings, and sensations). One might even say that whether or not the quality of solemnity – suggested as a symbolic characteristic of much Western organ music – is viewed as being transmediated to the visual part of the film is an open ques-tion. However, given that the visual is part of the overall audiovisual appearance of the film, one might instead observe that a certain tension exists between the visual iconic representations and the solemnity of the sounding music; for example, we do not see details of magnificent gothic architecture or Christian attributes in a cathedral but worn and broken surfaces and objects. If the quality of solemnity is transmediated from the auditory to the visual, it is done not in a straightforward manner (from the gravity of church music to the gravity of church architecture and decor) but rather with a twist, wherein the solemnity of the organ piece is transferred to the trivial and rundown details visible in the film. Similarly, the symbolic connection between organ music and Christian ideas, such as the transfiguration of Christ, may be said to have equiva-lence in the visual iconic part of the film: the transfiguration of Christ is transformed into the transfiguration of material objects.

Other compound media characteristics of the source medium that may be viewed as being transmediated to the visual part of the target medium are the contrast between arousal and serenity (reflected in marked shifts of differing visual rhythms created by, for instance, cuts and movements of the visual field), expansion and diminution (seen


DOI: 10.1057/9781137474254.0008

in the transforming holes and textures depicted in the film), ascension (the visual field repeatedly moves upwards in part 4), and happiness and liberation (the film depicts doors and windows that are opened, followed by an accelerated movement out of the building and a glimpse of the bright light of the sky in part 6).

Furthermore, a rather strong resemblance exists between the most general formal traits of MP1 and MP2. As was noticed in my intro-ductory description of the film, the structure created by the contrasts among parts of the film largely corresponds to the structure created by the interrelations of the sections of the music; these contrasts involve easily perceptible features, such as pitch, brightness, pace of notes, speed of visual cuts, sound texture, and observable qualities of stop-motion animation. Although the visual structure of the film certainly cannot be equalized with the auditory structure of the music (complete resemblance is impossible), note that most of the cuts in the film are synchronized with the development of the music. In part 2, heavy chords are matched with in zooms; furthermore, a long pedal point together with a winding melodic figure correspond to the field of vision sliding along white marks on a wall that resemble a sine curve or even, remotely, a score. In part 3, the pulse of the music corresponds to the pulse of the growth of visual forms. Additionally, subsequent parts of the film have similar correspondences between the auditory and the visual that may be construed as transmediation.

Although much more could be said about the relation between MP1 and MP2, I stop here and reflect on the thickness, direction, and exten-sion of the T arrow. The thickness of the T arrow should be considered in two steps.

From the point of view of the complete media transformation, first, the arrow is clearly very thick: Švankmajer’s film is mediated in such a way that several modality modes are actually added (compared with the mediation of Bach’s Fantasia), allowing us to see iconic, indexical, and symbolic signs on a two-dimensional surface. The film’s only restriction is that the full spatiality of a live performance of the music is missing. However, overall, a strong transfer of media characteristics based on modal similarities occurs from MP1 to MP2. Furthermore, consider-ing that the Fantasia is actually represented in its entirety, the media transformation is both complex and almost complete. Nevertheless, the media representation and transmediation involve significant trans-formation with all visually mediated characteristics added in the target

Three Analyses

DOI: 10.1057/9781137474254.0008

medium, transforming it into something clearly different from the source medium.

From the point of view of the segment of media transformation that has the visual part of the film as a target, second, the arrow is much thinner; although many structural traits can and have been transferred from the auditory to the visual, our mind’s limited cross-modal capaci-ties prevent us from perceiving and construing more than reminiscences of the auditory media content characteristics in the visual part of the film. This statement is not normative, of course; although the accurate transfer of media characteristics is a problem in much communication, it may be a creative asset in artistic media.

Regarding the direction of the T arrow, obviously, this entire analysis is based on the assumption that Švankmajer’s film may be understood as a target medium with Bach’s Fantasia as a source medium. Of course, although this perspective may be challenged, it is anchored in some clearly displayed characteristics of the film. However, equally clear is that Bach’s Fantasia may be said to import characteristics from Švankmajer’s film. In the film, they are integrated and, hence, parts of an indivisible whole; however, for a person who is influenced by the film, the interpreted meaning of the piece of music in itself may change. Indeed, the author of this study may have had his description of the musical Fantasia colored by the film from the beginning without knowing it.

Regarding the extension of the T arrow, we noted that the qualified medium of music in general might be understood as a source medium for Švankmajer’s film. In addition, Bach’s Fantasia is more than probably construed as a source medium for several other target media products.

Jabberwocky

Our next example is Švankmajer’s Jabberwocky from 1971 (original title: Žvahlav aneb šatičky Slamĕného Huberta). I view this thirteen-minute color film, dominated by stop-motion animation, as a target medium (M2) to be seen in relation to a different kind of source medium: Lewis Carroll’s poem with the same name (M1). Similar to J. S. Bach – Fantasy in G Minor, Jabberwocky has a rather firm structure that I initially describe in an approximate manner. Again, dividing the film into sections to facilitate the analysis is convenient.


DOI: 10.1057/9781137474254.0008

In part 1 (0:00–0:32), which displays the credits, one also sees buttocks being spanked, the rhythmic sound of which is integrated with music composed by Zdenĕk Liška. In this part and throughout the film, the music is primarily dominated by vocalizing song. Finally, one hears the voice of a little girl saying ‘Jabberwocky’. The music stops and part 2 (0:33–2:11) begins. The girl reads Carroll’s poem. Only her voice is heard; she is never seen as she reads. Instead, the viewer perceives a cupboard moving quickly among trees and eventually entering a room with medal-lion wallpapers full of old-fashioned toys, some jars, and a portrait of a bearded man on the wall. Some of the toys move around and one sees glimpses of a little girl. Neat children’s clothes arise from a washbasin in the cupboard.

When we enter part 3 (2:12–4:49), the poem is finished and the room becomes emptier. A few seconds earlier, the music returns and becomes increasingly playful as new motifs enter the visual field: one sees dolls, the empty but upright clothes merrily moving around and even flying, trees growing in the room, and apples ripening, falling, and rotting. Finally, picture puzzle cubes are built into a little wall and the music stops. Part 4 (4:50–6:59) starts with silence. The surface of the puzzle cube wall is transformed into a two-dimensional labyrinth in which a pencil line attempts to find its way. A black cat destroys the puzzle (and, hence, the labyrinth) and music begins. One sees dolls coming out of each other’s interiors, and then being grinded and ironed to the sound of merry tunes; the larger dolls cook and eat the small ones in a tidy and polished manner. A new puzzle cube wall is built and the music stops.

Parts 5–7 repeat the overall structure of part 4. They begin with silence and a line attempting to find the right way through the labyrinth on the little wall that is soon destroyed by the black cat, and they end with puzzle cubes forming a new wall and new pictures. However, the other events are varied. In part 5 (7:00–9:11), one sees tin soldiers coming out of the empty clothes and marching around, and toy castles being built and transformed – all to the sound of childish military music. In part 6 (9:12–10:42), a clasp knife with a handle sculptured as a woman dances to the sound of a jerky waltz until she is killed by her own blade and ends up in a pool of blood (see Figure 5.2). In part 7 (10:43–12:37), papers from school notebooks are folded by themselves into ships, airplanes, and other objects; the portrait on the wall sticks his tongue out and spits out dominoes; a doll dances in a chair and, in the end, a swarm of paper planes fly out through the open window to the sound of jubilant song.

Three Analyses

DOI: 10.1057/9781137474254.0008

Part 8 (12:38–13:20) starts with silence and the line attempting to find its way through the labyrinth, although this time no cat is destroying the puzzle. Furthermore, the line now actually finds the right way out. Happy music begins and the line escapes to the wall and destroys the portrait of the bearded man; finally, one sees the black cat in a cage in the cupboard together with the children’s clothes in a heap and a black suit on a hanger.

Carroll’s poem ‘Jabberwocky’, a specific media product, is without a doubt a source medium to Švankmajer’s film, Jabberwocky. This literary piece was originally part of the author’s novel, Through the Looking-Glass, and What Alice Found There (1872: 21–4) and is generally classified as nonsense verse:

’Twas brillig, and the slithy toves Did gyre and gimble in the wade;All mimsy were the borogoves, And the mome raths outgrabe.

“Beware the Jabberwock, my son! The jaws that bite, the claws that catch!

figure 5.2 From Švankmajer, Jabberwocky (10:20)


DOI: 10.1057/9781137474254.0008

Beware the Jubjub bird, and shun The frumious Bandersnatch!”

He took his vorpal sword in hand: Long time the manxome foe he sought –So rested he by the Tumtum tree, And stood awhile in thought.

And as in uffish thought he stood, The Jabberwock, with eyes of flame,Came whiffling through the tulgey wood, And burbled as it came!

One, two! One, two! And through and through The vorpal blade went snicker-snack!He left it dead, and with its head He went galumphing back.

“And hast thou slain the Jabberwock? Come to my arms, my beamish boy!O frabjous day! Callooh! Callay!” He chortled in his joy.

’Twas brillig, and the slithy toves Did gyre and gimble in the wabe;All mimsy were the borogoves, And the mome raths outgrabe.

Of course, this poem is far from true nonsense. Although many of the words escape simple explanation and refer to objects that the reader is at liberty to create herself, the poem displays a wide range of compound media characteristics called forth by the symbolic signs and their arrangement on the page. From a formal point of view, the poem clearly shows a symmetric structure with its seven stanzas all having four lines arranged in a simple pattern, and the first and the last stanza are identical (with the exception of one letter, probably a misprint). When reading the poem, an iambic rhythm is clearly and consistently discerned together with the sound of fairly regular end rhymes and internal rhymes, such as alliterations and assonances.

These formal media characteristics interact with the content created in the mind of the perceiver when the visual symbolic signs of the poem are decoded. The words clearly describe a few characters, their interactions, and a series of events, which is to say that the poem forms a narration. Additionally, this narrative content can, in its turn, be understood in

Three Analyses

DOI: 10.1057/9781137474254.0008

terms of form and content. The form of the tale may be summed up as ‘a boy leaves home to hunt a beast, kills it and returns with its head’, or in even more abstract terms, ‘a protagonist goes away, completes her or his mission and then returns.’ Indeed, this narrative form is a compound media characteristic, as are the particularities of the content that together build the overall form: a father and his speeches, a son and his behavior at different stages of the story, the different creatures and their traits (the toves, the borogoves, the raths, the Jabberwock, the Jubjub bird, and the Bandersnatch), the tree and other natural objects that are described, the sword, the battle, the decapitation, and so forth.

I argue that other compound media characteristics are themes and general traits of the narration, such as adventurousness, heroic deed, excessive violence, childish play, and father–son relations. The narration and the manner in which it is shaped are fanciful, grotesque, or even surreal. A very prominent characteristic of the poem on the level of language form is its neologisms.

More compound media characteristics could certainly be added and other readers would provide a partly different set of characteristics. However, instead of lengthening the list, we briefly investigate the char-acteristics that may be said to be transferred from the poem (M1) to the film (M2), and how.

First, clearly, Švankmajer’s film Jabberwocky symbolically represents Carroll’s printed poem of the same name in a simple manner. The film’s visual title refers to the poem’s visual title and, hence, draws attention to its appearance in an elementary manner – provided that one is familiar with the poem, of course. The poem is also read in the film. Yet, as I construe the transfer of media characteristics, the film does not represent the written poem in a complex manner because it does not really make present the outward appearance of the visual poem in a multifaceted way. That the auditory words in the film symbolically represent the visual words of the printed poem could be argued without a doubt; however, taking the written words as symbols for sounding words rather than the other way around is a much more established convention. Instead, I find it more straightforward to say that the source medium is intricately transmediated by the target medium. This is done in two ways.

First, in part 2 of the film, the reading of the poem must be understood as a transmediation from a visual source medium to the auditory part of an audiovisual target medium. Almost the same compound media char-acteristics are again represented by another technical medium when we


DOI: 10.1057/9781137474254.0008

hear the words of the poem in the air around us rather than see them on a flat surface. Transmediations such as these – between different sensory modes but within the same semiotic mode of the symbolic – are relatively exact given the extremely well-developed conventions of verbal language that regulate, among other things, how the looks and combinations of visual signs are rendered as auditory signs. These conventions frequently enable the reading and pronouncing of words that are unfamiliar or even newly invented, as in Carroll’s ‘Jabberwocky’. However, even though the reading of a visual symbolic text may transmediate its characteristic to such a high degree that there may appear to be a symbiotic relation between source and target, all readings are transmediations and involve significant transformation. Visual and static surfaces dominated by symbolic signs also produce cognitive import through visual iconic signs for which finding auditory equivalences may be difficult or impossible. Furthermore, a neutral reading of a verbal text is impossible and the reader may certainly choose to use her voice in a manner that signifi-cantly contributes to its meaning. In the film Jabberwocky, a girl with a vibrant voice reads the poem. She energetically reads the stanza describ-ing the killing of Jabberwock, whereas the father’s ensuing praise sounds a bit theatrical, as if to establish a certain distance from the deed.

Second, the visual and primarily symbolic source medium is further-more transmediated to the audiovisual target medium in its entirety. Apart from reading the poem, the film is clearly dominated by iconic-ity, both in what is seen and what is heard, which is to say that this transmediation is very much a transfer from the visual poem’s symbolic signs to the audiovisual film’s iconic signs. Which C are then T from M1 to M2 in this manner?

Initially, note that several characteristics of the poem are not transferred to the film; for example, the traits of the stanzas and their interrelations, the iambic rhythm, and the rhymes have no equivalences in the film apart from the very reading of the poem. However, both media products have structures based on apparent repetitions of elements. Given the firm connection between the poem and the film, other compound media characteristics may be seen as being vaguely transferred. For example, the poem describes what seems to be a row of odd animals (as the Jubjub bird and the Bandersnatch), and the film depicts several peculiar beings (as the moving clothes and the eating dolls). A loose connection also exists between the Tumtum tree and the woods in the poem and the trees seen in the film: outdoors in part 2 and growing in the room in part

Three Analyses

DOI: 10.1057/9781137474254.0008

3. Another detail to possibly note is that the Jabberwock is said to burble, which may be associated with the excessively bubbling water in which the small dolls are boiled in part 4 of the film.

More generally, the poem’s air of adventurousness finds its equivalence in the returning labyrinth of the film. The childish play displayed in the poem is also part of the film, which immediately shows the (presumably naughty) child being spanked in part 1 and glimpses of a little girl in part 2; furthermore, playful music is heard in part 3, children’s clothes and toys are visible in most parts of the movie, and so forth. Additionally, the fanciful, grotesque, and surreal characteristics of the poem are approxi-mately transmediated to the film; for example, trees grow in the room in part 3; dolls emerge out of other dolls and are then grinded, ironed, and eaten to cheerful music in part 4; and the portrait of the bearded man sticks his tongue out and spits dominoes in part 7. Compound media characteristics such as these (the fanciful, the grotesque, and the surreal) are transmedial to a significant extent as both visual and auditory descriptions and visual depictions represent detailed constel-lations of characters, objects, and events that are arranged in numerous inventive ways. However, they are certainly not completely transmedial. A characteristic such as the poem’s neologisms is undoubtedly trans-medially restricted by modal borders; because words are most of the time primarily formed on conventional ground, the border between the symbolic and other semiotic modes may be expected to be difficult to cross. However, neologisms may be formed in numerous ways, indicat-ing that different types of neologisms are more or less transmedial. The new words in the poem that are formed from combinations of parts of other words might be said to find equivalences in the sections of the film in which new combinations of puzzle cubes simultaneously display parts of different visual motifs.

Yet, I argue that the most distinct transmediation involves the narrative structure and some of its most prominent elements. The source medium tells the story about a boy who leaves home to hunt a dangerous brute, kills and decapitates the animal, and returns with its head. A significant resemblance exists between this narration and what happens in the target medium. The black cat that repeatedly destroys the puzzle corresponds to the Jabberwock; they have both ‘jaws that bite’, ‘claws that catch’, and ‘eyes of flame’. However, the cat is not killed but is caught in a cage in the cupboard in part 8 of the film. One cannot tell for certain who put the cat there, but its rowdy behavior is efficiently stopped. Furthermore, the


DOI: 10.1057/9781137474254.0008

father’s rejoicing words late in the poem correspond to the scene in part 7 of the film in which paper planes fly out through the window to the sound of triumphant song.

Taking into account this resemblance of narrative form, narrative content from the poem such as heroic deed, excessive violence, and death may also be traced in the film. In part 3, rotting apples remind the viewer of death; in part 4, one sees dolls being grinded, ironed, and cannibalized; in part 5, tin soldiers and military music represent battle and slaughter. Although no sword or decapitation is depicted in the film, the clasp knife in part 6 is killed by itself as its blade is folded in and it ends up in a pool of blood. Additionally, the father–son relation in the poem may be traced in the film, although in a less pronounced manner. If one wishes, the portrait on the wall may be viewed as the father and the moving empty clothes as the son. In part 3, the clothes ride on a rocking horse and disappear through the cupboard, which may remind the viewer of a fairy tale hero going off to new adventures. In part 8, the line in the labyrinth finds its way out, escapes, and destroys the portrait of the bearded man, which suggests that the film is a story beginning with being spanked and ending with a revolt against the authoritative father – a somewhat different story than the one told in the poem.

Overall, I suggest that the T arrow of this media transformation is indeed thick. Carroll’s poem is unambiguously represented in the title of Švankmajer’s film and is furthermore transmediated in two ways. The first transmediation – the reading of the printed poem in part 2 of the film – involves several modal changes: from flat solid surface to vibrating air, from seeing to hearing, and from static two-dimensional spatiality to temporality. However, the highly developed conventions of verbal language allow for a smooth transference of the visual symbolic signs to auditory symbolic signs. Whereas this transference is certainly not seamless, it is comprehensive enough to bring about a new repre-sentation of perhaps most of the compound media characteristics of the source medium. The second transmediation, which includes the entire film, involves more modal additions than restrictions. The page and the screen are both flat, solid surfaces; however, the film includes sound. Similarly, symbolic signs dominate the poem, whereas the film further-more displays a plenitude of sensory configurations that are understood as iconic signs. However, the nontemporality of the printed poem is replaced by the film’s temporality, even though this temporality is not a major restriction for transfer of media characteristics. Consequently,

Three Analyses

DOI: 10.1057/9781137474254.0008

although not done, transferring virtually all of the printed poem’s char-acteristics to the film is fully possible simply by displaying the complete visual poem on the screen. Instead, the second transmediation is predominantly a transfer of media characteristics from the visual poem’s symbolic signs to the audiovisual film’s iconic signs. Clearly, this transfer involves a higher degree of transformation than the first transmediation (and would have done so even if the filmmaker attempted to stay as close as possible to the source).

At least in this example, then, the crossing of borders between semi-otic modes leads to more profound transformations than the crossing of material, sensorial, and spatiotemporal borders. Nevertheless, note that rather strong connections exist between symbolic representation in general and visual iconic representation; as language is developed in close association with the perception of and interaction with the world around us, no abyss exists between meaning formed by advanced symbolic systems and comprehensive iconic information (see Kwiatkowska, 2013: 13–29). Hence, we have cognitive abilities that support developed trans-fers also between symbols and icons.

The T arrow is thick considering also the overall complexity of the various transfers involved and the relative completeness of, in particular, the first transmediation, which is the reading of the poem.

Regarding the direction of the T arrow, I conclude similar to the last example. My analysis was based on the assumption that Švankmajer’s film may be understood as a target medium with Carroll’s poem as the source medium, a postulation anchored in some evidently presented characteristics of the film. Nevertheless, seeing the film enables the poem to be interpreted in new ways.

However, much more could be added to this analysis with respect to the extension of the T arrow. Evidently, Carroll’s poem ‘Jabberwocky’ is the source of several other target media; however, we must also note that further relevant sources to Švankmajer’s film Jabberwocky may be traced. Whereas an extended analysis probably must include the entire novel, Through the Looking-Glass, including its illustrations, it also needs to consider another illustrated literary work that is less known to an interna-tional audience but that is indicated in the original title of Švankmajer’s film, Žvahlav aneb šatičky Slamĕného Huberta. ‘Žvahlav’ is the title of the first Czech translation of ‘Jabberwocky’ (the word has connotations to talking nonsense and gives the impression of being the name of a particular animal species). ‘Aneb’ means ‘or’ and ‘šatičky Slaměného Huberta’ could


DOI: 10.1057/9781137474254.0008

be translated as ‘the tiny clothes of Straw Hubert’, which refers to a fantas-tic fiction for children written by Czech surrealist Vítězslav Nezval in 1936, Anička skřítek a slaměný Hubert (Little Annie and Straw Hubert). The tale was originally illustrated by the surrealist painter Toyen (Marie Čermínová) and later by Jiří Trnka, an illustrator and animated filmmaker.2 Because Švankmajer’s film depicts children’s clothes moving around and mentions ‘the tiny clothes of Straw Hubert’ in the title, also analyzing Anička skřítek a slaměný Hubert as a source medium is likely to be rewarding.

Dimensions of Dialogue

Our third and last example chosen to illustrate the principles of media transformation presented in this study, and more specifically the formula ‘C are T from M1 to M2’, is Jan Švankmajer’s Dimensions of Dialogue from 1982 (its original title, Možnosti dialogu, may also be translated as ‘possibilities of dialogue’). This eleven-minute color film is stop-motion animated with the aid of a variety of materials and includes music by Jan Klusák. The filmmaker divided the film into three distinct parts, not including the brief introduction (0:00–0:28) that displays the credits to the sound of a distant buzz of voices.

Part 1 (00:29–5:21) is titled ‘factual dialogue’. Initially, one sees two composite heads in profile approaching each other. One of them, made of kitchenware, opens its mouth widely and swallows the other one, which is made of food – mostly vegetables (see Figure 5.3) – resulting in a massacre of unconventional ways of peeling, crushing, grating, slicing, and smashing food. Nonetheless, the complete mess is reorganized into a new composite head made up of food fragments, kitchenware, and other items. This new head throws up material that forms yet another head made of processed food fragments; in doing so, it is transformed into its original shape, so to speak, made of only kitchenware. The kitch-enware head now turns around and approaches a completely new head made of books, papers, and tools for writing, drawing, and painting. The book head swallows the kitchenware head, which leads to a new fanci-ful massacre. Finally, a new head is formed from the chaotic mess that throws up kitchenware fragments to form a new head made of processed kitchenware, whereas the vomiting head itself regains its original form consisting of materials for writing and drawing. The book head then approaches the head made of food fragments, which is followed by

Three Analyses

DOI: 10.1057/9781137474254.0008

several variations of butcheries and bizarre transformations displayed in a strict filmic composition. The heads eat and throw up one another with increasing speed and are grinded into ever finer particles until two clay heads finally emerge. One of the clay heads eats the other, leading to a seemingly endless continuation of vomiting up of new clay heads. The visual events in this section of the film are forcefully accompanied by music from wind instruments integrated with mechanical and organic sounds from the massacres.

In part 2 (5:22–8:06), ‘passionate dialogue’, the visual events are comple-mented by only a few intradiegetic auditory effects but are continually unified with music played by a few string instruments that closely follows the moods and movements of two interacting clay figures. A woman and a man sit by a table, are attracted to one another, and start to kiss; soon they melt together and become literally inseparable as they start to caress each other’s bodies. The two figures now form a constantly chang-ing, amorphous common body with parts of their physiques becoming discernible for short periods. After this sexual merge that mentally and materially unifies the two figures, they are gradually separated and return

figure 5.3 From Švankmajer, Dimensions of Dialogue (00:46)


DOI: 10.1057/9781137474254.0008

to the chairs by the table. However, a little lump of clay is left between them. As the lump approaches them, they reject it, become angry, and start throwing the lump at each other. Beginning with each other’s faces, they tear each other apart (again literally) until nothing much remains.

Part 3 (8:07–11:18) is called ‘exhausting dialogue’. Clay heads of two similar men facing each other are formed on a table and items start coming out of their mouths. Initially, the items interact smoothly: a toothbrush from one man is followed by toothpaste from the other, and the paste is nicely put on the brush; bread is followed by butter that is spread onto the bread; a shoe is completed by a shoelace and a pencil is assisted by a pencil sharpener. After that, the two heads change posi-tions: now bread is messed up with toothpaste, a toothbrush is ill-treated by a pencil sharpener, a shoe is greased with butter, a pencil is tied with shoelace, and so on, in a row of variations. Again, the two heads change positions, putting shoe against shoe, shoelace against shoelace, and so forth, at increased speed and degree of destruction until both heads are exhausted and demolished. These scenes occur to the sound of discrete railroad noises and comical sound effects created by instruments and other objects, many of them squeaky or piercing.

Švankmajer’s Dimensions of Dialogue is understood to be the target medium (M2) to which compound media characteristics (C) are trans-ferred (T) from the source medium (M1). The identified source medium is neither strictly a media product nor a recognized qualified medium; instead, it is a group of media products, namely a range of late 16th century paintings by Giuseppe Arcimboldo. These paintings are similarly composed and are well-known and imitated to such an extent that one might be tempted to say that they formed a genre or submedium; thus, the media transformation analyzed can be understood as a transfer of characteristics from something in between a specific media product and a qualified medium to a particular media product.

The main characteristic trait of these paintings is that they form figures that look like human heads and parts of bodies out of discrete objects. Thus, the most prominent formal compound media characteristic is the principle of forming composite bodies. The content of the overall corpo-real form consists of different types of items in different paintings. In most cases, the parts are organic pieces such as fruits, vegetables, berries, ears of corn, leaves, flowers, and decayed trees; however, Arcimboldo sometimes created human heads out of animals, naked infants, and, in a few paintings, books, kitchen utensils, and other artifacts. All of these

Three Analyses

DOI: 10.1057/9781137474254.0008

depicted entities are compound media characteristics and, although they are the content of the complete corporeal forms, they may also be understood in terms of form, of course: their visually displayed forms are made of parts having certain united traits.

The general principle of forming composite bodies borders other notions, such as the grotesque, understood as an amalgamation of living and dead or human and nonhuman; hence, the grotesque may be said to be a related media characteristic of Arcimboldo’s paintings. One might also argue that the wide-ranging principle of composition as such, the notion of shaping an overall form out of smaller forms, is a prominent characteristic of the paintings; this notion is related to the notion of transformation, which is another conspicuous characteristic of the source media products: most of the figures are formed using food, suggesting concepts such as eating and digesting food – processes that certainly include palpable transformation of substances. Furthermore, several of Arcimboldo’s paintings are very realistic depictions of bowls with fruit, vegetables, or other food that are transformed into depictions of heads (with the bowls transfigured to hats) when turned upside down.

How can these compound media characteristics of the source media products be said to be transferred to the target medium? I advocate the view that Švankmajer’s film holds no media representation. Arcimboldo’s paintings are not depicted, deicted, or described by the film, not even in a simple manner. Instead, they are transmediated in various ways in the three distinct parts of the film.

For viewers familiar with Arcimboldo’s work, part 1 of Dimensions of Dialogue immediately displays similarities between the paintings and the film. Several media characteristics of the sources are represented again by a new type of sensory configuration (not visual and static but audio-visual and temporal signs) mediated by other types of technical media (not a canvas but a screen and vibrating air). Clearly, Arcimboldo’s prin-ciple of forming composite bodies is also at work in Švankmajer’s film: although far from exact copies, the figures in the film – in particular the first ones – are composed of distinct objects that together form the shape of heads and shoulders in profile in a manner that very much echoes the overall appearance of the paintings. By the same token, the film, as do the paintings, depicts constellations and amalgamations of objects belonging to contrasting categories, such as living and dead and human and nonhuman, resulting in grotesque effects. Furthermore, most of the types of objects forming the heads in the film are also found in the


DOI: 10.1057/9781137474254.0008

16th century paintings: fruits, vegetables, cooked animals, books, and kitchenware. The film also represents transformation processes. Indeed, part 1 may be said to be about transformation from the beginning to the end because it represents human-like but nevertheless uncanny figures that eat and digest a variety of food and, simultaneously, eat and throw up one another; these processes are depicted by both what one sees and what one hears in the film. Again, this theme is certainly present in Arcimboldo’s paintings that contain both bowls of food and human heads, although Švankmajer’s film takes this concept to extremes.

In light of the strong presence of compound media characteristics from Arcimboldo’s paintings in part 1, part 2 of the film may also be understood to transmediate characteristics from the same source media – although to a limited extent. Here, too, one finds the principle of forming composite bodies, not in terms of combining different types of objects into a human body, but in terms of combining bodies with each other into a merged body – a slightly grotesque event – if one wishes. In any case, part 2 thoroughly represents mental and corporeal transforma-tions that connect it loosely to the source media.

I claim that part 3 of the film transmediates more media characteris-tics from the source media than part 2 does, even though part 3 needs the background of part 1 to truly establish the relation to Arcimboldo’s visual world. Although part 3 is not about forming composite bodies, it is without a doubt about composing objects that work or do not work together in meaningful ways, or that work together in confusing ways. The clashes resulting from the joining of objects that do not interact properly are comical and bizarre but perhaps not grotesque; however, the notion of objects coming out of people’s mouths the way they do in the film is certainly grotesque because it is explicitly depicted by the film, even though one is likely to construe the objects as utterances from the two men who clash in an increasingly fundamental manner in an ‘exhausting dialogue’. Furthermore, part 3 of Dimensions of Dialogue represents transformational events in which objects (or words) either cooperate or are unproductively mixed, distorted, or wrecked. Because part 3 represents mouths and their actions, it also indirectly creates asso-ciations to eating and digesting food.

Considering the thickness of the T arrow, we first observe that certain modal changes affect the transfer of C from M1 to M2: the paintings are visual and nontemporal and the film is audiovisual and temporal. Therefore, a sensorial mode is added and a spatiotemporal mode is

Three Analyses

DOI: 10.1057/9781137474254.0008

exchanged; however, because a temporally changing visual surface is able to display the same sensory configurations as a static visual surface (but not the other way around: moving formations on a screen can be frozen but static formations on a canvas cannot be set into motion), the media transformation from painting to film does not inflict any necessary restrictions on the transfer of media characteristics. Achieving extremely detailed transfers of almost all media characteristics of all of Arcimboldo’s paintings to film is possible but is not done in Dimensions of Dialogue; the possibilities of the added sound are also not utilized in a remarkable manner. Instead, the change from nontemporality to temporality is used to add further aspects to forming composite bodies, grotesque amalga-mations of objects, and processes of transformation such as eating. The source media are not transformed by necessity but by choice. I propose that the T arrow is thick in consideration of part 1 of the film, but is very thin in relation to parts 2 and 3. The transfer from paintings to part 1 of Švankmajer’s film is strong, as evidenced by the numerous visual, two-di-mensional, and iconic characteristics of the paintings that are kept intact in this section of the film. The transfer is complex, as evidenced by the intricate characteristics such as composition of merged bodies, grotesque arrangements, and transformations that are transferred on many levels. The transfer is relatively complete, as evidenced in the numerous features from a broad range of paintings that may be traced in the target medium. The other parts of the film do not display all these characteristics.

Not much more needs to be added to this analysis, which was to be as brief as possible, except for the following now obvious points: the T arrow might be reversed in the sense that the film Dimensions of Dialogue may well affect our way of grasping Arcimboldo’s paintings. Moreover, further source media for Švankmajer’s film could be identified and point-ing to a row of other target media that transform Arcimboldo’s paintings in various ways would be easy.

Notes

For a general presentation of Švankmajer’s films, see Hames (2008) and Jodoin-Keaton (2011); in particular, the edited volume of Hames also contains several investigations of the three short films analyzed in this study.I am obliged to Alice Jedličková, who provided me with all of the information on the Czech words, titles, works, and artists.

DOI: 10.1057/9781137474254.0009

6Conclusion

Abstract: The final chapter of the treatise offers a concentrated demonstration of how the theoretical notions developed in this study of transfer of media characteristics among different types of media may also be used to analyze transfer of media characteristics among similar media. I schematically map the border zones of intermedial and intramedial transfers of media characteristics. This chapter concludes with final considerations of the research field of media transformation.

Keywords: media transformation; intermedial; intramedial


Conclusion

DOI: 10.1057/9781137474254.0009

In the last chapter, three diverse examples of media transformation were analyzed. Although the target media were of the same type – the qualified medium of animated short film – the varieties in source media enabled the revelation of substantially different ways to transfer compound media characteristics. To nobody’s surprise, I presume, the transfers were noted as ranging from very thin to very thick. In some cases, the source medium was markedly transformed into the target medium, although the filmmaker could have chosen to stay much closer to the source. In other cases, the transformation was sooner forced by the differences among the media modalities. In particular, Bach’s Fantasia clarified that some media characteristics are less transmedial than others, indicating that they depend on certain combinations of modality modes. Again, that media types such as music are not easily transformed into other media types is hardly groundbreaking news. However, I attempted to demonstrate that the notion of media modalities and, more specifically, an emphasis on dissimilar modes of media modalities, might offer ample assistance in understanding and describing why and how transfers of compound media characteristics are possible or not. Because thousands of different media characteristics and thousands of ways to transfer these more or less transmedial characteristics between media with dissimilar modal traits exist, this study highlighted only a fragment of the field. However, the notions that were developed might be helpful for future investigations.

Furthermore, in the discussion of Bach, I noticed that this scholarly work primarily consists of symbolic signs mediated by a nontemporal visual surface, whereas the Fantasia primarily consists of iconic signs mediated by temporally evolving sound waves. My point was that fully describing the compound media characteristics of the Fantasia is unmanageable. A related point to be made is that this entire study and, in particular, the previous chapter is full to the bursting point of simple and complex media representations; this study in itself is an example of what it attempts to analyze and understand: the transfer of media characteristics among media. My descriptions of various transfer types among a plenitude of media types are constantly reduced to written, verbal language with its rich but nevertheless severely limited resources of symbolic signs mediated by a nontemporal visual surface, comple-mented by schematic diagrams – visual, two-dimensional but static icons. These media types only represent the characteristics of unalike media to a limited extent. Hence, the condition for scholarly work such as this study is that we are trapped in a communicative space from which


DOI: 10.1057/9781137474254.0009

we cannot escape. This space is similar to a room with a very good view; although its windows are large, they are all on the same side of the house and getting to the garden or to the roof for an overview of the entire building and its surroundings is not possible. In other words, although an abundance of compound media characteristics exist that are depicted and deicted in various material, sensorial, and spatiotemporal ways, and that may be understood as being transferred in substantial ways, many are simply impossible to adequately describe.

However, Chapter 5 not only presented media representations under-stood in the manner in which the notion was defined in this study (media representing other media of a different kind); it further put on media representations that are not transmedial and, consequently, not media transformations (media representing other media of the same kind). The complete poem ‘Jabberwocky’ was included in the analysis, which is to say that it was virtually fully mediated within the frames of the study (only marginally transmediated if one considers minimal modal differences such as, possibly, the shift from book page to screen – if this treatise is read as an e-book). Furthermore, the poem was commented on and briefly analyzed; in other words, it was represented in a fairly complex manner but by a media type with effectively the same modal traits. Clearly, media transformation borders transfers of media char-acteristics that are not intermedial in the sense that source and target media are the same and not different. In the following section, I sketch a method for approaching this multifaceted border area by building on parts of the conceptual framework developed in this treatise.

The border zones of media transformation

Using the formula ‘Compound media characteristics are Transferred from a source Medium to a target Medium’, or ‘C are T from M1 to M2’, as a point of reference makes obtaining an overview of the border areas of media transformation fairly easy. By simply dropping the criterion that the transfer should be among different media types, the formula captures a much wider range of media phenomena. In this extended area, C must still be understood in the same manner, although compound media char-acteristics that are not the least transmedial may also be part of transfers among similar media types. T may also be understood in very much the same way, with the exception, of course, that the transfer does not

Conclusion

DOI: 10.1057/9781137474254.0009

have an intermedial nature if it does not involve a media transformation. Although M1 and M2 are still construed as source medium and target medium in the extended area of transfers of media characteristics, no intermedial relation exists between the two media if they are understood to be similar rather than different.

Because the notion of medium is multifaceted, this explanation requires greater precision. I submit that in a broad sense, intermedial means rela-tions among different qualified media or among media products belong-ing to different qualified media. However, not a few qualified media are actually based on the same modality modes; consequently, they belong to the same basic media type. Thus, in a narrow sense, intermedial means relations among different basic media – relations among media types based on different modality modes. For instance, the two media types (written) poetry and scholarly article are qualified in very different ways, although they are both typically understood to consist of visual, static, and symbolic signs that are sequentially decoded from a flat surface. Whereas the interrelation between poetry and scholarly article is intermedial in a broad sense, it is not intermedial in a narrow sense.

Media interrelations between similar rather than different qualified media or between media products belonging to the same rather than different qualified media may be termed intramedial.1 Thus, the aim is to concisely put intermedial transfers of media characteristics in relation to intramedial transfers of media characteristics with the aid of some of my elaborated notions. Investigating possible source and target media combinations in terms of basic and qualified media, I discern the follow-ing three types of intermedial and four types of intramedial relations:

. Intermedial relation between two different media products belonging to different qualified media

. Intramedial relation between two different media products belonging to the same qualified medium

. Intermedial relation between a qualified medium and a media product belonging to a different qualified medium

. Intramedial relation between a qualified medium and a media product belonging to the same qualified medium

. Intermedial relation between two different qualified media

. Intramedial relation ‘between two qualified media that are the same’ = within a single qualified medium

. Intramedial relation ‘between two media products that are the same’ = within a single media product


DOI: 10.1057/9781137474254.0009

All of these types of relations are understood in terms of either remedia-tion in general (repeated mediation of sensory configurations leading to a new representation of any media characteristic) or media representa-tion in general (representation of any media). Of course, this distinction lies behind my fundamental division between transmediation and media representation that, so far, was thoroughly elaborated on within the framework of intermedial relations; however, the distinction may also be applied to intramedial relations, as subsequently demonstrated. Whereas the core of the former type of transfer could be said to consist of repeti-tion (with a difference), the core of the latter type of transfer is a meta view: media representing media in some way.

In the following overview, I provide examples of inter and intramedial transfers of media characteristics on the basis of the previously described distinctions. Whereas the examples of intermedial transfers are simply reappearances of illustrations of transmediation (or, more precisely, transmedial remediation) and media representation (or, more precisely, transmedial media representation), the examples of intramedial transfers reveal the intricate relations between media transformation and other types of transfer of media characteristics.

Intermedial relation between two different media products belonging to different qualified media

Remediation – a MP remediating the C of another MP belonging to another MQ: the typical adaptation such as a film transmediating a novel; reading aloud written verbal text.Media representation – a MP representing another MP belonging to another MQ: the typical ekphrasis; the analyses of animated films in this treatise.

Intermedial relation between a qualified medium and a media product belonging to a different qualified medium

Remediation – a MQ remediating the C of a MP belonging to another MQ or a MP remediating the C of a MQ to which it does not belong: a genre of painting formed on the basis of written verbal text; a television advertisement resembling graphic novels in general.Media representation – a MQ representing a MP belonging to another MQ or a MP representing a MQ to which it does not belong: ekphrases of a single painting such as Bruegel the Elder’s,

Conclusion

DOI: 10.1057/9781137474254.0009

Landscape with the Fall of Icarus, constituting a submedium; a website providing information about music.

Intermedial relation between two different qualified media

Remediation – a MQ remediating the C of another MQ: structural similarities between painted portraits and photographic portraits; narrative traits in adventure movies resembling narrative traits in computer games.Media representation – a MQ representing another MQ: submedia such as ekphrasis and television news that to a significant extent are supposed to represent other media types.

Intramedial relation between two different media products belonging to the same qualified medium

Remediation – a MP remediating the C of another MP belonging to the same MQ: translations between different verbal languages; transcriptions of musical pieces from one instrument to another; musical parodies and covers; film remakes; all types of plagiarisms within the same qualified medium.Media representation – a MP representing another MP belonging to the same MQ: an email message discussing another email message; a scholarly text referring to or quoting another scholarly text.

Intramedial relation between a qualified medium and a media product belonging to the same qualified medium

Remediation – a MQ remediating the C of a MP belonging to the same MQ or a MP remediating the C of a MQ to which it belongs itself: the submedium of installation art possibly being developed out of the characteristics of Marcel Duchamp’s first readymade object from the early 20th century; an after-dinner speech displaying the typical characteristics of after-dinner speeches.Media representation – a MQ representing a MP belonging to the same MQ or a MP representing a MQ to which it belongs itself: a submedium of scholarly work formed on the basis of discussing one specific scholarly study; a phone conversation about phone conversations.


DOI: 10.1057/9781137474254.0009

Intramedial relation ‘between two qualified media that are the same’ = within a single qualified medium

Remediation – a MQ remediating its own C: qualified submedia such as Japanese sign language or rock music upholding their characteristics.Media representation – a MQ representing itself: the submedium of scholarly reviews of scholarly work characterized by reflections on scholarly writing in general; minor submedia such as documentary films about documentary films.

Intramedial relation ‘between two media products that are the same’ = within a single media product

Remediation – a MP remediating its own C: a hip-hop song repeating phrases and rhythms; a computer game using similar graphic traits and demanding similar actions at different levels of the game.Media representation – a MP representing itself: a mise en abyme such as a drawing of itself representing a drawing of itself and so forth; a website explaining its own structure.

Whereas some intramedial transfer variations are admittedly rather obscure, others are exceedingly common. In any event, the scheme may be viewed as both a type of summary and the beginning of a new and broader investigation that cannot be accomplished in this study. My purpose is merely to demonstrate the applicability of the notions developed in this study to an even broader field of scholarly investiga-tion and to point to the border zones of media transformation. Again, note that some of these border zones may be counted as intermedial or not depending on how the notion of medium is understood. Broadly, intermedial means relations among different qualified media, whereas intermedial narrowly means relations among different basic media. By the same token, intramedial broadly means relations among similar basic media, whereas intramedial narrowly means relations among similar qualified media. Thus, my comments on (my representation of) ‘Jabberwocky’ in the previous chapter may be understood as both inter-medial in a broad sense and intramedial in a broad sense; preferably, one decides to combine one broad and one narrow notion.

The scheme presented here is construed on the basis of a broad inter-medial notion and a narrow intramedial notion. However, in fact, this

Conclusion

DOI: 10.1057/9781137474254.0009

treatise as a whole has hovered between a broad and a narrow notion of intermediality, or a broader and a narrower notion of what it means to say that characteristics are transferred among different media types. On the one hand, I found that discussions in terms of qualified media are convenient. On the other hand, I emphasized the importance of modal similarities and differences for the transfer of media characteristics; in other words, I effectively argued in terms of basic media. Consequently, if one aims for further precision, it should be stressed that media trans-formation in a broad sense occurs among different qualified media in general (also among qualified media grounded on the same basic medium), whereas media transformation in a narrow sense occurs among different basic media.

Final remarks

In this treatise, I largely avoided standard terminology that tends to blur the significant qualities of media. Because terms such as text and image stand for such a variety of only partly related notions, they run the risk of becoming useless unless they are carefully qualified. Instead, I decided to build my account of media transformation on the notion of four media modalities to facilitate the investigation – with some amount of precision – how media differ and why these differences matter for the transfer of media characteristics. Although a limited number of modality modes exist, this limitation certainly does not entail that mapping their innumerable means of interacting and rendering media transformations possible or impossible is an easy task. Moreover, given the effectively endless number of more or less transmedial compound media charac-teristics, no shortcuts are possible. This study marks only the beginning of further exploration that I have yet to determine how to shape.

However, I am convinced that studies of media transformation and of mediality in general must include both material and mental aspects. The presemiotic notion of mediation that emphasizes the materiality of the medium and the actual contact between the perceiver and the medium needs to be combined with the semiotic notion of representation, emphasizing the mental shaping of significance. Thus, that vital material, sensorial, and spatiotemporal media differences exist and that they – thanks to the cross-modal capacities of our minds – can be partly bridged should be acknowledged: media representation and transmediation over


DOI: 10.1057/9781137474254.0009

the borders of modality modes are, to a certain extent and in many ways, possible.

Considering the immensely complex ways in which media character-istics may be said to be transferred among media, attempting to capture these processes with the aid of definitions, distinctions, categories, elaborated concepts, models, and even visual diagrams using arrows and pseudo-scientific formulas may seem hopelessly optimistic or even naïve. However, recall that, as I repeatedly claimed, the proposed categories and models are supposed to be tools for thinking. My chief aim was to clas-sify analytical viewpoints and suggest methods for approaching media interrelations. Although I certainly do not deny that substantial media similarities and differences exist that are actually independent of the perceiver’s view, all of our actual encounters with media involve mental activity; therefore, efforts to systematize media must always include the manner in which we think about media, not only how they are.

Note

The term intramedial is commonly used to refer to slightly different conceptions depending on how the notion of medium is circumscribed. See, for instance, Rajewsky, 2002: 12.

DOI: 10.1057/9781137474254.0010

Bibliography

Barbetti, C. (2011). Ekphrastic Medieval Visions: A New Discussion in Interarts Theory (Basingstoke and New York: Palgrave Macmillan).

Bernhart, W. (2008). ‘From Novel to Song via Myth: Wuthering Heights as a Case of Popular Intermedial Adaptation’, Essays on Word/Music Adaptation and on Surveying the Field, ed. D. F. Urrows, 13–27 (Amsterdam and New York: Rodopi).

Bernhart, W. and W. Wolf, eds (2010). Self-Reference in Literature and Music (Amsterdam and New York: Rodopi).

Bolter, J. D. and R. Grusin (1999) Remediation: Understanding New Media (Cambridge MA and London: MIT Press).

Bruhn, J. (2013). ‘Dialogizing Adaptation Studies: From One-Way Transport to a Dialogic Two-Way Process’, Adaptation Studies: New Challenges, New Directions, eds J. Bruhn, A. Gjelsvik, and E. Frisvold Hanssen, 69–88 (London: Bloomsbury).

Bruhn, J., A. Gjelsvik, and H. Thune (2011). ‘Parallel Worlds of Possible Meetings in Let the Right One In’, Word & Image 27, 2–14.

Bruhn, S. (2000). Musical Ekphrasis: Composers Responding to Poetry and Painting (Hillsdale NY: Pendragon).

Bruhn, S. ed. (2008). Sonic Transformations of Literary Texts: From Program Music to Musical Ekphrasis (Hillsdale NY: Pendragon).

Carroll, L. (1872). Through the Looking-Glass, and What Alice Found There (London: Macmillan).

Bibliography

DOI: 10.1057/9781137474254.0010

Clüver, C. (1989). ‘On Intersemiotic Transposition’, Poetics Today 10, 55–90.

Clüver, C. (1997). ‘Ekphrasis Reconsidered: On Verbal Representations of Non-Verbal Texts’, Interart Poetics: Essays on the Interrelations of the Arts and Media, eds U.-B. Lagerroth, H. Lund, and E. Hedling, 19–33 (Amsterdam and Atlanta: Rodopi).

Clüver, C. (1998). ‘Quotation, Enargeia, and the Functions of Ekphrasis’, Pictures into Words: Theoretical and Descriptive Approaches to Ekphrasis, eds V. Robillard and E. Jongeneel, 35–52 (Amsterdam: VU University Press).

Dalle Vacche, A. (1996). Cinema and Painting: How Art Is Used in Film (Austin: University of Texas Press).

Eisenstein, S. (1949 [1944]). ‘Dickens, Griffith, and the Film Today’, Film Form: Essays in Film Theory, ed. and trans. J. Leyda, 195–255 (New York: Harcourt, Brace & World).

Elleström, L. (2002). Divine Madness: On Interpreting Literature, Music, and the Visual Arts Ironically (Lewisburg: Bucknell University Press).

Elleström, L. (2010a). ‘The Modalities of Media: A Model for Understanding Intermedial Relations’, Media Borders, Multimodality and Intermediality, ed. L. Elleström, 11–48 (Basingstoke and New York: Palgrave Macmillan).

Elleström, L. (2010b). ‘Iconicity as Meaning Miming Meaning and Meaning Miming Form’, Signergy, Iconicity in Language and Literature 9, eds C. J. Conradie, R. Johl, M. Beukes, O. Fischer and Ch. Ljungberg, 73–100 (Amsterdam and Philadelphia: John Benjamins).

Elleström, L. (2013a). ‘Adaptation within the Field of Media Transformations’, Adaptation Studies: New Challenges, New Directions, eds J. Bruhn, A. Gjelsvik, and E. Frisvold Hanssen, 113–32 (London: Bloomsbury).

Elleström, L. (2013b). ‘A Theoretical Approach to Media Transformations’, Translatio: Transmédialité et transculturalité en littérature, peinture, photographie et au cinéma, ed. A. de Toro, 97–105 (Paris: L’Harmattan).

Elleström, L. (2013c). ‘Spatiotemporal Aspects of Iconicity’, Iconic Investigations, Iconicity in Language and Literature 12, eds L. Elleström, O. Fischer and Ch. Ljungberg, 95–117 (Amsterdam and Philadelphia: John Benjamins).

Elleström, L. (2014). ‘Material and Mental Representation: Peirce Adapted to the Study of Media and Arts’, The American Journal of Semiotics 30, 83–138.

Bibliography

DOI: 10.1057/9781137474254.0010

Elliott, K. (2003). Rethinking the Novel/Film Debate (Cambridge: Cambridge University Press).

Gaudreault, A. and Ph. Marion (2004). ‘Transécriture and Narrative Mediatics: The Stakes of Intermediality’, trans. Robert Stam, A Companion to Literature and Film, eds R. Stam and A. Raengo, 58–70 (Malden: Blackwell).

Goehr, L. (2010). ‘How to Do More with Words: Two Views of (Musical) Ekphrasis’, British Journal of Aesthetics 50, 389–410.

Greene, Th. M. (1940). The Arts and the Art of Criticism (Princeton: Princeton University Press).

Groensteen, Th. (1998). ‘Le processus adaptatif (Tentative de récapitulation raisonnée)’, La transécriture: Pour une théorie de l’adaptation, eds A. Gaudreault and Th. Groensteen, 273–7 (Québec: Nota bene).

Hames, P. ed. (2008). Dark Alchemy: The Cinema of Jan Švankmajer, 2nd edition (London and New York: Wallflower Press).

Hayward, Ph. ed. (1988). Picture This: Media Representations of Visual Art and Artists (London and Paris: John Libbey).

Heffernan, J. A. W. (1993). Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery (Chicago and London: University of Chicago Press).

Hutcheon, L. (2006). A Theory of Adaptation (New York and London: Routledge).

Jakobson, R. (1971 [1959]). ‘On Linguistic Aspects of Translation’, Selected Writings, Volume 2, Word and Language, 260–6 (The Hague: Mouton).

Jodoin-Keaton, Ch. (2011). Jan Švankmajer: Un surréalisme animé (Pertuis: Rouge Profond).

Kirchmann, K. and J. Ruchatz, eds (2014). Medienreflexion im Film: Ein Handbuch (Bielefeld: Transcript).

Kranz, G. (1981). Das Bildgedicht, Band I: Theorie Lexikon (Cologne and Vienna: Böhlau).

Krieger, M. (1992). Ekphrasis: The Illusion of the Natural Sign (Baltimore and London: Johns Hopkins University Press).

Krieger, M. (1998). ‘The Problem of Ekphrasis: Image and Words, Space and Time – and the Literary Work’, Pictures into Words: Theoretical and Descriptive Approaches to Ekphrasis, eds V. Robillard and E. Jongeneel, 3–20 (Amsterdam: VU University Press).

Kwiatkowska, A. (2013). Interfaces, Interspaces: Image – Language – Cognition (Piotrków Trybunalski: Naukowe Wydawnictwo Piotrkowskie).

Bibliography

DOI: 10.1057/9781137474254.0010

McFarlane, B. (1996). Novel to Film: An Introduction to the Theory of Adaptation (Oxford: Clarendon Press).

Paech, J. (1997). Literatur und Film, 2nd edition (Stuttgart and Weimar: Metzler).

Peirce, Ch. S. (1932). Collected Papers of Charles Sanders Peirce, Volume II, Elements of Logic, ed. Ch. Hartshorne and P. Weiss (Cambridge MA and London: The Belknap Press of Harvard University Press).

Persin, M. H. (1997). Getting the Picture: The Ekphrastic Principle in Twentieth-Century Spanish Poetry (Lewisburg: Bucknell University Press).

Petermann, E. (2014). The Musical Novel: Imitation of Musical Structure, Performance, and Reception in Contemporary Fiction (Rochester NY: Camden House).

Pethő, Á. (2010). ‘Media in the Cinematic Imagination: Ekphrasis and the Poetics of the In-Between in Jean-Luc Godard’s Cinema’, Media Borders, Multimodality and Intermediality, ed. L. Elleström, 211–22 (Basingstoke and New York: Palgrave Macmillan).

Punzi, M. P. ed. (2007). Literary Intermediality: The Transit of Literature through the Media Circuit (Bern: Peter Lang).

Rajewsky, I. O. (2002). Intermedialität (Tübingen and Basel: A. Francke).Rajewsky, I. O. (2005). ‘Intermediality, Intertextuality, and Remediation:

A Literary Perspective on Intermediality’, Intermédialités 6, 43–64.Ryan, M.-L. (2004). ‘Introduction’, Narrative across Media: The Languages

of Storytelling, ed. M.-L. Ryan, 1–40 (Lincoln and London: University of Nebraska Press).

Sager Eidt, L. M. (2008). Writing and Filming the Painting: Ekphrasis in Literature and Film (Amsterdam and New York: Rodopi).

Schober, R. (2010). ‘Translating Sounds: Intermedial Exchanges in Amy Lowell’s “Stravinsky’s Three Pieces ‘Grotesques’ for String Quartet”’, Media Borders, Multimodality and Intermediality, ed. L. Elleström, 163–74 (Basingstoke and New York: Palgrave Macmillan).

Schober, R. (2011). Unexpected Chords: Musico-Poetic Intermediality in Amy Lowell’s Poetry and Poetics (Heidelberg: Winter).

Schober, R. (2013). ‘Adaptation as Connection – Transmediality Reconsidered’, Adaptation Studies: New Challenges, New Directions, eds J. Bruhn, A. Gjelsvik, and E. Frisvold Hanssen, 89–112 (London: Bloomsbury).

Bibliography

DOI: 10.1057/9781137474254.0010

Scher, S. P. (1968). Verbal Music in German Literature (New Haven: Yale University Press).

Schröter, J. (2013). ‘Visuality and Narration in Monsters, Inc.’, Acta Universitatis Sapientiae: Film and Media Studies 7, 35–47.

Spitzer, L. (1955). ‘The “Ode on a Grecian Urn”, or Content vs. Metagrammar’, Comparative Literature 7, 203–25.

Suhor, Ch. (1984). ‘Towards a Semiotics-based Curriculum’, Journal of Curriculum Studies 16, 247–57.

Švankmajer, J. (2007). The Complete Short Films (London: British Film Institute) (DVD).

Urrows, D. F. ed. (2008). Essays on Word/Music Adaptation and on Surveying the Field (Amsterdam and New York: Rodopi).

Weber, Th. (2014). ‘Futurische Medien im Kino: Die Darstellung nicht-existenter Medien als Medialitätsreflexion’, Medienreflexion im Film: Ein Handbuch, eds K. Kirchmann and J. Ruchatz, 427–37 (Bielefeld: Transcript).

Wolf, W. (1999). The Musicalization of Fiction: A Study in the Theory and History of Intermediality (Amsterdam and Atlanta, GA: Rodopi).

Wolf, W. (2002). ‘Intermediality Revisited: Reflections on Word and Music Relations in the Context of a General Typology of Intermediality’, Essays in Honor of Steven Paul Scher and on Cultural Identity and the Musical Stage, eds S. M. Lodato, S. Aspden, and W. Bernhart, 13–34 (Amsterdam and New York: Rodopi).

Wolf, W. (2008). ‘Description: A Common Potential of Words and Music’, Essays on Word/Music Adaptation and on Surveying the Field, ed. D. F. Urrows, 197–226 (Amsterdam and New York: Rodopi).

Wolf, W. and W. Bernhart, eds (2006). Framing Borders in Literature and Other Media (Amsterdam and New York: Rodopi).

Wolf, W. and W. Bernhart, eds (2007). Description in Literature and Other Media (Amsterdam and New York: Rodopi).

Wolf, W. ed. in collaboration with K. Bantleon and J. Thoss (2009). Metareference across Media: Theory and Case Studies (Amsterdam and New York: Rodopi).

Wolf, W. ed. in collaboration with K. Bantleon and J. Thoss (2011). The Metareferential Turn in Contemporary Arts and Media: Forms, Functions, Attempts at Explanation (Amsterdam and New York: Rodopi).

Yacobi, T. (1995). ‘Pictorial Models and Narrative Ekphrasis’, Poetics Today 16, 599–649.

Bibliography

DOI: 10.1057/9781137474254.0010

Yacobi, T. (1998). ‘The Ekphrastic Model: Forms and Functions’, Pictures into Words: Theoretical and Descriptive Approaches to Ekphrasis, eds V. Robillard and E. Jongeneel, 21–34 (Amsterdam: VU University Press).

Yacobi, T. (2013). ‘Ekphrastic Double Exposure and the Museum Book of Poetry’, Poetics Today 34, 1–52.

DOI: 10.1057/9781137474254.0011

Index

adaptation, 5, 11, 26–7, 47–8animated film, 62–85Arcimboldo, G., 82–4art, 4audiovisual media, 31, 69,

75–9, 83–4auditory media, 14, 17, 27, 29,

38–43, 49–52, 68–71, 75–79

Bach, J. S., 64, 87balance, 42Barbetti, C., 35n6, 48Bernhart, W., 43, 59Bildgedicht, 35n7Bolter, J., 7–8, 14Bruegel, P., 33Bruhn, S., 33, 34

Carroll, L., 71, 73Clüver, C., 7, 32, 34cognition, 5, 42cognitive import, 2, 47complex media representation,

28–34, 41complex transmediation, 20,

22–7, 35n3, 41composite bodies, 82–4compound media

characteristics, 39–49, 51–6, 66–71, 75, 82–3

content, 41–5contrast, 42

deiction, 13, 39depiction, 13, 39

description, 13, 39, 43diachronic perspective, 3, 4, 7,

10, 47Dimensions of Dialogue, 63,

80–5

Eisenstein, S., 25ekphrasis, 5, 11, 32–4, 35n6, 48,

61n2Elliott, K., 27enunciation, 48external technical media, 2extracompositional

intermediality, 8–9

Fantasia BWV 542, 66–71, 87fictitious media, 20, 25, 29, 30film, 8, 24–6, 51flat surface, 38form, 41–5, 67framing, 43

Gaudreault, A., 48Gautier, T., 32Greene, Th. M., 43Griffith, D. W., 25Groensteen, T., 48Grusin, R., 7–8, 14

Heffernan, J. A. W., 32Hutcheon, L., 26, 44–5

iconicity, 13, 30, 38, 50–1, 61n1, 67

image, 2

Index

DOI: 10.1057/9781137474254.0011

indexicality, 13, 30, 38, 67intermediality, 3, 5, 10, 89, 90–3

intracompositional vs. extracompositional, 8–9

overt vs. covert, 8intermedial references, 9, 10intermedial translation, 7intermedial transposition, 9–10internal technical media, 2interpretation, 12intersemiotic translation, 7intersemiotic transposition, 7intracompositional intermediality, 8–9intramediality, 89–93irony, 43

Jabberwocky, 63, 71–80, 88Jakobson, R., 7J. S. Bach – Fantasy in G Minor, 63–71

known media, 20, 31Kranz, G., 35n7

Marion, P., 48materiality, 49–50material modality, 37–9McFarlane, B., 47–8meaning, creation of, 2, 12media

audiovisual, 31, 69, 75–9, 83–4auditory, 14, 17, 27, 29, 38–43, 49–52,

68–71, 75–79basic, 58categories, 3–4circuits, 23–4concept of, 2content of, 41–5external technical, 2form of, 41–5internal technical, 2interrelations, 4, 6–10, 18, 47, 89–90materiality, 5modalities of, 36–9, 47–56, 87, 89, 94qualified, 19, 21–4, 38, 40, 53, 58–9,

87spatial, 38–9, 44, 50–5, 68, 70, 78

technical, 47–56temporal, 27, 31, 38–40, 42, 44, 50–2,

66, 78, 83–5, 87transfer of characteristics of, 5–6,

8–10, 47, 50–61use of, 2

visual, 14, 15, 17, 21, 27–35, 38–43, 48–52, 64–71, 74–85

media analysis, 62–85Dimensions of Dialogue, 80–5Jabberworcky, 71–80J. S. Bach – Fantasy in G Minor, 63–71

mediality, 2medial transposition, 9media products, 58–9

defined, 19representation of, 29transmediation of, 22, 24–7

media representation, 9–10, 12–13, 15–20, 27–34, 90–4

complex, 28–34, 41concept of, 27–8diagrams of, 16, 53–6in Dimensions of Dialogue, 83ekphrasis, 32–4in Jabberwocky, 75in J. S. Bach – Fantasy in G Minor,

63–8of media products, 28–30pure, 18, 28of qualified media, 28–34simple, 28–9structure of, 54–6, 57transmediation and, 17

media studies, 2, 5–10mediation, 13

concept of, 13–14defined, 12vs. representation, 11representation and, 53

media transformationanalyses of examples, 62–85border zones of, 88–93concept of, 4, 47conceptual framework for, 5–6defined, 1

Index

DOI: 10.1057/9781137474254.0011

field of, 6–10introduction to, 1–10media representation, 12, 15–20,

27–34modality modes and, 49–56model for, 46–61research, 5transmediation, 12, 15–27types of, 11–35

medium, 14, 17, 20metareference, 43modality modes, 36–9, 47–56, 87, 89,

94models, 18motion pictures, 8, 24–6, 51music, 66–71, 87

narration, 43–5, 50–1narrative structure, 26novels, 26, 51

pattern, 42Peirce, C. S., 12, 39, 48perception, 48perspective, 43Pethő, Á., 33picture, 2presemiotic processes, 39proportion, 42

qualified media, 19, 38, 40, 53, 58–9, 87representation of, 28–34transmediation of, 21–4

Rajewsky, I. O., 9, 10remediation, 90–2repetition, 42representation

see also media representationconcept of, 13–14defined, 12–13vs. mediation, 11mediation and, 53of technical media, 52types of, 13

rhythm, 42Ryan, M.-L., 44

Sager Eidt, L. M., 33Schober, R., 7, 40Schröter, J., 43semiotic modality, 37–9, 50semiotics, 5, 12–13, 48sensorial modality, 37–9sensory configurations, 14, 53sensory perception, 5source medium, 47spatial media, 38–9, 44, 50–5, 68,

70, 78spatial thinking, 42, 44spatiotemporal modality, 37–9, 50, 51Spitzer, L., 32structure, 42structured perception, 44style, 43submedia, 26Suhor, C., 34n1Švankmajer, J., 63–4symbolicity, 13, 38, 67synchronic perspective, 3, 4, 7

target medium, 47technical media, 14, 20, 47–56temporal media, 27, 31, 38–40, 42, 44,

50–2, 66, 78, 83–5, 87text, 2transfer of media characteristics, 3–10,

47–61transformation, 3, 4, 14–15, 34, 48–9translation, 7, 91transmedial basis, 36–45transmediality, 3, 5, 40transmedialization, 34transmediation, 9–12, 14–27, 34, 34n1,

57, 93–4adaptation, 26–7complex, 20, 22–7, 35n3, 41concept of, 14–15, 20–1diagrams of, 16, 53–6in Dimensions of Dialogue, 84–5

Index

DOI: 10.1057/9781137474254.0011

transmediation – Continuedextended, 60–1of fictitious media, 25in Jabberworcky, 75–9in J. S. Bach – Fantasy in G Minor, 69–71of media products, 22, 24–7media representation and, 17from novel to movie, 51of qualified media, 19, 21–4simple, 20–2, 35n3transformation and, 14–15

transmutation, 7

unknown media, 20, 29, 30Urrows, D. F., 26

visual media, 14, 15, 17, 21, 27–35, 38–43, 48–52, 64–71, 74–85

Wolf, W., 8–10, 43

Yacobi, T., 35n3, 61n2

media transformation: the transfer of media characteristics among media

Documents