wilkie

Analysis of conceptual metaphors to inform music interaction designs

Katie Wilkie

[email protected] Supervisors Dr Simon Holland

Dr Paul Mulholland Department/Institute Music Computing Status Part-time Probation viva After Starting date April 2008 Music is interwoven through many facets of our daily lives and experiences, from the deep to the trivial. It can be both an art form providing a conduit by which emotions and ideas can be communicated, and a means to communicate personal tastes through, for example, the choice of a mobile ringtone. Despite the ubiquity of music, opportunities for non-experts to interact with music in meaningful ways, to understand it and to affect it by, for example creating and manipulating melodies and harmonic progressions, are limited. Popular and pervasive though music is, understanding and analysing the structural properties of musical artifacts often requires knowledge of domain terminology and notation. Such specialist knowledge is generally restricted to highly trained domain experts who have pursued a path of detailed academic study. In particular, musical concepts such as harmonic progression and voice leading, which make use of a number of different terms and notations to describe various parameters and aspects, can be difficult to understand and describe. Furthermore, providing ways of interacting with music that are sufficiently expressive for experts whilst still being usable by non-experts remains an open challenge. We hypothesise that if we can represent this specialist knowledge in a form that exploits pre-existing and universally held sensory-motor experiences, we will be able to lower some of the barriers to musical expression. Thus we believe that music interactions designed in this manner would lessen the requirement for specialist domain knowledge and be more intuitive to both domain experts and novices alike. The identification of image schemas, exposed through linguistic constructs, provides a promising foundation for this work. Image schemas are defined by Johnson (2005) as “recurring patterns of our sensory-motor experience” where the experiences Johnson is referring to are those of interacting with other bodies, space and forces within our environment. Johnson further hypothesises that these image schemas can be applied to other, often abstract, domains through the creation of conceptual metaphors, enabling us to develop our understanding of more complex abstract concepts. Image schema and conceptual metaphor theories have already been applied to a number of different domains such as arithmetic (Lakoff, Nunez 2007), musical concepts (Saslaw 1996, 1997, Zbikowski 1997a, 1997b, Brower 2000, Larson 1997, Johnson 1997, Johnson, Larson 2003, Eitan, Granot 2006, Eitan, Timmers 2006), user interface design (Hurtienne, Blessing 2007) and music

2010 CRC PhD Student Conference

14 of 125

interaction design (Antle et al. 2008, 2009). In the domain of user interface design for example, Hurtienne and Blessing (2007) carried out experiments attempting to determine whether user interface controls which were configured to support simple conceptual metaphors such as MORE IS UP, a metaphorical extension of the UP-DOWN image schema, would be more intuitive to use. Their results do appear to support this hypothesis to an extent, however only a small number of user interface controls and conceptual metaphors were tested. In the domain of music theory, work by Saslaw (1996, 1997), Zbikowski (1997a, 1997b), Brower (2000), Larson (1997), Johnson (1997, Johnson, Larson 2003) and Eitan et al. (Eitan, Granot 2006, Eitan, Timmers 2006) has used image schemas and conceptual metaphors in an attempt to increase our theoretical understanding of musical concepts. This has yielded promising results indicating that musical concepts can be understood in terms of image schemas and conceptual metaphors. Antle et al. (2008, 2009) designed an interactive sound generation system based on embodied metaphors that allowed users to generate sounds and modify simple sound parameters through body movement. They ran a series of experiments attempting to establish whether this approach to interaction design enhanced the ability of children to learn about sound concepts. Although the results were inconclusive, they did highlight the importance of discoverability of the embodied metaphors used in the interaction model. This research draws upon these works, aiming to establish if the conceptual metaphors elicited from dialogues between musicians discussing various musical concepts can be used to inform interaction designs for communicating information about, expressing and manipulating complex musical concepts such as harmony and melody. Thus, the specific questions this research aims to address are as follows:

1. How can conceptual metaphors aid our understanding of the musical concepts of pitch, melody and harmony?

2. How can the conceptual metaphors identified be used to inform and evaluate the design of music interactions for communicating information about and manipulating pitch, melody and harmony?

Methodology In order to address the question of the ways in which conceptual metaphors aid our understanding of the musical concepts of pitch, melody and harmony, we must first identify the conceptual metaphors that experienced musicians use to understand, define and describe such phenomena. A series of studies have been planned involving musicians from both classical and popular traditions. The participants will be provided with musical artifacts in different representation formats (e.g. musical score, audio file and piano roll) and asked to discuss aspects of the artifacts in order to elicit a dialogue which can then be analysed to identify the conceptual metaphors in use. Once a collection of commonly used musical conceptual metaphors has been identified, it is planned to validate these with a wider audience through the use of an online questionnaire. The second research question, regarding the use of conceptual metaphors to evaluate and inform music interaction designs, will be addressed by firstly evaluating a number of existing music interaction designs using the identified


Page 115 of 125

musical conceptual metaphors. The results of these evaluations will be used to generate a series of guidelines for designing music interactions. In order to validate the guidelines, example music interactions will be developed based on the guidelines and subsequently evaluated with participants to establish their suitability. A summary of the work plan for these tasks is provided in the table below. Dates Task May 2010 – Dec 2010 Identify and validate the musical conceptual

metaphors used by musicians through a series of studies and an online questionnaire.

Jan 2011 – Apr 2011 Evaluate existing music interaction designs using the identified musical conceptual metaphors and establish a series of design guidelines/patterns for designing future music interactions.

May 2011 – Dec 2011 Implement a number of small-scale solutions based on the defined design guidelines and evaluate these solutions to further improve the guidelines.

Jan 2012 – Jun 2013 Write-up. At this stage, one study has already been completed (Wilkie, Holland and Mulholland 2009) and further studies are in the process of planning. Contributions It is envisaged that this research will provide the following contributions to the field:

1. Increase knowledge of how conceptual metaphors aid understanding of musical concepts such as pitch, melody and harmony. This will be achieved through identifying and validating the conceptual metaphors used by musicians when discussing various aspects of music.

2. Some preliminary indication of how different musical representation formats affect and align with the conceptual metaphors elicited during discussions.

3. Improve knowledge of how musical conceptual metaphors can be used to evaluate and inform the designs of intuitive music interactions. This will be achieved through the development of a series of design guidelines aimed at assisting designers to make decisions about the most appropriate manner for communicating information about and manipulating specific musical parameters.

References ANTLE, A.N., CORNESS, G. and DROUMEVA, M., 2009. Human-computer-intuition? Exploring the cognitive basis for intuition in embodied interaction. International Journal of Arts and Technology, 2(3), 235-254. ANTLE, A.N., DROUMEVA, M. and CORNESS, G., 2008. Playing with the


Page 116 of 125

sound maker: do embodied metaphors help children learn? Proceedings of the 7th international conference on Interaction design and children, 2008, ACM pp178-185. BROWER, C., 2000. A cognitive theory of musical meaning. Journal of Music Theory, 44(2), 323-379. EITAN, Z. and GRANOT, R.Y., 2006. How Music Moves: Musical Parameters and Listeners' Images of Motion. Music Perception, 23(3), 221-247. EITAN, Z. and TIMMERS, R., 2006. Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context, Proceedings of the 9th International Conference on Music Perception and Cognition, 2006, pp875-882. HURTIENNE, J. and BLESSING, L., 2007. Design for Intuitive Use - Testing Image Schema Theory for User Interface Design, Proceedings of the 16th International Conference on Engineering Design, 2007, pp1-12. JOHNSON, M., 2005. The philosophical significance of image schemas. In: B. HAMPE and J. GRADY, eds, From Perception to Meaning: Image Schemas in Cognitive Linguistics. Berlin: Walter de Gruyter, pp. 15-33. JOHNSON, M., 1997. Embodied Musical Meaning. Theory and Practice, 22-23, 95-102. JOHNSON, M.L. and LARSON, S., 2003. Something in the Way She Moves-Metaphors of Musical Motion. Metaphor and Symbol, 18(2), 63-84. LAKOFF, G. and NUNEZ, R.E., 2000. Where Mathematics Comes From. Basic Books. LARSON, S., 1997. Musical forces and melodic patterns. Theory and Practice, 22-23, 55-71. SASLAW, J., 1996. Forces, Containers, and Paths: The Role of Body-Derived Image Schemas in the Conceptualization of Music. Journal of Music Theory, 40(2), 217-243. SASLAW, J.K., 1997. Life Forces: Conceptual Structures in Schenker’s Free Composition and Schoenberg's The Musical Idea. Theory and Practice, 22-23, 17-34. WILKIE, K., HOLLAND, S. and MULHOLLAND, P., 2009. Evaluating Musical Software Using Conceptual Metaphors, Proceedings of the 23rd British Computer Society Conference on Human Computer Interaction, 2009, British Computer Society pp232-237. ZBIKOWSKI, L.M., 1997a. Conceptual Models and Cross-Domain Mapping: New Perspective on Theories of Music and Hierarchy. Journal of Music Theory, 41(2), 193-225. ZBIKOWSKI, L.M., 1997b. Des Herzraums Abschied: Mark Johnson's Theory of Embodied Knowledge and Music Theory. Theory and Practice, 22-23, 1-16.


Page 117 of 125