Towards a humane graphical user interface for live electronic music

Jamie BullockBirmingham Conservatoire

Birmingham City UniversityUK

james.bullock@bcu.ac.uk

Lamberto CoccioliBirmingham Conservatoire

Birmingham City UniversityUK

lamberto.coccioli@bcu.ac.uk

AbstractIn this paper we describe findings related to user interfacerequirements for live electronic music arising from researchconducted as part of the first three-year phase of the EU-funded Integra project. A survey of existing graphical toolsfor live electronics is presented along with observations aboutcurrent usage patterns and cultural trends. From these data aset of requirements are gathered, and resulting design ques-tions are discussed. A number of graphical user interface(GUI) prototypes developed during the Integra project ini-tial phase are described and conclusions drawn about theirdesign and implementation. Finally a proposal is made for anew GUI that takes into account the findings of our research.

Keywords: Integra, User Interface, Usability, Design, LiveElectronics, Music Technology

1. IntroductionAccording to Raskin[1] in order to be humane, interfacesshould meet the following criteria:

• modeless: user actions should have the same effectregardless of the application’s state.

• monotonous: there should only be one way to accom-plish any given task in the UI.

• visible: the ‘right’ features of an application shouldbe visible at any given time and users shouldn’t beforced to memorise that a feature exists.

• affordance: UI functions and operation should be ob-vious to most people in the culture for which it was in-tended and make use of already learned human skills.

A humane interface is a usable interface designed to besympathetic to the way humans instinctively interact withcomputers, and not necessarily designed around the struc-ture of computer hardware and operating systems. Interfaceusability is closely coupled with the notion of the humane

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page.c© 2009 Carnegie Mellon University

Figure 1. Graph of qualities for user interfaceacceptability.[4]

interface and is often defined in the broader context of sys-tem acceptability. Figure 1 shows a diagrammatic repre-sentation of an early acceptability graph devised by JakobNielsen[4]. Consideration of complete system acceptabil-ity is beyond the scope of the paper, instead, we will focuson the usability branch. Usability is defined by Nielsen ashaving the following five attributes:

• Learnability: The system should be easy to learn sothat the user can rapidly start getting some work donewith the system.

• Efficiency: The system should be efficient to use, sothat once the user has learned the system, a high levelof productivity is possible.

• Memorability: The system should be easy to remem-ber, so that the casual user is able to return to the sys-tem after some period of not having used it, withouthaving to learn everything all over again.

• Errors: The system should have a low error rate, sothat users make few errors during the use of the sys-tem, and so that if they do make errors they can easilyrecover from them. Further, catastrophic errors mustnot occur.

• Satisfaction: The system should be pleasant to use,so that users are subjectively satisfied when using it;they like it.

Nielsen argues that usability isn’t just a broad, multi-faceted concept, but something that can be quantified andqualitatively measured. In this paper we will evaluate soft-ware currently used for live electronic music in the contextof humane and usable interface considerations. In particularwe will discuss the usability evaluation processes employedas part of the Integra project prototyping phase.

2. Research processIntegra 1 is a project led by Birmingham Conservatoire inthe UK and supported by the Culture programme of the Eu-ropean Union. Integra brings together new music ensem-bles and research centres to collaborate on a wide range ofartistic and scientific activities centred around live electronicmusic. The initial 3-year phase of the project is now com-plete. During this period eleven European composers werecommissioned to write works for five professional ensem-bles, partners in the project. As part of the composition pro-cess the composers were each paired with one of the eightresearch centres in order to develop the ‘live electronics’ oftheir compositions.

The process resulted in successful performances of allworks involved including premieres at the first Integra Fes-tival held in Birmingham in June 2008. However it alsohighlighted some important issues relating to the Integraproject’s scientific objectives, namely the development ofa new environment for live electronics. The two followingobservations are particularly pertinent:

• The tool chosen by the majority of composers for thecomposition and performance of the live electronicswas Max/MSP by Cycling74 2 .

• The relationship between composers and research cen-tres was primarily collaborative, but in many casescomposers were highly dependent on the research cen-tres for technical assistance.

We observed in some cases, that the traditional relation-ships of composer and musical assistant emerged, with themusical assistants interpreting, facilitating or simply execut-ing the composer’s artistic wishes on their behalf. Com-posers often employed other tools, for example Protools,Csound, VST plugins as auxiliary applications in the cre-ative process. These were used for preparatory work suchas pre-processing source material prior to incorporating intoa ‘live’ context.

Since most of the compositions commissioned by Integrainvolved interaction between (acoustic instrumental) perform-ers and the electronics, we were also able to observe thereactions of the performers to the live electronics systemsdevised by the composers and researchers. Reactions were

1 www.integralive.org2 www.cycling74.com

Performer Composer 'Technician'

ElectronicsTEST SYSTEM

REPORTISSUE

REQUESTCHANGE

IMPLEMENTCHANGE

Figure 2. Diagram illustrating observed dependence be-tween performer, composer and ‘technician’.

Performer Composer

Electronics

TEST SYSTEM/IMPLEMENT

CHANGE

DISCUSSELECTRONICS

Figure 3. Diagram illustrating ideal scenario where per-former and composer interact with the software on equalterms without technical assistance.

mostly positive, with performers showing a sense of discov-ery and excitement in ‘playing’ with the electronics. How-ever, in some cases this was accompanied by a feeling ofuncertainty, especially when the electronics seemed not tobe working as expected. We also noticed a secondary levelof dependence for technical assistance if changes were re-quired in the electronics for musical reasons. This ‘distanc-ing’ of the performers from the electronics is shown in Fig-ure 2.

We believe that if the software systems used for the liveelectronics were sufficiently humane and usable then theperformer and composer would become empowered, andless dependent on the ‘technician’. This would lead to therelationship shown in Figure 3. Max/MSP, the tool em-ployed by the majority of composers in our case study, is apowerful and flexible dataflow programming language, whichpresents almost limitless possibilities for the digital artist.However, for composers and performers to gain sufficientfluency to avoid the need for technical assistance would re-quire years of study. Our hypothesis is that this is not andshould not be necessary for working creatively with instru-ments and live electronics.

2.1. Problems identifiedIn the course of our research we have identified the follow-ing key problems:

1. Musicians are discouraged from working with live elec-tronics due to difficulties using the software.

2. Musicians will often collaborate with a technical ex-pert if they intend to use live electronics in their work.

Figure 4. Screenshot of a basic Jamoma patch.

3. There are no purpose-built applications for live elec-tronic music; general purpose tools and programminglanguages are used instead.

By surveying a range of software available to musicians,we established that item 3 in the above list is one of thecauses of items 1 and 2. However, despite the lack of purpose-built systems for live electronics, there are a number of projectsin progress, which aim to address usability issues in generalpurpose ‘multimedia processing’ software. Some of thesewill be discussed in the following section.

3. Existing systemsThis review is not intended to be exhaustive, but rather togive a range of contrasting approaches to the problem ofuser interface design in this field.

3.1. JamomaJamoma consists of a number of interoperable modules builtin the Max/MSP environment, and a set of guidelines formodule construction[2]. Jamoma provides a number of us-ability improvements compared to Max/MSP including in-dividual ‘pre-built’ module GUIs, human-readable parame-ter addressing, HTML documentation and a flexible mech-anism for control over composite parameters[3]. The pri-mary advantage of using Jamoma over Max/MSP is the na-ture of the modules provided, which are generally ‘higherlevel’ from a systems architecture perspective, and encapsu-late functionality that is specifically tailored to the needs ofinteractive music and video. The Jamoma modules essen-tially provide an artist’s toolkit without presenting the userwith too many choices or requiring a significant amount ofspecialist knowledge.

Figure 5. Screenshot showing ixiQuarks ‘predators’ UI builtwith Supercollider.

3.2. ixiixi works on a different principle to Jamoma, treating thegraphical user interface not as a tool, but as a musical instrument[5].Each ixi GUI is a unique user experience, resembling a graph-ical toy or game to be played by the performer, artist or ca-sual experimenter. From a usability perspective ixi GUIsinvite an exploratory approach to interaction. They are usu-ally graphically simple and focus on one idea such as theinterconnection of certain lines and shapes for a given musi-cal task. Such tasks include granular synthesis, algorithmicnote generation or beat slicing. The strength of ixi lies in theexpressive but often non-obvious relationship between thegraphical interaction and the resulting change in the soundoutput. ixiQuarks is a recent development of the ixi conceptproviding a complete system of interoperable tools for audioprocessing and synthesis[6]. Some of the ixiQuarks compo-nents have deterministic user interfaces where the UI con-trols and DSP are closely coupled, whereas tools in the ‘in-struments’ category have an abstract game-like quality akinto the early ixi releases. An example ixiQuarks instrumentis shown in figure 5.

3.3. BiduleSimilar in emphasis to Jamoma, Bidule is a commercialsoftware package consisting of modules (called bidules), whichcan be connected up in a patcher environment. Like Jamoma,each module has its own UI. As a modular patching environ-ment Bidule has many usability advantages over Max/MSP:

• Adding modules to the patcher is drag n drop (no typ-ing).

• Textual labels are used instead of (non-obvious) icons.

Figure 6. Bidule with individual Delay module GUI shown.

• Modeless interface (no ‘edit’ or ‘performance’ mode).

• New patches contain sensible default objects (inputs,mixer, output) to get the user started.

• All parameters are OSC (Open Sound Control) ad-dressable by default.

• Colour-coded connections for audio, MIDI and spec-tral data types.

• Massively simpler preferences dialogue.

Bidule comes with a collection of more than one hundredmusically usable modules and can run as a VST plugin. Ascreenshot is shown in Figure 6.

3.4. IanniXOriginally developed by La Kitchen in Paris, IanniX is in-spired by the UPIC system devised by the late Iannis Xe-nakis. The software proposes a graphical representation ofa ‘multi-dimensional and multi-formal score, a kind of poly-temporal meta-sequencer’[8]. IanniX differs from the otherinterfaces discussed so far in that it has no audio process-ing ‘engine’ associated with it. Instead it generates controlmessages in OSC format, which can be processed by any ap-plication with a sufficiently configurable OSC server. Thismakes IanniX very flexible but impacts on its usability. Theuser must locate and configure a suitable piece of audio soft-ware before any sound can be produced. The IanniX teamprovides examples for a range of common applications andenvironments to assist with this.

IanniX is not directly concerned with the DSP graph ofa live electronics system, but rather presents a new way ofconceptualising the structure of a musical score and its re-lation to absolute (performed) time. In IanniX the visuali-sation in the user interface is the score, and indeed the ap-proach it takes is entirely idiomatic to and enabled by com-puter technology – a multidimensional score (in the IanniXsense) would be impossible for a human conductor to fol-low. Although the IanniX score is designed to be interpreted

Figure 7. IanniX GUI showing triggers cursors and trajec-tories in a IanniX score with time on both x and y axes.

by a computer rather than a performer, it suggests new pos-sibilities for human-computer interaction not found in con-ventional software dealing with time in a linear way, suchas MIDI sequencers and digital audio workstations. Time inIanniX is represented spatially using a pseudo-3D graphicalenvironment, with vector graphics such as circles and linesrepresenting control data trajectories. An example is shownin Figure 7.

4. Towards a GUI for live electronicsThe Integra project seeks to develop a new graphical userinterface drawing on best practices from the software dis-cussed in section 3, whilst taking account of the usabilitycriteria highlighted in section 1. In the following sectionswe outline our proposal for the specification of a new GUIfor this purpose.

4.1. User interface modelWe started our research and design process by identifyingthe possible personas, scenarios and prototypes for the inter-face using a model proposed by Tidwell[7]. This is shownin Figure 8. The diagram illustrates the large number ofstakeholders in a specialised interface for live electronics,and highlights the inadequacy of existing environments forthis task. For example the behavioural prototypes, ‘compo-sition’, ‘rehearsal’, ‘performance’, and ‘development’ aremostly not represented in the interfaces explored in sec-tion 3. Max/MSP is the exception, having a ‘performance’mode that allows the visual layout of certain components tobe independent from the logical layout of the patch. Sincecomposition, rehearsal and performance all require differ-ent modes of thought at different stages of completion ofthe musical work, it seems logical that the UI should reflectthis.

Figure 8. Possible user interface scenarios in software forlive electronics.

4.2. User interface conceptOverall, the Integra environment concept is based on a modelview controller (MVC) paradigm, where the GUI forms theview, a shared library (libIntegra) provides the controllerfunctionality and an online database (or local XML file hi-erarchy) provides the model.

A conceptual framework for the user interface has beendevised by analysing the requirements of composition, re-hearsal and performance activity in practical music mak-ing observed throughout the first 3-year phase of the Inte-gra project. The core components of the interface have beenidentified in Figure 9. This diagram shows the core elementsof the user interface as perceived from the user’s perspec-tive: components in the INPUT and OUTPUT sections areconceptually fixed and always accessible through the top-level UI without the need for the user to ‘build’ anything.The PROCESSING components are conceptually fluid andsubject to user instantiation, manipulation and interconnec-tion.

4.3. Interface prototypesA number of user interface prototypes have been developedreflecting the model outlined in section 4.2. We started witha small number of GUI mockups showing an initial layoutand workflow. These mockups and supporting documenta-tion are available on the project wiki 3 . UI prototypes havesubsequently been developed in Java, Javascript/XUL, andMax/MSP/Javascript. Multi-platform development has beenchosen in the early stages of the project in order to simul-taneously develop UI ideas and evaluate development envi-ronments for the final product. The most evolved prototypewas developed by Integra project partner CIRMMT and isshown in Figure 11.

This GUI is functionally compatible with libIntegra ver-

3 http://wiki.integralive.org/integra2 design

INPUT

OUTPUT

Inside World

Synthesis

Audio pools

Video pools

Image pools

Score

Keyboard

Sensors

Camera

Trackpad

Analysis data

Outside World

MicrophonesSensors

Video

MIDI

OSC

Electronic score Audio (line level)

Images

PROCESSINGʼFXʼ processing Timelines

Cuelists Conditioning

Mapping

Inside World

data visualisation

displays/monitor

Outside Worldloudspeakers video screens

force feedback

MIDI interfaces/instruments electronic scores

Figure 9. Integra user interface representational require-ments.

Figure 10. Early Integra UI mockup showing possible time-lines ’tab’.

Figure 11. Integra prototype GUI developed in Max/MSPby CIRMMT, McGill University, Montreal.

sion 0.3.1[9] and is capable of loading, connecting and send-ing control data to Integra modules hosted in a supportedDSP environment.

5. Future WorkNow that we have several GUI prototypes and a body ofresearch in performer/composer HCI to inform our work,our next task is to implement the user interface in a morerobust and full-featured manner taking into account usabil-ity deficiencies found in the GUI prototypes. We have ob-tained funding to achieve this and plan to make a publicrelease in 2010. We intend to capitalise on our positionas Conservatoire-based researchers to employ user testingearly in the development process, through to release.

6. ConclusionsAccording to Raskin “An interface is humane if it is respon-sive to human needs and considerate of human frailties”[1].The Integra project seeks to create a new graphical user in-terface specifically for use in live electronic music, whichmeets these demands. We have discussed the findings fromour work during the initial 3-year period of the Integra project,including evidence from HCI studies in the composition andperformance processes resulting from eleven commissionsof new works. We have identified a number of existing userinterfaces, which represent the state-of-the-art in design forlive audio and video processing in an electronic music con-text and conclusions have been drawn about the most salientfeatures of this software in relation to new interface design.Finally we have described a new conceptual model for a liveelectronics interface leading to a number of prototype sys-tems.

References

[1] J. Raskin, The Humane Interface: New Directions for De-signing Interactive Systems, Addison-Wesley Professional,2000.

[2] T. Place and T. Lossius “Jamoma: A modular standard forstructuring patches in Max.” in Proceedings of the Interna-tional Computer Music Conference (ICMC), New Orleans,2006.

[3] T. Place, T. Lossius, A. R. Jensenius and N. Peters “ Flex-ible control of composite parameters in Max/MSP” in Pro-ceedings of the International Computer Music Conference(ICMC), Belfast, 2008.

[4] J. Nielsen, Usability Engineering, Academic Press, 1993.[5] T. Magnusson. “ixi software: The Interface as Instrument” in

Proceedings of the 2005 International Conference on NewInterfaces for Musical Expression (NIME05), Vancouver,2005.

[6] T. Magnusson. “The ixiQuarks: Merging Code and GUIin One Creative Space” in Proceedings of the InternationalComputer Music Conference (ICMC), Copenhagen, 2007.

[7] J. Tidwell. Designing Interfaces, O’Reilly, 2005.[8] T. Coduys and G. Ferry “IanniX aesthetical/symbolic visu-

alisations for hypermedia composition” in Proceedings In-ternational Conference Sound and Music Computing, Paris,2004.

[9] J. Bullock and H. Frisk “LibIntegra: A System for Software-Independent Multimedia Module Description and Storage”Proceedings of the International Computer Music Confer-ence (ICMC), Copenhagen, 2007.