The Augmentalist: Enabling Musicians to DevelopAugmented Musical Instruments

Dan Newton and Mark T. MarshallInteraction and Graphics Group

University of Bristoldn6326@bristol.ac.uk, mark@cs.bris.ac.uk

ABSTRACTWe present the Augmentalist: a system that allows a musi-cian to augment their existing musical instrument using a va-riety of sensors. This system is focused on the musician andtheir musical aims, rather than the sensor technology beingused. The overall goal is to create allow musicians to eas-ily explore new musical performance techniques with aug-mented musical instruments. We present the design processand implementation of the system, an example of an instru-ment developed by one of the 10 participating musicians anda discussion of a number of issues raised by the project, in-cluding musicians as designers, the potential for explorationand mastery and the transferability of instruments betweenperformers.

Author KeywordsInteraction, Music, Augmented Instruments, Tangible Inter-faces, Embodied Interaction

ACM Classification KeywordsH.5.5 Information Interfaces and Presentation (e.g., HCI):Sound and Music Computing

General TermsDesign, Human Factors

INTRODUCTIONThe simultaneous availability of inexpensive computing hard-ware, new sensors capable of measuring a variety of phys-ical parameters, and software for real-time sound synthesisand manipulation has resulted in the development of newdigital musical instruments. Increasing numbers of such in-struments are being produced each year, with 266 new in-struments presented in the first 8 years of the NIME confer-ences[6]. However, with a few notable exceptions such asthe Reactable[1], these instruments tend only to be playedby their designers, rather than being taken up by other musi-cians. Why then do these new instruments not appear on the(more mainstream) music scene?

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.TEI’11, January 22–26, 2011, Funchal, Portugal.Copyright 2011 ACM 978-1-4503-0478-8/11/01...$10.00.

A number of possible reasons exist for this lack of uptake.Armstrong suggests one possibility: a disconnect betweenthe performer, the instrument and the audience[2]. The inter-actions involved in controlling a digital musical instrumentare not always obvious to the audience. Alongside this, theaudience often have no experience of the instrument beingplayed, and as such have no point of reference with regardsto the performance.

An exception to this can be the type of digital musical in-strument known as augmented or extended instruments[8].Such instruments are created by embedding additional sen-sors on a traditional instrument and using them to allow theperformer additional control over the sonic output of the in-strument (e.g. [7], [9], [5]). These instruments can maintainsome characteristics of the instrument with which the audi-ence is familiar, allowing the audience to better relate to theperformance.

Another possible issue is that of musicality. In the courseof our research, we asked participating musicians to view anumber of videos of digital musical instrument performances.Several musicians spoke of a lack of (perceived and/or tradi-tional) musicality in the performances that would discouragethem from playing such an instrument.

How then can we produce an instrument that offers a perfor-mance experience that is somewhat familiar to the audience,while also producing sounds that are “musical” enough toappeal to more traditional musicians and allow them to beincorporated into performances with other musicians and in-struments? The work discussed in this paper attempts to dealwith this issue by enabling musicians to create their ownaugmented instruments. This allows the performer to de-sign the sound and interaction of their new digital musicalinstrument, which is based on their existing traditional mu-sical instrument. During the entire design process, two coreprinciples formed the basis for all of the decisions made:the system needs to be able to produce musically pleasingresults and a musician must be able to tailor the system totheir needs.

SYSTEM DESIGNThe design process for the Augmentalist took an iterative,user-centred approach. This involved numerous consulta-tion, testing and design sessions with musicians. The overallgoal of this process was to ensure that the system is designedin such a way as to be useful to the musicians themselves.

249

The consultation process began as soon as the project itselfwas conceived. It began with a series of short meetings witha number of different musicians. These meetings includedsessions with single musicians and also with groups of mu-sicians. The aim of these initial sessions was simply to gaugeinterest in the project itself and to attempt to determine whatfeatures of the system would be useful to a variety of musi-cians.

From these sessions, we arrived at a design that involved al-lowing the musicians to attach sensors the their instrumentsand then map the output from these sensors to MIDI sig-nals to control parameters of audio software with which themusicians are familiar. Our job then is to provide a systemthat allows this mapping to take place in as simple a way aspossible, so that the musician can concentrate on the effectthey wish to create, rather than on the intricacies of reading,processing and translating sensor data. Figure 1 shows anexample of an augmented instrument created using the sys-tem.

Figure 1. An example of an instrument augmented using the Augmen-talist system. A slider, attached to the body of the guitar, allows theperformer to modify effects with their strumming hand.

IMPLEMENTATION

HardwareAfter experimenting with different available sensors and sen-sor systems, we decided to use Phidgets[4]. These sensorsrequired no soldering or programming on behalf of the users.Thus they are ideal for a system designed for musicians dueto their plug and play capabilities1. The choice of the Phid-gets system also allows for a large range of sensors to beavailable to the user, with dozens of sensors currently avail-able from the manufacturers that plug directly in to the inter-face with no electronic skills required to use them.

The sensors are connected via USB 2.0 to a computer. Forour initial implementation we used a 2.53GHz MacBook ProRunning OS X 10.6 using a Stanton Scratch Amp firewireaudio interface for audio input/output. This setup showed noperceptible latency for live performance.

Software

1Due to the modular nature of the system, other sensor interfacescould be used in place of the Phidgets.

To convert the sensor data into MIDI signals we used theMax/MSP programming environment. This had the advan-tage of being easy to use, as well as being fully compati-ble with the Phidget sensors. It is also an environment withwhich some musicians are familiar. The interface in Max/MSPallows the user to choose which sensors to map to specificMIDI channels, as well as setting the desired input rangefrom the sensor, output range for the MIDI channel and themapping between them.

The software allows the user to select which sensor is mappedto to which MIDI signal using a simple graphical interface.The range of sensor values to be mapped can be selected bydemonstration, with the performer moving the sensor throughits desired range. The MIDI output range can also be limitedto a specific range in the software. Finally, the user can spec-ify the mapping function used to convert sensor data to MIDIdata. This can be selected from a range of presets (linear in-creasing, logarithmic increasing, linear decreasing, etc.) orby drawing a mapping function in the interface.

The MIDI data from Max/MSP and the raw audio signalfrom the instrument are combined in the digital audio work-station Logic Pro 9. Here the MIDI signal can be linked toany control in the software, allowing the musicians to bothmanipulate the incoming audio and to add new synthesisedsounds to this audio.

There are two levels of mapping in this set up. Firstly thegesture data from the sensor must be mapped to a controlsignal in the software. The sensor parameters are set up bythe user within Max/MSP. Secondly the MIDI signal mustbe mapped to a parameter in Logic Pro, again by the user.Both levels of mapping are left to the musician, maintainingtheir control of the actions of the system.

THE AUGMENTALIST IN USE: AN EXAMPLE SETUPOver the course of the development of this project we haveconstantly involved a number of musicians in the design,testing and development process. In total 10 musicians havespent over 60 hours using the system with more signed upfor future development. The musicians involved have in-cluded guitarists, DJs, bassists, saxophonists and vocalists.In this section we discuss an example instrument developedby one of the musicians, as it provides an insight into thetype of instruments being developed and the process used bythe musicians in developing them.

For this example we have chosen an augmented guitar cre-ated by one of the guitarists that took part in the developmentprocess. This example has been chosen for a number of rea-sons. Firstly, the guitar is an instrument with which mostpeople have some familiarity, which allows for an easier un-derstanding of the final instrument, Secondly, the guitar hasbeen the focus of some research in terms of sensing, aug-mentation and interface development (for examples, see [5],[3]). Finally, electric guitarists in particular commonly makeuse of sound processing and effects in the course of their per-formances and so are comfortable with how such effects canbe used.

250

After much experimentation, the guitarist arrived at a config-uration using 3 sensors mapped to 3 specific effects in LogicPro. Firstly, an accelerometer was mounted to the guitarneck. Used as a tilt sensor this allowed the angle between theguitar neck and the ground to be used as a control. Secondly,a slider was placed on the body of the guitar, below the pick-ups and parallel to the strings. This allowed the guitarist tocontrol effects using fingers from their picking hand, with-out interfering with their playing. Finally, an infrared dis-tance sensor was attached to the body of the guitar, in such away that when the guitarist strummed the strings their handwould pass over the sensor. This sensor was configured as anon/off switch which triggered whenever the guitarist’s handpassed over it.

The data from these sensors were then mapped to effects inLogic Pro. The guitarist first mapped the angle of the neckto the volume of a delay effect. The system was set up sothat moving the neck up in respect to the body caused thevolume of the delay effect to increase. The slider on the gui-tar body was then mapped to a distortion effect known as theBitCrusher. The guitarist was able to use the extra fingers ofhis picking hand to move the slider forwards and backwardswhilst playing, changing the resolution of the BitCrusher ef-fect and thus adding different harmonics to the sound. Fi-nally the infrared detector was used as a switch to controlthe volume. By mapping the on signal of the switch to adrop in volume, a rhythmic effect was generated from thespeed of the strumming hand.

DISCUSSION

Musicians as developersThe Augmentalist system was designed from the start to al-low musicians to become the developers of their own aug-mented instruments. We believe that it is musicians whoknow the most about their instruments and about the soundsand music they wish to create and so it is the musician whoshould make the decisions on how the instrument shouldwork.

Over the course of the development of this system so far,the participating musicians developed hundreds of differentgesture to sound mappings. While this is a large number ofdifferent mappings, what is interesting is that the musiciansthemselves considered far more of these mappings to be suc-cessful than not. The system not only enabled them to de-velop new mappings, but resulted in mappings the musiciansfound to be interesting, useful and musical.

The use of the Augmentalist also resulted in some interest-ing discoveries. Most notable is the use of the infrared dis-tance sensor as a switch to detect strumming by one guitarist.This provides a very simple and robust method of detectingstrumming, and is one that we have been not previously seenin the literature on augmented instruments.

Secondly, we found that one of the musicians developed amapping for their guitar that mirrored that presented in [5]and did so within the first 2 hours of using the system. Thisemphasises how quickly interesting and usable instruments

can be developed by a musician when given access to such asystem.

Focus on technologyOur initial idea of how musicians would develop their instru-ments was that they would focus on cause and effect. Thatis, that they would decide on a gesture to detect and a sonicoutput to control with that gesture, However, over the courseof this work we found that many of the musicians insteadfocused on the technology. This meant that the started byexamining the sensors that were available to them, what pa-rameters these sensors could detect and where on the instru-ment they could be easily mounted. Only then would theythink about the gestures that the sensors could be used for.

Such a focus on the technology is a somewhat interestingfinding. We had expected that musicians would concentrateon the gesture and the sound, rather than the technology. Itseems that the musicians consider the sensor technology tobe the weakest link in the system and so allow themselvesto be guided by the limitations of the sensors. While mu-sicians’ creativity can often thrive off such boundaries andlimitations [10], if we wish to develop a system that trulyfocuses on the gestures and sounds then we must alter theusers’ perception of such limitations of the sensors.

Potential for exploration and masteryWessel and Wright state that a goal for designing new dig-ital musical instruments should be for them to have a “lowentry fee” together with “no ceiling on virtuosity” [11]. Thismeans that such instruments should be simple to begin play-ing, but complex and engaging enough to offer the possibil-ity of exploration and mastery.

One of the advantages of augmented instruments is that theyare based around existing musical instruments. A guitaristwill still be able to play an augmented guitar as though itis a regular guitar. This makes the instrument easy to beginusing. The additional sensors then extend the performancepossibilities of the instrument, thus allowing for more poten-tial for creative exploration.

When speaking with the musicians who worked with the sys-tem, we found they commonly expressed the belief that theycould gain full control of the instrument and make best use ofthe system given enough time. This shows that the Augmen-talist offers potential for further exploration and creativity.

Transferability of developed instrumentsWe have already mentioned that one musician developed amapping for their instrument that directly mirrored a systemdiscussed on the literature. This indicates that there is somecommon pool of gestures that are suggested by the designof the instrument itself. As such, we would also expect thatit is possible to transfer developed instruments between per-formers that play the same instrument.

To examine this we asked another guitarist to try and per-form using the mappings developed by one of the guitaristsworking with our system. The new guitarist found it easy

251

to begin performing using any of these mappings. In eachcase, it took only a few minutes of practice before they wereable to utilise the gestures in performance. The guitarist alsomade a number of comments on how “easy” and “natural”the mappings were to use.

It is possible therefore to transfer mappings between per-formers of the same instrument. This means that the poten-tial exists to share instrument designs and mappings acrossusers. One possibility would be the creation of a communityto promote such sharing between musicians interested in de-veloping augmented instruments. We are now beginning toinvestigate this possibility.

Creativity and enjoymentOne of the main aims of this project was to produce a systemthat facilitated both creativity and enjoyment for the musi-cians using it. As has already been discussed, the musicianswho used the system expressed the belief that the Augmen-talist allows for new musical possibilities and offers muchpotential for further exploration and mastery. The questionthat then arises is: did the musicians enjoy using the system?

Throughout the development of the Augmentalist there wereregular testing, development and performance sessions in-volving musicians. At the end of each session, we ask eachmusicians to fill out a short questionnaire, which involvedrating the system on a 1-to-5 scale on a number of criteria.Perhaps the most interesting result of this was that every mu-sician gave the system a maximum rating of 5 (Very High)for enjoyment at the end of every session.

While these ratings show that musicians definitely enjoyedworking with the system, we think that the following quote,received from one participant several hours after a session,fully illustrates the level of enjoyment felt by those using thesystem:

“I haven’t stopped smiling for ages, that was *******awesome. When can I come back?”

CONCLUSIONThe main goal of the Augmentalist system was to enablemusicians to begin experimenting with digital musical in-struments through augmenting their existing musical instru-ments. Our belief was that by focusing on existing instru-ments and augmenting them with sensors, musicians couldproduce new instruments with extended interaction and per-formance possibilities. Such instruments would also havethe advantage of reducing the performer-instrument and au-dience instrument disconnect that can be present with manynew digital musical instruments.

The Augmentalist project has met all of our expectations.From working with musicians we have developed a system

that they find easy to use, that offers new performance poten-tial and that provides an enjoyable creative experience. TheAugmentalist allows musicians to explore new musical tech-niques, while also allowing them to design and create theirown instruments. This opens up new avenues for researchinto the longer term development of these instruments bymusicians, the possibility of sharing and exchanging ideasand mappings for such instruments and the innovative per-formance and interaction techniques that musicians developas part of this process.

REFERENCES1. Reactable. http://www.reactable.com.

2. N. Armstrong. An Enactive Approach to DigitalMusical Instrument Design. PhD thesis, PrincetonUniversity, Nov. 2006.

3. N. Bouillot, Z. Settel, J. R. Cooperstock, andM. Wozniewski. A mobile wireless augmented guitar.In Proc. NIME 2007, 2007.

4. S. Greenberg and C. Fitchett. Phidgets: easydevelopment of physical interfaces through physicalwidgets. In Proc. UIST 2001, pages 209–218. ACM,2001.

5. O. Lahdeoja, M. M. Wanderley, and J. Malloch.Instrument augmentation using ancillary gestures forsubtle sonic effects. In Proc. SMC 2009, pages327–330, 2009.

6. M. T. Marshall, M. Hartshorn, M. M. Wanderley, andD. J. Levitin. Sensor choice for parameter modulationsin digital musical instruments: Empirical evidencefrom pitch modulation. Journal of New MusicResearch, 38(3):241–253, 2009.

7. M. T. Marshall, J. Malloch, and M. M. Wanderley.Non-conscious control of sound spatialization. In Proc.ENACTIVE 2007, pages 377–380, 2007.

8. E. R. Miranda and M. M. Wanderley. New DigitalMusical Instruments: Control and Interaction beyondthe Keyboard. A-R Editions, 2006.

9. C. Palacio-Quintin. The hyper-flute. In Proc. NIME2003, pages 206–207, 2003.

10. A. Tanaka. Musical performance practice onsensor-based instruments. In M. M. Wanderley andM. Battier, editors, Trends in Gestural Control ofMusic, pages 389–405. IRCAM - Centre Pompidou,2000.

11. D. Wessel and M. Wright. Problems and prospects forintimate musical control of computers. ComputerMusic Journal, 26(3):11–22, 2002.

252