Representation-Plurality in Multi-Touch Mobile VisualProgramming for Music

Qi YangComputer Science & Engineering Division

University of Michigan2260 Hayward Ave

Ann Arbor, MI 48109-2121yangqi@umich.edu

Georg EsslElectrical Engineering & Computer Science and

MusicUniversity of Michigan

2260 Hayward AveAnn Arbor, MI 48109-2121

gessl@eecs.umich.edu

ABSTRACTMulti-touch mobile devices provide a fresh paradigm forinteractions, as well as a platform for building rich musi-cal applications. This paper presents a multi-touch mobileprogramming environment that supports the exploration ofdifferent representations in visual programming for musicand audio interfaces. Using a common flow-based visualprogramming vocabulary, we implemented a system basedon the urMus platform that explores three types of touch-based interaction representations: a text-based menu rep-resentation, a graphical icon-based representation, and anovel multi-touch gesture-based representation. We illus-trated their use on interface design for musical controllers.

Author KeywordsNIME, proceedings, Interaction design and software tools,Mobile music technology and performance paradigms, Mu-sical human-computer interaction

ACM ClassificationD.1.7 [Software] Visual Programming, H.5.5 [InformationSystems] Sound and Music Computing, H.5.2 [InformationSystems] User Interfaces — Interaction Styles

1. INTRODUCTIONThe growing popularity and ubiquity of personal mobile de-vices enabled creative touch-based interactions for a wideaudience. As the sensing and computational capacity ofthese devices become sufficient for rich, creative musical ap-plications, many efforts also seek to make audio and musicprogramming accessible on mobile (e.g. [5, 19]).

A variety of approaches to mobile music programmingexists, some uses text-based programming on PC and webwhich are then distributed to device (e.g. [17, 21]). Multi-touch interface on mobile is different in many aspects fromtraditional mouse and keyboard input. For example, handocclusion and lower pointing accuracy of finger requireslarger tap targets and other adaptations [8, 14]. Some ap-proach this by adapting text-based and visual block-basedprogramming [19] for touch, borrowing from extensive works

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.NIME’15, May 31-June 3, 2015, Louisiana State Univ., Baton Rouge, LA.Copyright remains with the author(s).

Figure 1: (left) Touch-based piano keyboard inter-face and (right) music mixer interface built with ourenvironment.

on visual programming for PCs. We present an on-devicesystem that is based solely on visual programming.

Eaglestone et al. [4] found that electroacoustic musicians’preference for musical tools correlates highly with varyingcognitive styles, suggesting that no single interaction stylein musical programming is best-suited for everyone. Weexplored this design space of visual representation on mo-bile music programming by building an environment wheresimple dynamic interfaces can be interactively assembled,and implemented three different visualization and interac-tion paradigm using the same underlying visual language.

2. RELATED WORKSThe design of our system draws from previous work in vi-sual programming, mobile music programming, and visual-ization for multi-touch.

2.1 Visual ProgrammingGeneral purpose visual programming has being explored ex-tensively ever since graphical interface became prevalent onPCs, often with the aim of making programming accessibleto a wider audience. Visual programming are popular inthe interactive media domain, often using a data-flow visualmetaphor such as Pure Data [16] or Max/MSP. The visualaspect of programming is important in live coding, whereprogramming-as-performance is closely coupled with audio-visual media. Some live coding languages are designed witha predominantly visual component [2, 13], since the per-formances usually rely on exposing and conveying the pro-gramming process to the audience.

The design of our visual environment draws from a num-ber of previous visual programming representations and vo-cabularies, from data-flow to interactive or live program-ming (there is no distinction between editing and runningof the program in our system).

2.2 Mobile Music ProgrammingGeneral mobile programming environments such as Hop-scotch 1 adapts a block-based visual metaphor. Others useblock or scripting languages on mobile with a mixed text,iconic based interface (e.g. LiveCode2, [22, 12]). Closest toour approach in mobile programming is Pong Designer [11],which combines a 2D physics engine with directly manipu-latable objects in a sandbox. Programming logic and causalrelations can be inferred by events to build simple games.

The common graphical representation of our system alsouses a visual sandbox environment for assembling elements,eschewing text-based coding. Unlike most environmentswhere a program is built and then ran, our system is fullyinteractive with no distinction between assembling a pro-gram versus running one. For now our environment is notgeneral purpose but enables construction of simple musicinterfaces.

In music and audio domain, programming for mobile de-vices have received growing attention in recent years. Mul-tiple approaches use text-based or visual language for audioand interface programming and distribute it remotely tomobile devices, using frameworks such as Max/MSP [17].urMus provides self-contained on-device text-based program-ming framework accessible through web browser [5]. Whilenot designed for building graphical music interfaces, ChucKaudio programming environment has also been adapted fortouch interface with some block-based visual elements [19].Our work seeks to provide a platform on which musicalinterfaces can be created directly on device without text-based programming.

2.3 Visualization and Multi-Touch GesturesVisual feedback is an important part of multi-touch inter-faces. For example visual feedback is found to lead to sig-nificant improvement on accuracy of pointing/crossing taskson touch-screens [10]. One main criticism by Donald Nor-man on gestural interfaces in consumer software is the lackof feedback to guide learning as well as execution of gesturecommands [15].

There has been many approaches to address this usingvisual feedback for multi-touch gesture interactions. Manyhave focused the ease of recognition and recall of multi-touch gestures, but mapped arbitrarily to commands [18,6]. Using simulated physical objects for visual affordancehas also being explored [3]. Many works used continuousvisualization to show possible commands given the currentstate of interaction [1, 9, 20], showing potential gestureseither at the site of the interaction (near the finger touchlocation), or mirrored at a more visible location.

We explored three different visual representations in oursystem. The menu mode uses only single tap gestures. Theicon and gesture mode make use of more complex drag se-lection or drag-and-drop gestures, and use different visualsto guide the gesture interactions continuously.

3. REPRESENTATION DESIGNOur system is built on the urMus [5] platform. urMus pro-vides a set of Lua API for mobile phone sensors and audioand graphics programming, using which a block-based drag-n-drop interface can be implemented. For us, urMus servesas a general purpose platform for prototyping representa-tions and interactions.

1http://www.gethopscotch.com2http://livecode.com

3.1 Shared GrammarThe basic grammar of the visual environment starts witha full-screen canvas where user can create basic elementscalled regions and arrange them spatially, as well as otherelements that are discussed below. This is where a user cre-ates a dynamic interface (see Figure 1 for example). Next wepresent the general programming and interaction elementsthat will be shared between all representations.Region: The most basic building block is a Region (Fig-ure 2 (a)). Each region is visually represented by a rectangleon screen and can be directly manipulated via dragging andresized via pinching gesture. Regions can be created witha simple tap gesture on the canvas, and can be arbitrarilyarranged on canvas. Their visual appearance can also bemodified, and they can be pinned to the canvas to preventmovement.

Regions possess characteristics of a generic variable orobject in the context of programming, in that they can beused both as abstract containers for values such as a vari-able, and as a visual object in the interface. All regionsand can send and receive events (similar to function calls).Regions can be configured to have other functionalities aswell, such as playing an audio sample (which may be usefulfor building musical instruments) or moving its position onscreen. Similar to how class can be instantiated as manytimes as needed, regions can be duplicated while retainingits properties and links (which defines how it interacts withother regions). In all representation modes regions are cre-ated by tapping on the empty canvas, and its appearance ischanged using a common texture picker interface (Figure 3).Links: Following the concepts of node and edges in event-driven data-flow languages, each region can receive eventsthrough links and respond to them with actions. The rout-ing of these events between regions constitute the mainmechanism of building interactions in the visual environ-ment. Links are visually shown as lines connecting one re-gion to another, similar to visual patching interfaces. Eachlink is directional and stores an event type that is generatedfrom the sender of the link and an action that is to be takenby the receiver when the event is detected.

In the example in Figure 2 (c), a region on the left islinked to send its vertical position value to the region onthe right (which displays a numerical value), acting as avertical slider.

Figure 3: Interface forchanging a region’s tex-ture

This mechanism canbe used for buildingmore complex interac-tions, a region can re-spond to multiple eventsfrom multiple sources, aswell as respond to eachevent with multiple ac-tions. Events can in-clude touch-based events(fired when the regionis dragged, tapped, orheld etc.) and condi-tional on its properties.The actions that can betriggered by events can

changing the spatial properties of the region (size, positionor movement), or passing the event to another region. Whena region is duplicated, all the incoming and outgoing linksare also duplicated, preserving its interaction between otherregions.

We implemented a basic set of touch-based events andactions. The move event which are evoked when the regionis dragged and sends the current position of the region; this

Region RegionRegion

Group regionRegion

Links

(a) (b) (c)

0.54

Figure 2: (a) Regions and links (b) groups, (c) an example slider

event can be responded by move action which move the re-ceiving region in the same direction and distance, or displayone of the coordinates of the received position and send itto a music synthesizer.Grouping: We use groups to encapsulate and organize setof regions spatially. A group region spatially constrains themovement of its child regions, so they cannot be moved outof the group. It can be used to create common interface wid-gets such as slider (see Figure 1 and Figure 2 for examples).Because each group is also treated as another region, it canreceive and respond to events, as well as being duplicated.

3.2 Visual Representation ModesGiven the above shared mechanisms we implemented threetypes of representations for manipulating and interactingwith them: (1) Menu-driven, (2) Icon-driven, and(3) Gesture-driven.Menu-Driven Mode:

Figure 4: The text-based contextual menu

We created a menu repre-sentation mode which usestext menu that has simi-larity to traditional desk-top UI. Except for regionand link deletions, andthe shared gestures men-tioned above, every othercommand such as link-ing or grouping are acti-vated through a contex-tual text-based menu (Fig-

ure 4) that is associated with one region. Deletion buttonsshown for each region whenever the menu is activated by asingle tap.

For commands that require a second region (for example,creating a link between two regions), the user activates thecommand and then is prompted to select the second region(region to create link to, or in case of creating a group,the region to be used as the container) by tapping. Afterthe selection the command is executed on the two regions.Creation of groups uses the identical steps.

Due to its widespread use on different platforms, we ex-pect this representation/interaction mode to be most famil-iar to average users who have some prior experience withPCs and commercial touchscreen operating systems such asiOS or Android, where text-based menu or list interface iscommon.Icon-Driven Mode:In icon-driven mode, most commands are accessed througha contextual graphic symbolic menu (Figure 5). The iconsare arranged similar to a radial layout surrounding the re-gion, which borrows from previous work on radial menudesign for touch-screens ([7] for example).

Two types of gesture interactions exist in the icon-driven

Figure 5: Left: Icon mode contextual menu withlabels, right: draggable icon handle for linking

menu. Static buttons can be activate by tapping (for exam-ple, the Delete button on the top left corner), while drag-gable buttons acts like handles for more complex drag anddrop gestures. These drag gestures activates different typesof semantically related commands. For linking, the link iconcan be dragged and dropped on the region to link to, thepotential link is visually shown using a cable-like metaphor(Figure 5). The grouping gesture handle is used as a lassoselector to select other regions to add to the parent groupregion. The copy handle is visually represented as a smallerversion of the region, can be tapped and produces a copyof the parent region. It can also be dragged into any otherarea on screen and produces a copy of the parent region atthat location when released (Figure 5). To visually differen-tiate draggable gesture handles from the static buttons, thedraggable handles are animated periodically when not used,moving slightly away and back to their original position, asvisual affordance suggesting that they can be dragged asopposed to responding only to taps.Gesture-Driven Mode:The last representation mode is designed to be almost en-tirely driven by direct manipulation gestures that act onthe regions. In our approach, gestures are designed for di-rectly manipulating the on screen elements (regions in thiscases). The mapping between gestures and the associatedcommand is not arbitrary, but semantically related to orreinforce the command being triggered.

Figure 6: Gesture modeuses pinch gestures forcreating and deletinglinks.

For linking and un-linking two regions, apinch or stretch ges-ture is used (Figure 6).A pinch gesture wheretwo regions are draggedconcurrently and movedtowards each other isused for linking, whilethe opposite, movingaway from each other isused for unlinking, re-inforcing the concept oflinking and unlinking.For each gesture, the re-

gions have to be moved past a threshold (which is visually

Figure 7: A drag-n-drop gesture is used to add aregion to a group (a), the reverse for removal (b).

shown when reached) for the action to be completed, thisacts as a confirmation for each action to prevent mistakenlyactivating gestures. If the threshold is not crossed then re-gions are automatically restored to their previous positionsand the action is cancelled.

A background colour guide is displayed faintly after theinitiation of the gesture as a visual guide, and continuouslyupdated depending which direction the user is performingthe gesture (pinch or stretch), the colour area correspondingto the potential command is displayed in increased intensitywhile the colour corresponding to the opposite command isfaded (see Figure 6 for the visual guide). The large colourbackground is meant to alleviate the problem of occlusionby the user’s hand.

For grouping regions, a drag and drop gesture is used.When two regions are dragged, and one is released over an-other one with a larger size, the smaller or released region isadded to the group associated with the larger region (Fig-ure 7). To remove a region from a group, the reverse gestureis used: the user simply drags both the region and its parentgroup region, and then move the child region outside of thegroup region and release. The movement restriction of childregions within their group region is temporarily lifted whilethis gesture is performed, and restored after the remove-from-group action is either executed or cancelled. Similarto the linking gesture, visualization provides guidance in en-larging the potential group region or showing a drop-zonefor the removing gesture with background colour.

The region creation gesture of a single tap on the emptycanvas (shared among all three interaction modes) is modi-fied to support copying. While holding onto a source regionto be copied, tapping the empty space on the canvas cre-ates instead a copy of the source region. Since there is noneed to release the hold on the source region, this allows forefficient creation of copies.

To pin a region, a double tap gesture is used to togglebetween restricting or allowing movements (see Figure 8(c)). For region deletion and modifying a region’s texture,a hold-and-slide gesture menu is used. The gesture menu isdisplayed after holding on the region for a short time andwhile no other commands (moving, resizing etc.) are acti-vated, and its two commands are activated by maintainingthe hold gesture while sliding towards the command icon(similar symbol as the icon mode) and releasing the finger.Figure 8(b) shows the visual feedback for activating eachcommands after the hold-and-slide gesture.

Continuous visual feedback is also used for interactionmodes that relies on drag gestures (e.g. pinch, lasso se-lection), in previous works continuous visual feedback hasbeen found to be beneficial in visual programming context[23]. In icon mode, the cable and lasso selection visualiza-tion are continuously animated, and in gesture mode the

Figure 8: (a, b) hold-gesture-based menu and itsdifferent activation states, (c) visual feedback forpinning a region

colour background is continuously updated during the link-ing gesture (Figure 6).

4. SYSTEM IMPLEMENTATIONOur system was implemented on top of the urMus frame-work using Lua scripting language. The system consists ofthe functional language components (which include classesthat represent the basic entities mentioned earlier: Region,Link and Group) and visual interface components that en-ables the interaction and feedback.

In the language components, region is an extension (sub-class) of the built-in visual region class in urMus, as it re-tains appearance properties and drag-n-drop interactions.In our environment, region class also stores interactionstates as part of the gesture state-machine, as well as mech-anism for dispatching and responding to events (those cre-ated by user interaction, and sent from other linked enti-ties), managing links, and managing associated contextualinterfaces for text and icon-based menus.Group is a subclass of the region class, which acts simply

as a container for other region objects, providing methodsfor managing its child regions.Link object is a utility class for connecting two region

objects, and managing the flow of events from object toobject in a single direction (specified sender and receiver).Since group is a subclass of region, they can also be con-nected by links as either a sender. Visual appearance oflink objects on screen is handled centrally by a separateclass which draws and updates all links on screen.

4.1 User InterfaceOther than the visual representation of Regions and Links,the rest of the user interface are managed by text and iconmenu classes, and two utility singleton classes for gesturerecognition and visual gesture guides.

Text menu and icon menu classes are implemented simi-larly using textured urMus regions and text labels. They aredesigned to be reusable and are instantiated and configuredeach time they are needed. Callback methods/functions areused for specifying each commands in the menus.

Gesture interactions are mediated by two singleton classes,gesture manager which handles the actual multi-touch in-teraction state-machine, and gesture guide which drawsthe visual guides for gestures on screen. The multi-touchgesture state-machine recognize each gesture such as pinchor drag-and-drop-onto by listening for low-level touch eventsfrom all regions in the environment, and triggering the as-signed actions (grouping, linking) when action states arereached. The modular design of the gesture manager al-lows a variety of gestures to be swapped in by specifyingdifferent state-machines within the manager class, withoutchanging the region objects which are involved in the actualgesture.Gesture guide is called to update onscreen visual feed-

back by the manager class, allowing the visualization of ges-

tures to be configured separately from the recognition.Other utilities including logging which records each touch

events and user-triggered actions in a detailed log on device,and notification class providing simple onscreen help mes-sages.

In addition to visual interface, a sound module currentlyin development serves as a conduit to some of urMus’s soundsynthesis API, allowing user interaction from the program-ming environment to be fed into sound synthesis parame-ters, enabling the building of musical controllers. In theexample in Figure 1, each slider controls a different param-eter of a simple sine oscillator.

5. CONCLUSIONWe presented a visual programming environment for creat-ing music interfaces. Our system allows a variety of rep-resentation and interaction modes using a basic grammarthat supports the construction of musical controller inter-faces. All paradigms are visual and we explore the potentialof using both familiar and novel gesture-based multi-touchparadigms to construct representations. Ongoing and fu-ture work include evaluation of the different representationswith respect to their usability in musical end-user program-ming. We are currently in the process of conducting andanalyzing a user study comparing the different represen-tations. Furthermore we plan to create an accessible APIhelping musicians embed their preferred multi-touch inter-action paradigm in their own performance interfaces.

6. REFERENCES[1] O. Bau and W. E. Mackay. OctoPocus: A Dynamic

Guide for Learning Gesture-Based Command Sets. Inthe 21st annual ACM symposium, pages 37–46, NewYork, New York, USA, 2008. ACM Press.

[2] A. Blackwell and N. Collins. The ProgrammingLanguage as a Musical Instrument. Proceedings ofPPIG05 (Psychology of Programming Interest Group),pages 120–130, 2005.

[3] A. Bragdon, A. Uguray, and D. Wigdor. Gesture play:motivating online gesture learning with fun, positivereinforcement and physical metaphors. In ITS 2010,pages 39–48, Saarbru ILcken,Germany, 2010.

[4] B. Eaglestone, N. Ford, and P. Holdridge. AreCognitive Styles an Important Factor in Design ofElectroacoustic Music Software? Journal of NewMusic, 2008.

[5] G. Essl and A. Muller. Designing Mobile MusicalInstruments and Environments with urMus. InProceedings of the 2010 conference on New Interfacesfor Musical Expression, Sydney, Australia, 2010.

[6] E. Ghomi, S. Huot, O. Bau, M. Beaudouin-Lafon, andW. E. Mackay. Arpege. In the 2013 ACMinternational conference, pages 209–218, New York,New York, USA, 2013. ACM Press.

[7] D. Kammer, F. Lamack, and R. Groh. Enhancing theexpressiveness of fingers: multi-touch ring menus foreveryday applications. In AmI’10: Proceedings of theFirst international joint conference on Ambientintelligence. Springer-Verlag, Nov. 2010.

[8] S. Lee and S. Zhai. The Performance of Touch ScreenSoft Buttons. In Proceedings of the SIGCHIconference on Human . . . , 2009.

[9] S. Lundgren and M. Hjulstrom. Alchemy : dynamicgesture hinting for mobile devices. In the 15thInternational Academic MindTrek Conference, pages53–60, New York, New York, USA, Sept. 2011. ACM.

[10] Y. Luo, D. Vogel, and Y. Luo. Crossing-basedselection with direct touch input. ACM, New York,New York, USA, Apr. 2014.

[11] M. Mayer and V. Kuncak. Game programming bydemonstration. In Proceedings of the 2013 ACMinternational symposium on New ideas, newparadigms, and reflections on programming &software, pages 75–90. ACM, 2013.

[12] S. McDirmid. Coding at the speed of touch. In the10th SIGPLAN symposium, pages 61–76, New York,New York, USA, Oct. 2011. ACM.

[13] A. McLean, D. Griffiths, N. Collins, and G. Wiggins.Visualisation of live code. In EVA’10: Proceedings ofthe 2010 international conference on ElectronicVisualisation and the Arts. British Computer Society,July 2010.

[14] M. A. Nacenta, P. Baudisch, H. Benko, andA. Wilson. Separability of spatial manipulations inmulti-touch interfaces. In GI ’09: Proceedings ofGraphics Interface 2009. Canadian InformationProcessing Society, May 2009.

[15] D. A. Norman and J. Nielsen. Gestural Interfaces: aStep Backward in Usability. Interactions, 17(5), Sept.2010.

[16] M. Puckette et al. Pure data: another integratedcomputer music environment. Proceedings of theSecond Intercollege Computer Music Concerts, pages37–41, 1996.

[17] C. Roberts, M. Wright, J. Kuchera-Morin, andT. Hollerer. Rapid creation and publication of digitalmusical instruments. In Proceedings of theInternational Conference on New Interfaces forMusical Expression (NIME), 2014.

[18] J. s, E. Lecolinet, and T. Selker. Multi-finger Chordsfor Hand-held Tablets: Recognizable and Memorable.In CHI 2014, 2014.

[19] S. Salazar and G. Wang. miniAudicle for iPad:Touchscreen-based Music Software Programming. InInternational Computer Music Conference, 2014.

[20] R. Sodhi, H. Benko, and A. Wilson. LightGuide:Projected Visualizations for Hand MovementGuidance. In the 2012 ACM annual conference, pages179–188, 2012.

[21] B. Taylor, J. Allison, W. Conlin, and Y. Oh.Simplified Expressive Mobile Development withNexusUI, NexusUp and NexusDrop. In Proceedings ofthe International Conference on New Interfaces forMusical Expression (NIME), 2014.

[22] N. Tillmann, M. Moskal, J. de Halleux, andM. Fahndrich. TouchDevelop: programmingcloud-connected mobile devices via touchscreen. InONWARD ’11: Proceedings of the 10th SIGPLANsymposium on New ideas, new paradigms, andreflections on programming and software, 2011.

[23] E. M. Wilcox, J. W. Atwood, M. M. Burnett, J. J.Cadiz, and C. R. Cook. Does continuous visualfeedback aid debugging in direct-manipulationprogramming systems? In CHI ’97: Proceedings of theACM SIGCHI Conference on Human factors incomputing systems, pages 258–265, 1997.