report version: 1 classiﬁcation: pub. deliverable no. 2.3...

FP6-NEST-PATH project no: 29085Report Version: 1

Report Preparation Date: April 30, 2009Classification: Pub.Deliverable no. 2.3

Closing the Loop of Sound Evaluation and Design(CLOSED)

Deliverable 2.3

CONTROL LAYER

Stefano Delle Monache, Delphine Devallez, Carlo Drioli,Federico Fontana, Stefano Papetti, Pietro Polotti,

Davide Rocchesso (UNIVERONA)

Contents

I. Sound Design Toolkit - User’s Guide 1

1. Low-level models 7

1.1. Solids contact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.1. linpact inertialb modalb∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.1.2. linpact 2modalb∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.1.3. impact inertialb modalb∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.1.4. impact 2modalb∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1.1.5. impact inertialb wg∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.1.6. impact modalb wg∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.1.7. friction 2modalb∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.1.8. friction modalb wg∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

1.2. Liquids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

1.2.1. onebubble∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

2. Higher-level models 35

2.1. Bouncing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.2. Crumpling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.2.1. control crump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.2.2. control contcrump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.3. Rolling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.4. Breaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.5. Boiling, frying, streaming, pouring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.5.1. bubblestream∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.6. Dripping, splashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3

2.6.1. splash∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.7. Particles models sonification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.7.1. particles sim∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.8. Vacuum Cleaner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

3. Acoustic depth 51

3.0.1. mesh2D static∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.0.2. mesh2D dynamic∼ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

3.0.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

GNU Free Documentation License 57

II. Generation and Testing of a Sonic Palette 63

4. Strategies for the Use and Control of the Sound Design Toolkit 65

4.1. Synthesis parameter conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4.1.1. Low level models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4.1.2. High level models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.1.3. Interpolation between different reference sonic states . . . . . . . . . . . . . . . 70

4.2. Control by means of graphic interfaces, mouse and keyboard . . . . . . . . . . . . . . . 72

4.2.1. Usability versus data flow accuracy in Max/MSP environment . . . . . . . . . 72

4.2.2. Availability of multidimensional UI objects . . . . . . . . . . . . . . . . . . . . 74

4.2.3. Management of presets and data interpolation . . . . . . . . . . . . . . . . . . 74

4.2.4. Instantiating and initialisation issues . . . . . . . . . . . . . . . . . . . . . . . . 74

4.3. Physical control issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.3.1. Control of the models through a variety of sensors . . . . . . . . . . . . . . . . 75

4.3.2. Relation between gesture and sound with different kinds of physical mediation 76

5. The Sonic Palette 79

5.1. Low level models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.1.1. Simple contact sounds [EX00 impact.mxb] . . . . . . . . . . . . . . . . . . . . . 79

5.1.2. Continuous contact sounds [EX00 friction.mxb] . . . . . . . . . . . . . . . . . . 83

5.2. High level models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.2.1. Complex contact sound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

4

5.2.2. Liquid sound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

6. Interactive Sonification Examples 93

6.1. GUI-based interactive simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

6.1.1. Spreading of jam on toast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

6.1.2. Drying the salad with the spinner . . . . . . . . . . . . . . . . . . . . . . . . . 93

6.1.3. Shaving with a razor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

6.1.4. Wiping the window with a buff . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

6.1.5. Screwing the cork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

6.2. Gamelunch activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

6.2.1. Basic sonic interaction design approach . . . . . . . . . . . . . . . . . . . . . . 98

6.2.2. Chronology: Gamelunch set up and sensor mapping issues . . . . . . . . . . . . 100

7. Acoustic Depth: Rendering Auditory Distance 107

7.1. The Rectilinear Digital Waveguide Mesh . . . . . . . . . . . . . . . . . . . . . . . . . . 107

7.2. Acoustical Properties of the Membrane . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

7.2.1. Overall Intensity Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

7.2.2. Direct-to-Reverberant Energy Ratio . . . . . . . . . . . . . . . . . . . . . . . . 109

7.3. Externals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

7.3.1. Sampling rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

7.3.2. Parameters of the DWM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

7.3.3. Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

7.4. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

7.4.1. DepThrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

7.4.2. Audio-Haptic Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

7.4.3. A Peculiar DWM Geometry for Linear Auditory Distance Estimates . . . . . . 112

III. A Tool for Sound Designers 115

8. The High Level User Interface 117

8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

8.2. GUI implementation and description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

8.2.1. Navigating the SDT package . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

5

8.2.2. Clarity and intuitiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

8.2.3. User-centered features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

9. Sonic Landmarks for the Exploration of the SDT Timbral Space 123

9.1. The SDT explorer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

9.2. Design and implementation of the Spike-based parameter interpolation tool . . . . . . 123

9.3. The sonic space, sonic landmarks and GUI . . . . . . . . . . . . . . . . . . . . . . . . . 125

9.4. Interpolation examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

References 131

Part I.

Sound Design Toolkit

User’s Guidev0.4.3

1

Introduction

The Physically-based Sound Design Toolkit (SDT from now on) aims at providing perception-oriented and physically-coherent advanced tools for the next generation of sound designers.

The SDT is the main software product of a project activity which begun in 2001 with the EUproject SOb - the Sounding Object [18]. In continuous development, the SDT inherits basic theoreticalconcepts of sound synthesis initiated with the physical analysis of impact sounds, later complementedwith more low-level physically-based sound models (such as friction) and further provided with higherlevel synthesis paradigms, integrating low-level models into more elaborate synthesis contexts (suchas rolling and crumpling).

Such theoretical concepts found early realizations in the form of Pure Data patches and externalsfor Linux and Windows which had to be ad hoc compiled. In SDT all the externals have beenthoroughly revised and rewritten to comply with the multi-OS programming layer flext and to runin the Max/MSP 41 environment. Moreover, the adoption of Max/MSP has allowed to restyle theearly Pure Data patches in order to create more accessible, usable sound synthesis controls and betteronline documentations.

The SDT is maintained and continuously developed by the UNIVERONA (VIPS group) unit of theEU project CLOSED - Closing the Loop Of Sound Evaluation and Design.

The basic (low-level) algorithms, the corresponding externals and some example patches are de-scribed in Chapter 1.

Higher level algorithms, based on the basic ones, are described in Chapter 2. In some cases thehigher level algorithms are just implemented as Max/MSP patches which exploit the basic externals;in others they rely on both low-level and additional externals.

Externals descriptions are easily identifiable by boxed titles .Throughout this manual we follow the Mac OS X (or UNIX) file-system conventions: e.g. paths are

expressed with slashes “/”. Of course Windows users just need to consider back-slashes “\” insteadof slashes.

How to use the SDT

The SDT is split into three main sections (corresponding to subdirectories): source code (under/SDT_sources), Max/MSP patches (under /SDT_patches) and, for the faint hearted, pre-compiledMax/MSP externals for both Mac OS X2 and Windows (under /SDT_binaries).

All main source files are coded in C++ (while some other auxiliary files are coded in C) usingflext3, a layer for cross-platform development of Max/MSP and Pure Data externals. This meansthat, with minimum effort, one can build the SDT on Mac OS X, Windows and Linux platforms, thisway getting externals for Max/MSP or Pure Data. In other words, although this manual is orientedtowards Max/MSP, still sound designers keen on Pure Data can build the very same externals fortheir application of choice. Unfortunately though, the SDT only include patches for Max/MSP.

1Compatibility with Max 5 has not been tested yet.2Mac OS X externals are in universal binary format, that is they are compatible with both PPC and Intel Macs3http://grrrr.org/ext/flext/

3

http://grrrr.org/ext/flext/

What is needed

Despite the provided Max/MSP externals being ready for use, dauntless users would surely liketo build their own ones. Other users may want to modify some of the provided externals, thereforethey would need to alter the corresponding source code, and then build their updated externals fromscratch.

For all these advanced users the first thing to do is to download and install the “Max/MSP SoftwareDevelopment Kit” (SDK) from http://www.cycling74.com/downloads/max4.

Secondly, it is necessary to download and install flext. At the time of writing, the version of flextrequired to correctly build the SDT is only available as SVN check-out (starting with the currentv0.5.1, any bleeding edge version should fit).

Below are all the informations needed in order to configure any SVN client and then being able tocheck-out the SVN version of flext:

Hostname: pure-data.svn.sourceforge.net

Port: 443

Protocol: HTTPS

Repository Path: /svnroot/pure-data/trunk/externals/grill/flext

In order to actually “check-out” (equivalent to “download”, in SVN language) the SVN version offlext just go to your command shell and write4:

svn co https://pure-data.svn.sourceforge.net/svnroot/pure-data/ ...... /trunk/externals/grill/flext flext

where the dots ... are used for display purpose only (i.e., the command must be entered on one line).Once the check-out is completed, you can find a fresh flext distribution inside the local directory

/flext.Notice that at this point it is still necessary to configure, build and install flext on your machine.

Please refer to the files readme.txt and build.txt inside your flext distribution for detailed instruc-tions on how to do that.

Building the SDT

All source files lie inside the directory /SDT_sources (under /Solids and /Liquids).To each external corresponds a .txt file containing instruction for the compiler and being used

by flext own build system. In flext-based distributions, the default name for this kind of file ispackage.txt so that, when invoking the build batch-file, you don’t have to specify its name. However,since SDT is made up of many externals which often share the same source files, we opted for putting allshared sources together, providing a EXTERNAL_NAME.txt file for each external. Hence, when invokingflext build batch-file it is necessary to add the macro ‘‘PKGINFO=EXTERNAL_NAME.txt’’ (includingquotation marks) at the end of your sequence of instructions.

Hint 1: In order to refer to them more quickly, you can rename each package-type .txt file as youwish, e.g. with progressive numbers: 1.txt, 2.txt, etc..

Hint 2: The easiest way to compile the externals is to copy the whole directory /SDT_sources underthe directory /flext of your flext distribution. Then it will be straightforward to invoke flextbuild batch-file as:

sh ../build.sh

4There are also very convenient GUI-based SVN clients like http://tortoisesvn.net/ for Windows.

4

http://www.cycling74.com/downloads/max4

http://tortoisesvn.net/

from bash shell, or

..\build

from DOS command prompt.

Example 1: To build the external impact 2modalb∼ with gcc under Mac OS X, just open the terminal,go inside /SDT_sources and write:

sh ../build.sh max gcc "PKGINFO=impact_2modalb~.txt"

or change ../build.sh accordingly to the specific path of your flext distribution. Then you’llfind the compiled external within the directory /max-darwin under /SDT_sources.

Example 2: To build the external onebubble∼ with Microsoft Visual Studio under Windows, openthe Visual Studio Command Prompt, go inside \SDT_sources and write:

..\build max msvc "PKGINFO=onebubble~.txt’’

or change ..\build accordingly to the specific path of your flext distribution. Then you’ll findthe compiled external within the directory \max-msvc under \SDT_sources

Installing the SDT

In order to install the SDT follow the instructions below:

1. Copy all the externals for your OS into Max/MSP’s /externals directory or, provided that youupdate accordingly Max/MSP’s preferences, into your directory of choice.

2. Copy all .help files found under /SDT_patches and subdirectories into Max/MSP /max-helpdirectory.

3. Start using the SDT by double-clicking on any .mxb example patch under /SDT_patches andhave fun!

5

1. Low-level models

Basic models for solids-contact and liquids sound events are presented. After some theory, externalswhich implement the models and some example patches are described.

We will see in Chapter 2 how these basic models serve as a basis for more complex sound events,textures and processes.

1.1. Solids contact

The models considered here apply to basic contact events between two solid objects. As the mostrelevant contact sound events in everyday life come down to impacts and frictions, the providedexternals model these two kinds of interactions.

The algorithms implemented here share a common structure: two solid object models interactthrough (what we call) an interactor (see Fig. 1.1).

Figure 1.1: The common structure underlying solid contact algorithms.

Contact models

An interactor represents a contact model or, so to say, the “thing” between the two interactingobjects. As for the impact model, it can be seen as the “felt” between the striking object and thestruck object, while in the friction model it simulates friction as if the surfaces of the two rubbingobjects would be covered with “micro-bristles”.

Two impact models (one of which is a simplified version) and a friction model are provided:

impact interactor - implements a non-linear impact force. It receives the total compression (thedifference of displacements of the two interacting objects at interaction point) and returns thecomputed impact force.The latter is made of the sum of an elastic component and a dissipative one. The elastic

7

component is parameterized by the force stiffness (or elasticity) and a non-linear exponentwhich depends on the local geometry around the contact area. The dissipative componentis parameterized by the force dissipation (or damping weight). For further details refer to [18].

simplified impact interactor - implements a linear impact force. As it is a linearized version of theimpact interactor described above, the non-linear exponent is not present (or, equivalently, isequal to 1), while the force stiffness and the force dissipation still remain.

friction interactor - implements a non-linear friction force. It receives the relative velocity of the tworubbing objects and returns the computed friction force.The latter is made of the sum of four components, each of them corresponding to one coefficient.These are: an elasticity coefficient, an internal dissipation coefficient, a viscosity coefficient, andfinally the gain of a pseudo-random function (noise related to surface roughness). The modelis parametrized by several others quantities: the dynamic friction and the static friction coeffi-cients, a break-away coefficient and the Stribeck velocity (both of them relate to the transient).For further details refer to [18].

Object models

Three distinct object models are provided:

modal object - in the modal description, a resonating object is described as a system of a finitenumber of parallel mass-spring-damper structures. Each mass-spring-damper structure modelsa mechanical oscillator which represents a normal mode of resonance of the object. The oscillationperiod, the mass and the damping coefficient of each oscillator correspond respectively to theresonance frequency, the gain and the decay time of each mode.In our implementation it is possible to choose the wanted number of modes and to separatelycontrol their properties. Furthermore, each modal object has a number of pickup points, fromwhich the sound is output. There must be at least one pickup point but they must be less thanthe number of modes. The first pickup point is also where the contact takes place (interactionpoint).

inertial object - simulates a simple inertial point mass. Obviously this kind of objects is useful solelyas an exciter for other resonators.The only settable object property is its mass.

waveguide object - the digital waveguide technique [22] models the propagation of waves along elasticmedia (or air pressure inside a tube). In the one-dimensional case implemented here [14], thewaveguide object models an ideal elastic string.In our implementation it is possible to set length, tension and mass of the string. Further, onecan set the position of the interaction point along its length. The interaction point coincideswith the only available pickup point (i.e. the place from where the sound is output).

Generalities

Having a look at Fig. 1.1, the way two objects interact through an interactor appears evident: ateach discrete time instant (sample) both objects send their internal states (displacement and velocityat the interaction point) to the interactor, which in turn sends the newly computed (opposite) forcesto the objects. Knowing the new applied forces, the objects are able to compute their new states forthe next time instant. In other words, there’s a feedback communication between the three models.

The SDT framework differs remarkably from the approach to physically-based sound synthesisfound in most existing implementations and literature. Being this not the seat for an exhaustive andanalytical dissertation and comparison, of interest is the IRCAM software Modalys [11], a powerfulphysical modeling interactive tool for musical applications, based on modal synthesis.

8

The Modalys working space is characterized by a modular set of modal objects, such as tubes,membranes, strings, “two-mass” objects and hybrids; types of linear connections, that is types ofinteractions between objects; controllers, that specify exactly how the connections will be executed;“accesses”, that set the physical location of the connection on a Modalys object in order to interactwith other objects.

Recently, Modalys developments included the possibility to create three dimensional meshes andto compute their modes by finite elements numerical methods.

Available classes of modal objects include the following opcodes: “bi-string”, a string or rod whosevibration moves in two transverse directions; “cello-bridge”; “circ-membrane”, a circular membranewith zero thickness; “clamped-circ-membrane”, a circular plate fixed at its edges; “closed-closed-tube”, an acoustic tube sealed at both ends; “closed-open-tube”, an acoustic tube sealed at one endand opened at the other; “melt-hybrid” and “mix-hybrid” that create a hybrid of two different objects.The latter can be seen as a box with the two objects inside, with a sound mix of the two objects, whoseexcitation transmits energy to the sub-objects in proportion to the current position of the hybrid.

Connections-interactions can be modularly set on each object. Among the others, the relativeopcodes includes: “adhere”, adherence between two accesses; “bi-fingerboard”, that simulates theinteraction between a finger and a string with a fingerboard underneath; “bow”, a two dimensionalconnection between four accesses; “hole”, that makes a hole of variable diameter in an acoustic tube;“pluck”, where one access plucks another one; “position”, for changing the access position along itsaxis. At last, controllers include noise generators, break-point envelope functions, filters, oscillatorsand arithmetical operators.

It can be noticed how types of objects and interactions already form parametrically high levelsubclasses to be connected in order to achieve more complex virtual musical instruments.

As far as the interaction structures are concerned, a main difference between the SDT packageand Modalys resides in the feed-forward structure of the latter: in Modalys a form exciter ↔resonator is adopted, where non-linearities are concentrated in the interaction part of a structure(e.g. blow, bow, etc.).

Bearing in mind the musical purposes conception of Modalys, elements like key noise for an acoustictube are sometimes simulated by means of a usual random generator, rather than actually beingcalculated as a physical result of other stimuli.

On the contrary, the SDT package takes advantage of a cartoonified approach in sound design andimplements a feedback network within the interaction object1 ↔ interactor ↔ object2, with non-linear characteristics of the interactor. This allows the accurate modeling of complex interactions (e.gfriction) and to output the sound of both the interacting objects. Besides, the continuous feedbackapproach adopted into the SDT is memory consistent, that is the system takes record of each previousstate, during the interaction and manipulation. Secondly, the SDT package implements low levelmodels that are already capable of all possible issues and interactions, with an open and wide varietyof sound possibilities, and a consistent physical behavior.

Externals conventions

According to the described framework, the naming of each external conforms to the general forminteractor object1 object2∼.

The linear impact interactor is referred as linpact, the non-linear impact interactor as impact, whilethe friction interactor is referred as friction.

The modal and the inertial objects are respectively referred as modalb and inertialb1, while thewaveguide object is referred as wg. The only naming exception happens when two modal objectsinteract, in which case the whole object1 object2 section becomes 2modalb.

1The ending b is just a reminder for developers which tells that the bilinear transformation has been used in order todiscretize the objects’ continuous-time equations.

9

Notice that object1 is always either a modal or an inertial object. This is because those are theonly objects which are able to move: as already said, an inertial object behaves as a point mass, whilethe first mode of a modal object1 is its inertial mode, that is the whole object behaves as a mass-spring-damper structure. In both cases it is therefore possible to make object1 move (e.g. imposingan external force or an initial velocity on it) so that it starts interacting with object2.

As modal objects have a variable number of modes and pickup points, when an external involvesa modal object the arguments are of variable length and composition. Moreover, each pickup pointcorresponds to a signal outlet, therefore outlets of such externals are also in variable number.

In the following externals descriptions, elements which are in variable number are underlined. Ifapplicable, each argument type is followed by the name of the element (interactor, object1, object2) itrefers to.

Help patch conventions

The best way to understand how the supplied externals work is to have a look to their help patches.Just be sure to put them under Max/MSP /max-help directory to have access to them when invokingthe help.

Figure 1.2: Solids contact example (help) patch

All help patches share a common structure (cfr. Fig. 1.2): the control panels on the left andright sides (orange backgrounds) are respectively for object1 and object2, those in the center (greenbackground) are for the interactor.

Control panels for object1 only operate on a subset of its own properties (basically for screen space’ssake), yet they allow to trigger the contact with object2. Conversely, the control panels provided forthe interactor and object2 operate on the whole set of their available physical-geometric properties.

10

1.1.1. linpact inertialb modalb∼

Linear impact between one inertial and one modal object.

Input

signal object1 1st inlet. External force applied to object1.

signal object2 2nd inlet. External force applied to object2.

signal 3rd inlet. Additional displacement offset between the objects.

float object1 4th inlet. Initial strike velocity.

list: [3×float] 5th inlet. The first two arguments are respectively the interactor’sforce stiffness and damping coefficient.The third is the mass of object1.

list: [3×float] object2 6th inlet. Base (global) factors. These multiply respectively: thefrequency, the decay time and the gain of all modes.

list: [float] object2 7th inlet. As many arguments as the number of modes.Frequency of each mode.

list: [float] object2 8th inlet. As many arguments as the number of modes.Decay time of each mode.

list: [int float] object2 9th inlet. The int value indicates the considered pickup point. Thesubsequent list of float values sets the gain of each mode at thespecified pickup point.

strike object1 1st inlet. Followed by a float value, sets the initial strike velocity.

nteract strikr 1st inlet. Followed by three float values. The first two set respec-tively the interactor’s force stiffness and damping coefficient. Thethird sets the mass of object1.

actmodes2 object2 1st inlet. Followed by one int value which sets the number ofcurrently active modes. Obviously the maximum number of activemodes must be lower than the total number of available modes.

base2 object2 1st inlet. Followed by three float values correspondent to the base(global) factors. These multiply respectively: the frequency, thedecay time and the gain of all modes.

mode freqs2 object2 1st inlet. Followed by as many float values as the number of modeswhich set each mode frequency.Notice that for this message to have effect you need to refresh thebase message.

11

mode ts2 object2 1st inlet. Followed by as many float values as the number of modeswhich set their decay times.Notice that for this message to have effect you need to refresh thebase message.

mode contribs2 object2 1st inlet. Followed by one int value which specifies a pickup point,and as many float values as the number of modes. These set thegain (0-100 on a logarithmic scale) of each mode at the specifiedpickup point.

Output

All outlets come from pickup points of object2, in progressive order from left to right.

signal object2 As many signal outlets as the number of pickup points.Depending on the chosen output mask (see the Arguments sectionbelow), they output either the object’s velocity or displacement.

Arguments

All arguments are mandatory. They initialize the two objects and the interactor with their physical-geometric properties. In progressive order they are:

list: [3×float] The first two arguments are respectively the interactor’s force stiff-ness and damping coefficient.The third is the mass of object1.

int object2 Number of modes.

int object2 Number of pickup points.

symbol object2 As many arguments as the number of pickup points.Mask for the output of each pickup point: ’1’ sets the pickup pointto output the object’s velocity, while anything else (e.g. ’d’) setsit to output the object’s displacement.

list: [3×float] object2 Base (global) factors. These multiply respectively: the frequency,the decay time and the gain of all modes.

list: [float] object2 As many arguments as the number of modes.Frequency of each mode.

list: [float] object2 As many arguments as the number of modes.Decay time of each mode.

list: [int float] object2 As many lists as the number of pickup points. The length of eachlist is equal to the number of modes +1.The int value indicates the considered pickup point. The subse-quent list of float values sets the gain of each mode at the specifiedpickup point.

12

1.1.2. linpact 2modalb∼

Linear impact between two modal objects.

Input





list: [2×float] interactor 5th inlet. The two arguments are respectively the force stiffnessand the damping coefficient.





list: [3×float] object2 10th inlet. Same as for the 6th inlet.

list: [float] object2 11th inlet. Same as for the 7th inlet.


list: [int float] object2 13th inlet. Same as for the 9th inlet.


nteract interactor 1st inlet. Followed by two float values which set respectively theinteractor’s force stiffness and damping coefficient.



13




actmodes2 object2 1st inlet. Same as actmodes1.

base2 object2 1st inlet. Same as base1.

mode freqs2 object2 1st inlet. Same as mode freqs1.

mode ts2 object2 1st inlet. Same as mode ts1.

mode contribs2 object2 1st inlet. Same as mode contribs1.

Output

Outlets come from pickup points of both object1 and object2 (starting from left).For example, considering that object1 has one pickup and object2 has three, the first outlet would

come from pickup0 of object1 and the subsequent ones respectively from pickup0, pickup1 and pickup2of object2.

signal object1 As many signal outlets as the number of pickup points of object1.Depending on the chosen output mask (see the Arguments sectionbelow), they output either the object’s velocity or displacement.

signal object2 Same as for object1.

Arguments


list: [2×float] interactor The two arguments are respectively the force stiffness and thedamping coefficient.



14








list: [int float] object1 As many lists as the number of pickup points. The length of eachlist is equal to the number of modes +1.The int values indicates the considered pickup point. The subse-quent list of float values sets the gain of each mode at the specifiedpickup point.





15

1.1.3. impact inertialb modalb∼

Non-linear impact between one inertial and one modal object.

Input





list: [4×float] 5th inlet. The first three arguments are for the interactor. Theyare respectively the force stiffness, a parameter which depends onthe contact-surface’s shape, and the dissipation coefficient.The fourth is the mass of object1.






nteract strikr 1st inlet. Followed by four float values. The first three set respec-tively the interactor’s force stiffness, a parameter which dependson the contact-surface’s shape, and the dissipation coefficient.The fourth sets the mass of object1.




16



Output

All outlets come from pickup points of object2, in progressive order from left to right.

signal object2 As many signal outlets as the number of pickup points.Depending on the chosen output mask (see the Arguments sectionbelow), they output either the object’s velocity or displacement.

Arguments


list: [4×float] The first three arguments are for the interactor. They are respec-tively the force stiffness, a parameter which depends on the contactsurface’s shape, and the dissipation coefficient.The fourth is the mass of object1.








17

1.1.4. impact 2modalb∼

Non-linear impact between two modal objects.

Input





list: [3×float] interactor 5th inlet. The three arguments are respectively the force stiffness,a parameter which depends on the contact-surface’s shape, andthe dissipation coefficient.










nteract interactor 1st inlet. Followed by three float values which set respectively theforce stiffness, a parameter which depends on the contact-surface’sshape, and the dissipation coefficient.



18









Output





Arguments


list: [3×float] interactor The three arguments are respectively the force stiffness, a param-eter which depends on the contact-surface’s shape, and the dissi-pation coefficient.



19













20

1.1.5. impact inertialb wg∼

Non-linear impact between one inertial and one waveguide object.

Input





list: [4×float] 5th inlet. The first three arguments are for the interactor. Theyare respectively the force stiffness, a parameter which depends onthe contact-surface’s shape, and the dissipation coefficient.The fourth is the mass of object1.

float object2 6th inlet. Normalized (0-1) contact position along string’s length.

float object2 7th inlet. String’s length in meters.

float object2 8th inlet. String’s tension in Newtons.

float object2 9th inlet. String’s mass in kilograms.


nteract strikr 1st inlet. Followed by four float values. The first three set respec-tively the interactor’s force stiffness, a parameter which dependson the contact-surface’s shape, and the dissipation coefficient.The fourth sets the mass of object1.

contact pos2 object2 1st inlet. Followed by a float value, sets the normalized (0-1)contact position along the string.

str length2 object2 1st inlet. Followed by a float value, sets the string’s length inmeters.

str tension2 object2 1st inlet. Followed by a float value, sets the string’s tension inNewtons.

str mass2 object2 1st inlet. Followed by a float value, sets the string’s mass in kilo-grams.

Output

The outlet comes from the only pickup point of object2.

21

signal object2 Depending on the chosen output mask (see the Arguments sectionbelow), the outlet outputs either the object’s velocity or displace-ment at the contact position.

Arguments


list: [4×float] The first three arguments are for the interactor. They are respec-tively the force stiffness, a parameter which depends on the contactsurface’s shape, and the dissipation coefficient.The fourth is the mass of object1.

float object2 Normalized (0-1) contact position along string’s length.

float object2 String’s length in meters.

float object2 String’s tension in Newtons.

float object2 String’s mass in kilograms.

symbol object2 Mask for the output of pickup point: ’1’ sets the pickup point tooutput the object’s velocity, while anything else (e.g. ’d’) sets itto output the object’s displacement.

22

1.1.6. impact modalb wg∼

Non-linear impact between one modal and one waveguide object.

Input





list: [3×float] interactor 5th inlet. The three arguments are respectively the force stiffness,a parameter which depends on the contact-surface’s shape, andthe dissipation coefficient.










nteract interactor 1st inlet. Followed by three float values which set respectively theforce stiffness, a parameter which depends on the contact-surface’sshape, and the dissipation coefficient.



23








Output

Outlets come from pickup points of both object1 and object2 (starting from left). The rightmostone comes from the only pickup point of object2.


signal object2 Depending on the chosen output mask (see the Arguments sectionbelow), the rightmost outlet outputs either the object’s velocityor displacement at the contact position.

Arguments


list: [3×float] interactor The three arguments are respectively the force stiffness, a param-eter which depends on the contact-surface’s shape, and the dissi-pation coefficient.


24












25

1.1.7. friction 2modalb∼

Non-linear friction between two modal objects.

Input




float object1 4th inlet. Initial rubbing velocity.

list: [9×float] 5th inlet. The nine arguments are respectively: the mean bristles-stiffness, the mean bristles-dissipation, a viscosity coefficient,the dynamic-friction coefficient, the static-friction coefficient, thebreak-away coefficient, the Stribeck velocity, the perpendicularpressure which object1 applies on object2, and finally the noiseintensity.









start rubbing object1 1st inlet. Followed by a float value, sets the initial rubbing velocity.

nteract pressr 1st inlet. Followed by nine float values which set respectively: themean bristles-stiffness, the mean bristles-dissipation, a viscositycoefficient, the dynamic-friction coefficient, the static-friction co-efficient, the break-away coefficient, the Stribeck velocity, the per-pendicular pressure which object1 applies on object2, and finallythe noise intensity.

26











Output





Arguments


27

list: [9×float] The nine arguments are respectively: the mean bristles-stiffness,the mean bristles-dissipation, a viscosity coefficient, the dynamic-friction coefficient, the static-friction coefficient, the break-awaycoefficient, the Stribeck velocity, the perpendicular pressure whichobject1 applies on object2, and finally the noise intensity.














28


29

1.1.8. friction modalb wg∼

Non-linear friction between one modal and one waveguide object.

Input




float object1 4th inlet. Initial bowing velocity.

list: [9×float] 5th inlet. The nine arguments are respectively: the mean bristles-stiffness, the mean bristles-dissipation, a viscosity coefficient,the dynamic-friction coefficient, the static-friction coefficient, thebreak-away coefficient, the Stribeck velocity, the perpendicularpressure which object1 applies on object2, and finally the noiseintensity.









start bowing object1 1st inlet. Followed by a float value, sets the initial bowing velocity.

nteract pressr 1st inlet. Followed by nine float values which set respectively: themean bristles-stiffness, the mean bristles-dissipation, a viscositycoefficient, the dynamic-friction coefficient, the static-friction co-efficient, the break-away coefficient, the Stribeck velocity, the per-pendicular pressure which object1 applies on object2, and finallythe noise intensity.

30










Output

Outlets come from pickup points of both object1 and object2 (starting from left). The rightmostone comes from the only pickup point of object2.


signal object2 Depending on the chosen output mask (see the Arguments sectionbelow), the rightmost outlet outputs either the object’s velocityor displacement at the contact position.

31

Arguments


list: [9×float] The nine arguments are respectively: the mean bristles-stiffness,the mean bristles-dissipation, a viscosity coefficient, the dynamic-friction coefficient, the static-friction coefficient, the break-awaycoefficient, the Stribeck velocity, the perpendicular pressure whichobject1 applies on object2, and finally the noise intensity.













32

1.2. Liquids

Low level models of basic events responsible for acoustic emission in liquids are considered. Inparticular, we consider the formation of single resonating cavities (bubbles) observed in dripping,boiling, or pouring.

Bubble model

The formation of radially oscillating bubbles under the surface of a liquid volume is modeled as-suming that the impulse response of the single bubble is well represented by a damped sinusoid withtime-varying frequency, given by Minnaert’s formula [24]. One of the most common events involvingthe formation of radially oscillating bubbles under the surface of a liquid volume is dripping, thefalling of a drop or an object into a quiescent liquid. In the simplest case, when the initial impactsound is neglected and when there is no crown formation following the initial impact, dripping can berepresented by a single bubble event.

However, the more the initial impact sound is perceivable and the cavity of the bubble becomes larger(e.g., for large impacting mass), the less the simple single bubble sound model becomes adequate torepresent the dripping event. This is even more evident when large objects or drops falling into aresting liquid generate many secondary bubbles and droplets events due to the mass displaced by theprincipal impact event (splashing).

The external onebubble∼ provides controls for the bubble radius, and for the slope of the frequencyrise due to radius change in time.

Figure 1.3: Single bubble help patch

33

1.2.1. onebubble∼

Single radially oscillating bubble.

Input

bang 1st inlet. Bubble event trigger.

float 2nd inlet. Bubble initial radius.

float 2nd inlet. Frequency slope.

radius 1st inlet. Followed by a float value, sets the initial bubble radius.

slope 1st inlet. Followed by a float value, sets the frequency slope.

Output

Radiated pressure.

signal Signal outlet.

Arguments

None.

34

2. Higher-level models

This chapter covers algorithms which exploit the low-level models seen in Chapter 1. Notice that theexpression “higher-level” indicates more complex and structured algorithms, corresponding to some-what large-scale events, processes or textures. In a way, that matches the meaning of the expression“high-level” in Computer Science, where it often denotes languages similar to those of human beings.Of course, in order to achieve that, high-level languages are indeed more complex and structured thanlow-level ones.

The higher-level algorithms discussed here implement temporal patterns or other physically con-sistent controls (e.g., external forces) superimposed to low-level models. These algorithms are madeavailable either as dedicated externals, or as patches using Max/MSP visual programming. In theformer case, an external may implement both the control and the low-level synthesis layers (that is, itis self-contained), and it is therefore labeled with a final tilde ∼; or it may implement only the controllayer (that is, it is just meant to drive other externals), in which case it is labeled without a finaltilde ∼.

2.1. Bouncing

Bouncing is a sound process resulting from repeated macro-impact events [18].

Figure 2.1: Bouncing (falling object) patch

The example patch (EX00_bouncing.mxb) (Fig. 2.1) simulates the case of a falling object - a typical

35

situation in which bouncing happens. There, an instance of the low-level external impact inertialb modalb∼is driven by a control layer implemented into the sub-patch dropper (which lies inside the sub-patchsub-bouncing).

The available high-level user’s controls are:

falling height: sets the time interval between the first two bounces.

object elasticity: sets the acceleration and deceleration rate of bounces.

object shape regularity: sets the level of randomness to be applied to bounces. This corresponds tothe symmetry properties of the falling object.

object weight: sets the maximum hitting velocity.

36

2.2. Crumpling

Crumpling is a widely general-purpose sonification paradigm based on the control of atomic sonicevents (for instance impacts). The crumpling process governs the timbre and dynamics of every atomicevent as well as the temporal gap in between two successive events.• Concerning dynamics and time, their control is exerted by statically setting macroscopic param-eters of specific stochastic laws which, in their turn, produce sequences of micro events whoseoverall intensity and temporal density follow directly from the parameter values. However, everysequence individually exhibits statistical variability.• Concerning timbre, its control follows by informing the model with microscopic features (viz. thenature of the atomic sounds) along with changing (usually coarse) structural evolutionary featuresof the physical model dynamically along time.

Detailed explanations about the physical nature of this sonification paradigm, in particular the cor-respondence existing between an atomic crumpling event and its representation as an impact betweena hammering and a resonating object, can be found in [18].

2.2.1. control crump

Figure 2.2: Main crumpling example patch

The external control crump implements such controls and is expressly meant to drive impact eventsmodeled by two instances of impact inertialb modalb∼ externals in which both object2s (modal objects)have two resonating modes. Hence it controls two couples of interacting objects (left: object1 andobject2, right object1 and object2). Fig. 2.2 shows the main example patch (EX00_crumpling.mxb) inwhich control crump drives the impact sound models.

Input

bang 1st inlet. A new atomic crumpling event is triggered.

37

reset 1st inlet. The crumpling process is reset by restoring initial energyand settings.

float 2nd inlet. Size of the crumpling. Initializes the energy.

float 3rd inlet. Force of the crumpling.

float 4th inlet. Smoothness of the crumpling.

list: [2×float] 5th inlet. Sets the range in which the frequency-factors (multi-plying coefficients for resonances) of both object2s vary along theduration of crumpling process.The two float arguments are respectively the initial and the finalvalues of the coefficients.

list: [4×float] 6th inlet. Sets the range in which the decay times of the two modesof both object2s vary along the duration of crumpling process.The four float arguments are, in order: the initial and the finalvalues of the decay for the first mode of both object2s; the initialand the final values of the decay for the second mode of bothobject2s.

list: [2×float] 7th inlet. Sets the range in which the masses of both object1s varyalong the duration of crumpling process.The two float arguments are the initial and the final values of themasses.

Output

Outlets provide controls to two instances of impact inertialb modalb∼ externals, where both object2shave two resonating modes.

float 1st outlet. Delay value after which to back-bang the control crumpexternal itself by means of a loopback containing a delay object.

float 2nd outlet. Mass of the left object1.

float 3rd outlet. Frequency-factor of the left object2.

float 4th outlet. Decay value of the left object2’s first mode.

float 5th outlet. Decay value of the left object2’s second mode.

float 6th outlet. Energy of the signal outcoming from the left object2.

float 7th outlet. Mass of the right object1.

float 8th outlet. Frequency-factor of the right object2.

float 9th outlet. Decay value of the right object2’s first mode.

float 10th outlet. Decay value of the right object2’s second mode.

38

float 11th outlet. Energy of the signal outcoming from the right object2.

Arguments

All arguments are mandatory.

float Crumpling size.

float Crumpling force.

float Crumpling smoothness.

list: [2×float] Default range (initial and final values) of the frequency-factors ofboth object2s.

list: [4×float] Default range of the decay times. The four float arguments are,in order: the initial and the final values of the decay for the firstmode of both object2s; the initial and the final values of the decayfor the second mode of both object2s.

list: [2×float] Default range (initial and final values) of the mass of the bothobject1s.

list: [2×float] Cutoff frequency sweep range (initial and final values). It is meantto be connected to a lowpass filter with controllable cutoff fre-quency.

39

2.2.2. control contcrump

Figure 2.3: Main continuous crumpling example patch

The external control contcrump is a simplified version of the previous control crump, but may turnout to be even more versatile and effective [1]. In this case, there are no complex energy policies but,instead, a continuous generation of stochastic events. Again, the external control contcrump is meantto drive impact events modeled by any external.

Fig. 2.3 shows the main example patch EX00_continuous_crumpling.mxb where control contcrumpdrives the external impact inertialb modalb∼.

Input

force 1st inlet. Followed by a float, sets the force of the crumplingprocess. Notice that it is sufficient to vary the force in order totrigger more atomic crumpling events.

resistance 1st inlet. Followed by a float, sets the crumpling resistance: forlow resistance the crumpling events are more dense (i.e., the forceis spent more easily), for high resistance the crumpling events areless dense (i.e., the force is spent less easily).

Output

Outlets provide controls to any impact model.

float 1st outlet. Delay value after which to bang the impact external.

float 2nd outlet. Velocity of impact events.

40

Arguments

None.

41

2.3. Rolling

The rolling model simulates a ball rolling on a smooth surface. The model [18] combines differentunderlying components: the heart of the model is an impact model (namely, provided by the externalimpact inertialb modalb∼) which is driven by two pseudo-physical models that simulate, respectively,the geometry of the contacting surfaces (which controls the temporal pattern of micro-impacts), andthe ball’s asymmetries (which superimpose a modulation of the impact force). Only normal impactsare taken into account, discarding friction components (which are comparatively small).

The shape of the plane surface is simulated by a band-pass filtered noise signal. This serves as inputfor a specifically designed filter, which in turn calculates the trajectory of (the barycenter of) a ballrolling on such surface. Given a surface, the trajectory depends on the ball’s diameter. Again, forsimplicity, the possible bouncing of the ball is not taken into account.

Figure 2.4: Main rolling example patch

The main example patch EX00_rolling.mxb (Fig. 2.4) implements the model described above withinthe sub-patch holy roller∼.


key ’s’: starts / stops the rolling ball accordingly to the velocity map.

key ’a’: keeps the rolling ball in contact with the surface. This is useful in the case the ball experiencedinstabilities.

diameter: sets the diameter of the rolling ball. Given a velocity, the diameter affects both the pitchand the modulation pattern superimposed to the contact force.

velocity: sets the current velocity of the rolling ball. Given a diameter, the velocity affects both thetemporal pattern of micro-impacts, and the modulation of the contact force.

amplification: amplifies the rolling sound signal.

42

2.4. Breaking

The breaking model simulates the event of solids breaking. Again, this high-level model makes use ofa low-level impact model, namely by using the external impact inertialb modalb∼. More precisely, thebreaking model is made of a pseudo-random sequence of impact events, plus an initial noise burst [18].Therefore this model shares a common basis with the bouncing model described in section 2.1, and infact it makes use of the same abstraction dropper with generally high values of “randomness”, and aquickly decreasing temporal density.

Figure 2.5: Main breaking example patch

The main example patch EX00_breaking.mxb (Fig. 2.5) implements the model described above,and the available high-level user’s controls are:

impact velocity: sets the velocity of the first (rupturing) impact.

initial interval: sets the initial interval in ms between the first two impact events and affects theinterval between subsequent impacts.

deceleration factor: two homonym controls set the deceleration of impact occurrences in two separaterandomization processes.

maximum interval: sets the maximum interval in ms between impact events.

initial value: sets the initial value feeding the randomization process.

maximal random interval-reduction: two sliders drive two separate randomization processes, namelycontrolling the maximum random decrement of the time span between impact occurrences.

43

2.5. Boiling, frying, streaming, pouring

High level models of complex liquid events were also considered. The one illustrated here concernsthe formation of high quantities of bubbles observed in boiling, frying, streaming or pouring. Thesingle resonating bubble model is well suited to serve as the elementary brick to represent populationsof bubbles, whose parameters and triggering instants are designed according to given statistical distri-butions. Typical phenomena in which bubble distributions are observed are boiling or frying liquids,water streaming, pouring of a liquid into a container or into another quiescent liquid, breaking waves.

The external bubblestream∼ provides controls for bubble triggering frequency, for the mean and vari-ance of radius and radius slope values, for the variance of bubbles amplitude, and for the smoothnessof bubbles onset. A sample patch is provided which contains presets for boiling, frying, and runningwater sounds.

Figure 2.6: Bubblestream patch

44

2.5.1. bubblestream∼

Population of bubbles.

Input

bang 1st inlet. Toggles the bubbles flow.

float 2nd inlet. Mean bubbles radius.

float 3rd inlet. Variance of bubbles radius.

float 4th inlet. Mean radius slope.

float 5th inlet. Variance of radius slope.

float 6th inlet. Mean amplitude.

float 7th inlet. Bubbles onset smoothness.

float 8th inlet. Bubbles frequency.

Output

Radiated sound pressure.

Arguments

None.

45

2.6. Dripping, splashing

This modeling study focuses on splash-like events observed in the interaction of liquid volumes.As an attempt to reproduce the temporal structure of a splashing sound, we choose to design it asa sequence of three distinct low-level events: 1. a short initial impact sound, 2. a bubble soundmodeled as detailed in the low-level models section, and 3. a secondary droplets event texture. In thepresent implementation, sampled waveforms are used to reproduce the initial impact sound and thefinal sound due to droplets formation, whereas the principal bubble formation is based on the singlebubble model.

The external splash∼ provides controls for gain balance of the single components, and for theparameters of the principal bubble sound. A sample patch is provided which contains presets forsingle bubble and complete splashing sounds.

Figure 2.7: Splash patch

46

2.6.1. splash∼

Splash or dripping event.

Input

bang 1st inlet. Splash event trigger.

float 2nd inlet. Principal bubble initial radius.

float 2nd inlet. Radius slope.

float 3d inlet. Initial impact event gain.

float 4th inlet. Principal bubble gain.

float 5th inlet. Droplet event gain.

Output

Radiated pressure.

Arguments

None.

47

2.7. Particles models sonification

We model here the sound produced by a liquid in motion, in the particular case in which thesimulation is based on physically-based numerical models of the underlying fluid dynamics. In partic-ular, a particle-based fluid simulation method known as smoothed particle hydrodynamics (SPH) isadopted[13]. This example concerns an experimental configuration representing the falling of a liquidvolume into an underlying rigid container.

In SPH, fluids are modeled with particles representing small fluid volumes. Each particle is describedby a set of physical quantities (i.e. position, velocity, acceleration, pressure, and density) which areupdated at each simulation step based on the interaction with neighbors particles. This interaction isruled by the Navier-Stokes equations for the conservation of mass and momentum. Moreover, externalforces such as the ones due to gravity or collisions with solids objects, can be accounted for in thesimulation.

To address the sonification of particles in motion, we adopt commonly used computation rates forparticles update (usually 10 to 30 frames per second) and we detect events for which acoustic emissionis predictable. When such an event is detected, a corresponding acoustic event is triggered. A minimalset of low level models representing basic events responsible for acoustic emission in liquids has beenused. In particular, we rely on sounds originating from the formation of single resonating cavities(bubbles) under the liquid surface (such as in dripping, or pouring), and from surface impacts (suchas those occurring between two liquid volumes or between a solid and a liquid surface).

The analysis of the instantaneous fluid configuration is performed by recognition of a set of relevantconfigurations and events, commonly observed in experimental situations involving liquids in motion.Among the configurations that are considered interesting, two are worth noting: a) the impact of afalling liquid volume with an empty container, and b) the impact of liquid volumes with the surfaceof a resting liquid in the container. In the first case, there is no possibility for the bubble formationphenomenon to occur, and only the impact sound due to the collision of the liquid volume with thesolid surface may arise. In the second case, together with the sound from liquid-liquid impact, soundfrom the formation of cavities under the liquid surface (bubbles) is also generated.

In the design of the mapping from given system states to audio events, the physical parametersof particles are taken into consideration. It is the case, for example, of the radius of bubbles arisingfrom dripping events, which is directly related to the mass of impacting liquid and to the depth of theresting liquid in the container.

2.7.1. particles sim∼

Sonification of particle-based simulations representing the falling of a liquid volume into a rigidcontainer. In the present implementation, the external reads the result of a precomputed simulationfrom a file provided by the user.

Input

bang 1st inlet. Play simulation from start.

int 2nd inlet. Video frames per second.

int 3rd inlet. Downsampling factor for particles visualization.

float 4th inlet. Bubbles mean radius.

Output

48

signal radiated acoustic pressure.

list positions of particles (x,y,z) for visualization purposes.

Arguments

string simulation file name.

49

2.8. Vacuum Cleaner

The vacuum cleaner model is built as simple simulation (digital waveguide [22]) of a tube, which hasone switchable open/closed termination and is open-ended at the other side. The input is provided bya pink noise signal, somewhat simulating the overall effect of motor noise and air getting through thetube. In Fig. 2.8 the diagram of the vacuum cleaner model is represented: it is a simple comb filter in

Figure 2.8: The diagram of the vacuum cleaner model

which the delay line is in series with a low-pass filter (simulating the losses at one termination) and areflection coefficient; in order to refine the sound, at the output stage there two more filters (low-passand high-pass); moreover, a low frequency oscillator (LFO) provides some modulation, namely varyingthe length of the delay line (i.e., the pitch) and the cutoff frequencies of both the low-pass filters.

Figure 2.9: The main example patch for the vacuum cleaner model

The main example patch EX00_vacuum_cleaner.mxb (Fig. 2.9) implements the model describedabove within the sub-patch vacuum cleaner∼.


switch ON/OFF: triggers the simulations of a vacuum cleaner turning on and off, respectively. Thisis mainly achieved by varying the length of the delay line and opening or closing the filters.

cork the tube: when the tube is open, triggers the simulation of a corked tube, namely by settinga negative sign for the reflection coefficient and lowering the cutoff frequencies of the low-passfilters.

open the tube: this is the default configuration. When the tube is corked, triggers the simulationof the opening, namely by setting a positive sign for the reflection coefficient and increasing thecutoff frequencies of the low-pass filters.

50

3. Acoustic depth

This physics-based model aims at providing an acoustical environment for rendering distance cues,namely the intensity and the direct-to-reverberant energy ratio.The model consists of a bidimensional rectilinear digital waveguide mesh. In addition to the dimensionsof the mesh and the positions of the listener and of the sound source(s), the properties of the boundariesmay be modified by varying the coefficients [b0,b1,a1] of the first-order filters modeling the boundariesalong the two axes of the mesh (implemented as Y (z) = b0+b1z−1

1+a1z−1 X(z)). Furthermore, due to theheavy computational load of the model, a downsampling may be required prior to the computation inthe mesh, set by the downsampling factor.Two externals are provided:

1. mesh2D static∼ : suitable for rendering one or more static sound sources located at differentpositions from the listener (see Fig. 3.1).

2. mesh2D dynamic∼ : suitable for rendering one moving sound source along the depth dimension,i.e. one approaching/receding source. In this case, the computation in the mesh is made efficientby switching the sound source and the listening positions, such that the measurement positionvaries at runtime and not the position of the source (see Fig. 3.2).

Figure 3.1: Mesh for multiple static sound sources help patch

51

Figure 3.2: Mesh for one moving sound source help patch

3.0.1. mesh2D static∼

Rectilinear digital waveguide mesh for simulating various static sound sources at different distancesfrom the listening point.

Input

list: [signal] 1st inlet. As many mono input signals as the number ofsound sources.

float 2nd inlet. Downsampling factor.

list: [2×float] 3rd inlet. Dimensions of the mesh, length and width re-spectively, in meters.

list: [float] 4th inlet. As many source positions as the number of soundsources, in meters.

float 5th inlet. Listening position in the mesh, in meters.

list: [3×float] 6th inlet. Coefficients [b0,b1,a1] of the first order digitalwaveguide filter modeling the boundary segments on thelengths of the mesh.

list: [3×float] 7th inlet. Coefficients [b0,b1,a1] of the first order digitalwaveguide filter modeling the boundary segments on thewidths of the mesh.

52

Output

signal Mono output signal.

Arguments

One argument is mandatory, it initializes the number of sound sources present in the mesh.

int Number of sound sources.

53

3.0.2. mesh2D dynamic∼

Rectilinear digital waveguide mesh for simulating one approaching/receding sound source.

Input

signal 1st inlet. Mono input signal.

signal 2nd inlet. Source position, in meters.

float 3rd inlet. Downsampling factor.

list: [2×float] 4th inlet. Dimensions of the mesh, length and width re-spectively, in meters.

float 5th inlet. Listening position in the mesh, in meters.

list: [3×float] 6th inlet. Coefficients [b0,b1,a1] of the first order digitalwaveguide filter modeling the boundary segments on thelengths of the mesh.

list: [3×float] 7th inlet. Coefficients [b0,b1,a1] of the first order digitalwaveguide filter modeling the boundary segments on thewidths of the mesh.

Output

signal Mono output signal.

Arguments

None.

54

3.0.3. Example

DepThrow is an audio game using the physics-based models of rolling and the 2D mesh for simulatinga rolling ball inside an inclined tube. The game consists in throwing a virtual ball inside a virtualopen-ended tube which is inclined. This is done by setting the initial velocity of the ball, either usingthe Wii Remote (the handed wireless controller for Nintendo’s Wii console), or the slider on the mainpatch DEPTHROW.mxb. Then the current velocity and position of the ball are deduced from the initialvelocity set by the user and the tilt angle of the tube. Besides, the physics-based model of bouncingsimulate the ball bouncing if it falls out at the far end of the tube. Refer to Part 3 Section 7.4.1 formore details about the game.

Figure 3.3: DepThrow main patch

The main Max/MSP patch (DEPTHROW.mxb) is shown in Fig. 3.3. The computation in the meshis implemented in the abstraction mesh2D, the model of rolling lies within the abstraction rolling HPand the bouncing within the abstraction bouncing thing. Data from the Wii Remote are handled inthe abstraction wiiremote trolley1.


initial velocity sets the initial velocity of the virtual ball, in meters/second.

bang throws the object in the tube.

tube tilt angle sets the tilt angle of the tube, in degrees.

tube length sets the length of the tube, in meters.

1In order to interface the Wii Remote controller with Max/MSP the external aka.wiiremote is needed and can bedownloaded from http://www.iamas.ac.jp/∼aka/max/. Notice that the external is currently for Mac OS X only.

55

http://www.iamas.ac.jp/~aka/max/

GNU Free Documentation License

Version 1.2, November 2002

Copyright (C) 2000, 2001, 2002 Free Software Foundation, Inc.59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

Everyone is permitted to copy and distribute verbatim copiesof this license document, but changing it is not allowed.

0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other functional and useful document“free” in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, withor without modifying it, either commercially or noncommercially. Secondarily, this License preservesfor the author and publisher a way to get credit for their work, while not being considered responsiblefor modifications made by others.

This License is a kind of “copyleft”, which means that derivative works of the document mustthemselves be free in the same sense. It complements the GNU General Public License, which is acopyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free softwareneeds free documentation: a free program should come with manuals providing the same freedoms thatthe software does. But this License is not limited to software manuals; it can be used for any textualwork, regardless of subject matter or whether it is published as a printed book. We recommend thisLicense principally for works whose purpose is instruction or reference.

1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placed bythe copyright holder saying it can be distributed under the terms of this License. Such a notice grantsa world-wide, royalty-free license, unlimited in duration, to use that work under the conditions statedherein. The “Document”, below, refers to any such manual or work. Any member of the public is alicensee, and is addressed as “you”. You accept the license if you copy, modify or distribute the workin a way requiring permission under copyright law.

A “Modified Version” of the Document means any work containing the Document or a portion ofit, either copied verbatim, or with modifications and/or translated into another language.

A “Secondary Section” is a named appendix or a front-matter section of the Document that dealsexclusively with the relationship of the publishers or authors of the Document to the Document’soverall subject (or to related matters) and contains nothing that could fall directly within that overallsubject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may notexplain any mathematics.) The relationship could be a matter of historical connection with the subjector with related matters, or of legal, commercial, philosophical, ethical or political position regardingthem.

The “Invariant Sections” are certain Secondary Sections whose titles are designated, as being thoseof Invariant Sections, in the notice that says that the Document is released under this License. If

57

a section does not fit the above definition of Secondary then it is not allowed to be designated asInvariant. The Document may contain zero Invariant Sections. If the Document does not identify anyInvariant Sections then there are none.

The “Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-CoverText may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A “Transparent” copy of the Document means a machine-readable copy, represented in a formatwhose specification is available to the general public, that is suitable for revising the document straight-forwardly with generic text editors or (for images composed of pixels) generic paint programs or (fordrawings) some widely available drawing editor, and that is suitable for input to text formatters orfor automatic translation to a variety of formats suitable for input to text formatters. A copy madein an otherwise Transparent file format whose markup, or absence of markup, has been arranged tothwart or discourage subsequent modification by readers is not Transparent. An image format is notTransparent if used for any substantial amount of text. A copy that is not “Transparent” is called“Opaque”.

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfoinput format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of trans-parent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats thatcan be read and edited only by proprietary word processors, SGML or XML for which the DTD and/orprocessing tools are not generally available, and the machine-generated HTML, PostScript or PDFproduced by some word processors for output purposes only.

The “Title Page” means, for a printed book, the title page itself, plus such following pages as areneeded to hold, legibly, the material this License requires to appear in the title page. For works informats which do not have any title page as such, “Title Page” means the text near the most prominentappearance of the work’s title, preceding the beginning of the body of the text.

A section “Entitled XYZ” means a named subunit of the Document whose title either is preciselyXYZ or contains XYZ in parentheses following text that translates XYZ in another language. (HereXYZ stands for a specific section name mentioned below, such as “Acknowledgements”, “Dedications”,“Endorsements”, or “History”.) To “Preserve the Title” of such a section when you modify theDocument means that it remains a section “Entitled XYZ” according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this Licenseapplies to the Document. These Warranty Disclaimers are considered to be included by reference inthis License, but only as regards disclaiming warranties: any other implication that these WarrantyDisclaimers may have is void and has no effect on the meaning of this License.

2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially or noncommercially,provided that this License, the copyright notices, and the license notice saying this License applies tothe Document are reproduced in all copies, and that you add no other conditions whatsoever to thoseof this License. You may not use technical measures to obstruct or control the reading or furthercopying of the copies you make or distribute. However, you may accept compensation in exchangefor copies. If you distribute a large enough number of copies you must also follow the conditions insection 3.

You may also lend copies, under the same conditions stated above, and you may publicly displaycopies.

58

3. COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have printed covers) of the Docu-ment, numbering more than 100, and the Document’s license notice requires Cover Texts, you mustenclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Textson the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legiblyidentify you as the publisher of these copies. The front cover must present the full title with all wordsof the title equally prominent and visible. You may add other material on the covers in addition.Copying with changes limited to the covers, as long as they preserve the title of the Document andsatisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first oneslisted (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you musteither include a machine-readable Transparent copy along with each Opaque copy, or state in or witheach Opaque copy a computer-network location from which the general network-using public has accessto download using public-standard network protocols a complete Transparent copy of the Document,free of added material. If you use the latter option, you must take reasonably prudent steps, when youbegin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thusaccessible at the stated location until at least one year after the last time you distribute an Opaquecopy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redis-tributing any large number of copies, to give them a chance to provide you with an updated versionof the Document.

4. MODIFICATIONS

You may copy and distribute a Modified Version of the Document under the conditions of sections2 and 3 above, provided that you release the Modified Version under precisely this License, with theModified Version filling the role of the Document, thus licensing distribution and modification of theModified Version to whoever possesses a copy of it. In addition, you must do these things in theModified Version:

A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, andfrom those of previous versions (which should, if there were any, be listed in the History sectionof the Document). You may use the same title as a previous version if the original publisher ofthat version gives permission.

B. List on the Title Page, as authors, one or more persons or entities responsible for authorship ofthe modifications in the Modified Version, together with at least five of the principal authors ofthe Document (all of its principal authors, if it has fewer than five), unless they release you fromthis requirement.

C. State on the Title page the name of the publisher of the Modified Version, as the publisher.D. Preserve all the copyright notices of the Document.E. Add an appropriate copyright notice for your modifications adjacent to the other copyrightnotices.

F. Include, immediately after the copyright notices, a license notice giving the public permissionto use the Modified Version under the terms of this License, in the form shown in the Addendumbelow.

G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts givenin the Document’s license notice.

H. Include an unaltered copy of this License.I. Preserve the section Entitled “History”, Preserve its Title, and add to it an item stating atleast the title, year, new authors, and publisher of the Modified Version as given on the Title

59

Page. If there is no section Entitled “History” in the Document, create one stating the title, year,authors, and publisher of the Document as given on its Title Page, then add an item describingthe Modified Version as stated in the previous sentence.

J. Preserve the network location, if any, given in the Document for public access to a Transparentcopy of the Document, and likewise the network locations given in the Document for previousversions it was based on. These may be placed in the “History” section. You may omit a networklocation for a work that was published at least four years before the Document itself, or if theoriginal publisher of the version it refers to gives permission.

K. For any section Entitled “Acknowledgements” or “Dedications”, Preserve the Title of the sec-tion, and preserve in the section all the substance and tone of each of the contributor acknowl-edgements and/or dedications given therein.

L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles.Section numbers or the equivalent are not considered part of the section titles.

M. Delete any section Entitled “Endorsements”. Such a section may not be included in the ModifiedVersion.

N. Do not retitle any existing section to be Entitled “Endorsements” or to conflict in title withany Invariant Section.

O. Preserve any Warranty Disclaimers.If the Modified Version includes new front-matter sections or appendices that qualify as Secondary

Sections and contain no material copied from the Document, you may at your option designate someor all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in theModified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled “Endorsements”, provided it contains nothing but endorsements ofyour Modified Version by various parties–for example, statements of peer review or that the text hasbeen approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 wordsas a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passageof Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by)any one entity. If the Document already includes a cover text for the same cover, previously added byyou or by arrangement made by the same entity you are acting on behalf of, you may not add another;but you may replace the old one, on explicit permission from the previous publisher that added theold one.

The author(s) and publisher(s) of the Document do not by this License give permission to use theirnames for publicity for or to assert or imply endorsement of any Modified Version.

5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, under the termsdefined in section 4 above for modified versions, provided that you include in the combination all of theInvariant Sections of all of the original documents, unmodified, and list them all as Invariant Sectionsof your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical InvariantSections may be replaced with a single copy. If there are multiple Invariant Sections with the samename but different contents, make the title of each such section unique by adding at the end of it, inparentheses, the name of the original author or publisher of that section if known, or else a uniquenumber. Make the same adjustment to the section titles in the list of Invariant Sections in the licensenotice of the combined work.

In the combination, you must combine any sections Entitled “History” in the various original docu-ments, forming one section Entitled “History”; likewise combine any sections Entitled “Acknowledge-ments”, and any sections Entitled “Dedications”. You must delete all sections Entitled “Endorse-ments”.

60

6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released under thisLicense, and replace the individual copies of this License in the various documents with a single copythat is included in the collection, provided that you follow the rules of this License for verbatim copyingof each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under thisLicense, provided you insert a copy of this License into the extracted document, and follow this Licensein all other respects regarding verbatim copying of that document.

7. AGGREGATION WITH INDEPENDENTWORKS

A compilation of the Document or its derivatives with other separate and independent documents orworks, in or on a volume of a storage or distribution medium, is called an “aggregate” if the copyrightresulting from the compilation is not used to limit the legal rights of the compilation’s users beyondwhat the individual works permit. When the Document is included in an aggregate, this Licensedoes not apply to the other works in the aggregate which are not themselves derivative works of theDocument.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if theDocument is less than one half of the entire aggregate, the Document’s Cover Texts may be placed oncovers that bracket the Document within the aggregate, or the electronic equivalent of covers if theDocument is in electronic form. Otherwise they must appear on printed covers that bracket the wholeaggregate.

8. TRANSLATION

Translation is considered a kind of modification, so you may distribute translations of the Documentunder the terms of section 4. Replacing Invariant Sections with translations requires special permissionfrom their copyright holders, but you may include translations of some or all Invariant Sections inaddition to the original versions of these Invariant Sections. You may include a translation of thisLicense, and all the license notices in the Document, and any Warranty Disclaimers, provided thatyou also include the original English version of this License and the original versions of those noticesand disclaimers. In case of a disagreement between the translation and the original version of thisLicense or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled “Acknowledgements”, “Dedications”, or “History”, therequirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly provided forunder this License. Any other attempt to copy, modify, sublicense or distribute the Document is void,and will automatically terminate your rights under this License. However, parties who have receivedcopies, or rights, from you under this License will not have their licenses terminated so long as suchparties remain in full compliance.

10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free DocumentationLicense from time to time. Such new versions will be similar in spirit to the present version, but may

61

differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.Each version of the License is given a distinguishing version number. If the Document specifies that

a particular numbered version of this License “or any later version” applies to it, you have the optionof following the terms and conditions either of that specified version or of any later version that hasbeen published (not as a draft) by the Free Software Foundation. If the Document does not specifya version number of this License, you may choose any version ever published (not as a draft) by theFree Software Foundation.

ADDENDUM: How to use this License for your documents

To use this License in a document you have written, include a copy of the License in the documentand put the following copyright and license notices just after the title page:

Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute and/ormodify this document under the terms of the GNU Free Documentation License, Version1.2 or any later version published by the Free Software Foundation; with no InvariantSections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is includedin the section entitled “GNU Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.”line with this:

with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts beingLIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, mergethose two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing theseexamples in parallel under your choice of free software license, such as the GNU General PublicLicense, to permit their use in free software.

62

Part II.

Generation and Testing of a Sonic Palette

63

4. Strategies for the Use and Control of theSound Design Toolkit

A considerable effort was devoted to critically reconsidering the sound models in terms of their effec-tiveness in shaping the sonic behaviour of objects. The main addressed issues are:• Accessibility of a rich timbral palette by means of synthesis parameter conditioning• Interpolation through different reference sonic states of the timbral palette• Control of the models by means of graphic interfaces plus mouse and keyboard.• Control of the models through a variety of sensors (with issues of offset, resolution and non-linearmapping)• Relation between gesture and sound with different kinds of physical mediation e.g., acting on aforce sensor by pressing with a fork or by squeezing a pepper).

A number of sound examples and Max/MSP patches concerning this part of the deliverable arecollected in a repository available on the project website and referenced in the following discussion.

4.1. Synthesis parameter conditioning for the availability of a rich timbralpalette

As a first step, we carried out a free exploration of the timbral space offered by the sound models inorder to understand as widely as possible their sonic responses and usability characteristics.This first stage addressed four main interconnected issues:• Spectral modeling of the modal objects• Exploration parameter by parameter of the sound models• Perceptually based definition of different configuration for various materials and types of inter-actions• GUI definition process.

A “synthesis by analysis” experimental approach has been adopted for the production of this veryfirst set of static sounds: we proceeded by listening to a conspicuous set of sound libraries and thenanalysing sound samples of various glass, metallic and plastic materials, both frictions and impacts.Unexpectedly the mediation of the recorded sound samples, though of high quality, made the sounddesign quite difficult and plodding, both for the extreme complexity and richness of the samples, whichcontradicts the cartoonification approach, and the intrinsic framework of the models that actuallyemulates a close sound take at the pick-up points of the modal objects.As far as modal objects configuration can be seen as a sum of oscillators (e.g. number of modes), eachdescribed by its own frequency, gain and decay time, we experimented a spectral modeling approach,since these parameters basically sketch out the material (spectrum) and size (resonance). This revealeditself to be very intuitive and effective, in such a way to more easily shape the interaction parameters,namely the physical properties of the materials involved. In this view the mode 1 of the modal object2is considered to be the fundamental or centre frequency, while subsequent available modes frequenciesare set as multiplying factor of first. This leaded to a first improvement of the GUI (see Figure 4.1.During the sound design survey, we decided to lay by the sound samples and to rely upon real soundsto creatively tune the sound models. The choice of considering real sounds was also due to the factthat the models pointed the necessity of a constant analysis of the gestures involved in the interactionswith the real world. This made possible to create a huge number of presets with different materialsand hybrids.

65

Figure 4.1: Impact model graphical user interface, the mode 2 frequency is set as factor of the mode 1(yellow number box).

This study was directed to a primary raw tuning of the physical parameter ranges, as to avoid allthose settings for which inconsistent values produced distorted or no sound.

4.1.1. Low level models

Impact - [EX00 impact.mxb]Due to its major intuitiveness, we started from the impact model and we developed a catalogue ofpresets. For each sound model, a large practical work of sonic simulation of materials has been made:shape, size, typology of gesture and effective cross-combinations of the three, achieved through iteratedtrials (see for example Figures 4.2, 4.3 and 4.4).

Figure 4.2: EX00 impact.mxb. Crystal glass preset.

66

Figure 4.3: EX00 impact.mxb. Wood knocking preset.

Figure 4.4: EX00 impact.mxb. Control parameters for a wood sound.

67

Friction - [EX00 friction.mxb] The survey of the friction model started by systematically testing thespectral modeling approach successfully applied to the modal objects of the impact model while con-sidering the physical to phenomenological correspondences of the friction model parameters depictedin Figure 4.5 (listen to the sound example: 01 friction phenom description.mp3). More in detail, wewent through a timbral exploration starting from some initial state (see for example Figure 4.6) andvarying one parameter at a time in order to find a new state of the model providing some characteristicand novel sound. The process is illustrated in Figure 4.7.

Figure 4.5: A phenomenological guide to the friction model parameters.

Figure 4.6: EX00 friction.mxb. Initial state.

As expected, the complex non-linearity of the model stressed remarkable differences with respect toa mere transposition of impact model parameters onto the friction parameters, especially concerningthe decay values of the modal object2 (resonator). Moreover all the physical to phenomenologicalcorrespondences of the interaction parameters appeared to be effectively perceivable in static sounds.

68

Figure 4.7: EX00 friction.mxb. Phenomenological timbral exploration through parameter by param-eter variations.

However the specificity of the friction model resides in its dynamic control and mutual conditioning ofthe external rubbing force applied to the object1 (1st inlet of the external) and the pressure on rubbervalue, namely the perpendicular pressure that object1 applies to object2 (5th inlet of the external,eighth parameter of the interaction list).The gained understanding of the model outlined the greater perceptive effectiveness of the followinginteraction parameters: the bristle stiffness (sigma0), the bristle viscosity (sigma1), the Stribeck ve-locity and the noise coefficient. By simply working on these parameters, while keeping the othersconstant, it’s possible to design almost all kinds of produced presets. The remaining parameters weremainly used to refine the sounds in terms of sound quality.A conspicuous catalogue of timbral categories has been produced for the friction sound model, startingfrom the every sound taxonomy already proposed by UNIVERONA (see CLOSED Deliverable 4.1).Compared to the impact model, a relevant aspect of the sonic results of the friction model is its deepdependence on gesture modelization of the resulting sound event, i.e. on the coupling between theexternal rubbing force and the exerted pressure on rubber. As a general rule, the sound event creationprocess proceeded by designing materials first and then modeling gestures on basic events such asrubbing and braking, by interpreting what suggested by the sonic results. Especially for this model, acareful observation and action analysis have been made, in order to understand the gesture patterns.On the GUI front, a control layer was added to the synthesis algorithms (see Figure 4.8, as to variatethe above mentioned external forces in a pseudo-random way. These control parameters concerned thesettings of independent random ranges and their variations over time for the external rubbing force,that is the extent of the gesture, and for the pressure on rubber, namely the intensity of the gesture.In a second time, interpolation functions, graphically editable, were added to control the nuance ofthe sound event.

4.1.2. High level models

Continuous crumpling - [EX00 continuous crumpling palette.mxb] This is a recent addon to thesound design tools, the continuous control crumpling has been under study and testing. As for the

69

Figure 4.8: EX00 friction.mxb. Graphical user interface revision and improvement.

friction model case, the continuous crumpling effectiveness highly depends on continuous variationsover time of its dynamic control parameters, the exerted force and the resistance.Improvements concerned the adoption of a spectral modeling approach, as already explained, and theconditioning of the force and resistance parameters. Two function graphical editors, one for the forceand one for the resistance, were added to the GUI, as to help sketching sound events, such as walking,tearing papers, pouring sand, crushing ice and a “moka spilling coffee-like” sound (see Figure 4.9).

Bubblestream - [EX00 bubblestream palette.mxb] Being liquid sounds complex events that oftenare characterised by the combination of various low level models, it was not simply possible to properlymodel events such as pouring liquids, dripping or filling. However major efforts were devoted to tryout the bubblestream model towards the sound design of such sound events.For this purpose, a dynamic control layer was added in order to variate the bubblestream parameters(see Figure 4.10. For instance simpler burbling events, like frying or stirring sounds, were effectivelymodeled by pseudo-randomly changing and coupling the mean bubble radius, the radius variation, themean blasting slope and the envelope smoothness. By refining these ranges, the sensation of both thesize of the vessel and the density of the liquid were almost achieved.On the contrary, in order to give a filling sound sensation, it has been sufficient to work on the couplingof the first two parameters (mean bubble radius and radius variation). Somehow this appeared to beconvincing also for providing a sensation of the size of the vessel.

4.1.3. Interpolation between different reference sonic states of the timbral palette

A further enrichment of timbral palette was set up by exploiting the effectiveness and expressivity ofdynamic interpolations through a) different materials, b) different configurations of the same material,c) different gestures or sound events. This work was also aimed at testing the stability of the modelsand eventually the CPU consumption.The practising with the sound design tools went beyond the simple emulation of everyday sounds and

70

Figure 4.9: EX00 continuous crumpling palette.mxb. Graphical user interface revision and improve-ment. Preset nr 5: Ice crush.

Figure 4.10: EX00 bubblestream palette.mxb. Graphical user interface revision and improvement.

71

the more straight use of the physical models. The modularity of the control physical parameters opensthe possibility to creatively approach the models and achieve effective sound design by also manipulat-ing parameters others than conventional applied forces. From this point of view, the physically-basedsound models can be used to exploit the auditory modality in the interaction design, through a freeyet consistent sound design.Interpolations through various parameters configurations of the friction sound model have been suc-cessfully used in the sonic moka case, in the cutlery and in the bottles sonification in the Gamelunchset up, and recently in the chopping board case study (see Chapter 6).In the spirit of basic design, this approach to physically based models in sound design has been used toadd significant auditory information on the progression or the status of the interactions under study.As a general rule, the most perceptual effectiveness of such a method was found in interpolatingthe interaction properties, since its physical parameters are directly linked to a phenomenologicallayer, while leaving unaltered the resonator parameters. Rather, first experimentations were made onsimply applying interpolation to the frequencies of the resonator object: the expectation of a gesturaleffectiveness as in the musical “glissando” was soon contradicted by a feedback which resulted in anartificial and weak sound.

4.2. Control by means of graphic interfaces, mouse and keyboard

This intensive work of sound design leaded to a more general issue with respect to the graphic userinterface design and possibility to dynamically control more parameters at once by means of theavailable devices, namely mouse and keyboard. Starting from the help patches provided with the SDTpackage, an iterative process of interface refining has been made. This emphasised major issues, oftendealing to Max/MSP efficiency.

4.2.1. Usability versus data flow accuracy in Max/MSP environment

Major efforts were undertaken to provide a more intuitive interface. By means of subpatches andclustering on screen of the UI objects, the size of main patches has been considerably reduced (see forexample Figure 4.11.An open issue for the interface design is the data flow efficiency in Max/MSP environment that largelydepends on the position on screen of the objects and eventually of the subpatches. However this shouldbe solved with the new version of Max. This represents a relevant bottleneck since for the accuratecomputation of most of the externals the control data flow needs to be synchronised at audio rate.A scaling of consistent physically-based ranges of control parameters into more user-friendly andreadable ranges has been introduced as to have a weighted display of the physical values. Sincethe sound models are highly interactive and the Max/MSP environment allows a great flexibility forworking with communication protocols such as MIDI, OSC and Bluetooth, the ranges were initiallyrescaled on the MIDI range (0 - 127), in order to be directly controlled via physical interfaces.While this approach made the interface visualisation more comfortable, it nonetheless introduced issuesof resolution, latency and synchronisation of the UI objects, that became quite cumbersome whenstarting dealing with a large number of instances at the same time, as tested with the Gamelunchenvironment. From this point of view a compromise should be found between data visualisation andcomputational efficiency.Moreover we rethought the nomenclature of the control parameters, by attempting a study for theadoption of a new one, based on perceptive and/or phenomenological attributes. This is still a work inprogress that will take advantages of the perceptual studies and ongoing experimentations at IRCAM.

72

Figure 4.11: EX00 crumpling palette.mxb. Graphical user interface rationalization.

73

4.2.2. Availability of multidimensional UI objects others than sliders, button and knobs

Being the consequent sound highly dependent on the dynamic configuration of the forces, the GUIhas been provided of multidimensional UI objects, for concrete interactive control and sonification,during the sound sketching stage. This introduced some control mapping issues related to the use ofthe mouse on screen and the availability of different responsiveness (non-linear mapping). These weresolved by making use of UI objects, that let the sound sketching noticeably easier and faster. See theexamples in form of tutorial of Chapter 5 for further details.

4.2.3. Management of presets and data interpolation

The [pattrstorage] framework, available in Max/MSP, it’s very useful since it makes possible to savepresets as XML extensions, to map at designer’s choice which object to include in the preset (thestorage window), to make linear or custom interpolations between even non-adjacent presets, to decideobjects priorities within the interpolation process and to have a printable list of the saved values foreach preset (see Figure 4.12. Thus interpolation through reference sonic states becomes available tothe designer’s interaction via GUI or external physical device.

Figure 4.12: EX00 bubblestream palette.mxb. An example of pattrstorage usage, client and storagewindows.

4.2.4. Instantiating and initialisation issues of the Max/MSP externals and patches

A major revision of the SDT externals initialisation is under way. At present the initialisation requiresthe manual data entry, as arguments of the externals, in order to give a first configuration to the model,that is the number of modes for each modal objects and their relative frequency gain and decay, thenumber of pick up points which coincide with the number of outlet (audio signals) of the external,

74

the whole required set of interaction parameters. The ongoing practice showed the need to be ableto modify the configuration in a more flexible way, this especially concerning the add or removal ofmodes. Examples of the possibility of activating different modes via GUI is illustrated in Figures 4.13and 4.14Therefore the planned revision takes into account the default initialisation of the external with a largenumber of modes, to be activated at designer’s choice by using the “activemode $1” message. Moreover,the use of the attributes approach, as recently addressed by Jitter in the Max environment, will beintroduced. As a consequence, the so called bpatcher techniques available in Max (see Max/MSPdocumentation) will be employed to dynamically adjust the GUI and rebuild the necessary audioconnections.

Figure 4.13: EX00 impact.mxb. Example of mode activation via GUI. Two modes are active.

Figure 4.14: EX00 impact.mxb. Example of mode activation via GUI. Six modes are active.

4.3. Physical control issues

4.3.1. Control of the models through a variety of sensors

In order to move from the desktop to the realm of the physical computing, a variety of sensors havebeen used to control the models so to exploit their powerful interactive features. Various sensingeveryday objects have been sketched. For this purpose the Gamelunch environment was set up duringthe last year and a half, for testing the effectiveness and usability of the sound models, towards theresearch and the development activities at UNIVERONA (see Chapter 6 for a detailed description ofthe Gamelunch).Lots of work has been done on the friction, the high level control impact model (crumpling, bothcontinuous and discrete, and rolling) and the bubblestream liquid model.The first attempts made use of a La-kitchen Kroonde board1), that comes equipped with light, flex,acceleration, pressure and magnetic wired sensors, with a 16 bit resolution.As this board showed to be quite reliable, soon issues arose around the wiring limitations: this boardis hardly hideable, and only works with wired proprietary sensors.

1http://www.la-kitchen.fr/kitchenlab/kroonde-en.html

75

Moreover, the cartoonification approach for the sound design stressed the redundancy of such a greatresolution, bearing in mind the scaling operations adopted at the GUI level. Indeed, as far as thesound feedback does not aim at musical expressivity, the sonic augmentation doesn’t need such a greatprecision and accuracy of control mapping, as it happens for the augmentation of musical instruments.For this reasons we made a large use of the Nintendo Wii remote controllers2), that are furnished of3-axis accelerometers with a 8 bit resolution, IR camera and a set of buttons that can be combined atdesigner’s choice. The controller is connected via Bluetooth and is able to dialogue with Max/MSPthrough various third party externals. This way we have been able to sensorize the mobile objects, andconnect up to 7 Wii remote at the same time. Secondly, we started building custom sensing systemsby hacking gamepads: the gamepad is used as sensor board since it provides from 4 to 8 analogueinputs, which comply to the 0-5 volt sensor outputs. The plug & play connection to the USB-port alsosolves any difficulty related to the programming of the board, as it typically happens for the Arduino3

or Wiring cases4, that were also taken into consideration.The work on mapping of the sensed values onto the sound models control parameters mainly focusedon rendering the developed timbral palette in the interactive realm, as a kind of litmus test. For thispurpose a principle of contradiction in terms of perceived effort or materials has been adopted for thesonic interaction design, in such a way to point out the effectiveness of the physically based models.A sort of proof per absurdum.A non-linear mapping has been used to change the sound feedback perceived effort during the action:for instance non-linear mapping has been applied with respect to the forces in the friction model insuch a way to bear on perception of the intensity and extent of the performed gesture. Also, thecrumpling models have been used to sonify the actions exerted on the contents of vessels or bottles,namely stirring and pouring sandy or tinware-like contents. In these cases, the mapping insisted onthe perceived quantity, and finally in combination with the physicality of the sound feedback on theeffort of the action.On the bubblestream model non-linear mapping has been used to control the bubble frequency in sucha way to give a different sensation of the density of the stirred or poured liquid.Major issues of latency and computation arose around the Max/MSP interface for the models anduse of sensors. For this reason we developed two versions of the same patch, one with graphical userinterface for sketching sound and mapping and one with no real user interface, that read the sameXML preset file.Further development will concern the separation of the GUI from the synthesis engine, running theGUI on the Max/MSP runtime and the synthesis engine on the Max application, in order to exploitand optimise the modularity of the models in the Max environment.

4.3.2. Relation between gesture and sound with different kinds of physical mediation

Recent investigations concerned the control of the sound models through different kinds of physicalmediation, namely coupling different sensors applied to different objects to drive the sound synthesis.That is the case of the squeaking fork in the Gamelunch environment: the accelerometers of the forkdrive the interpolation of three sonic reference states and control the loudness of the synthesised sound,while a force sensor applied to the dish senses the pressure exerted with the fork on the surface. Thesound design makes use of the friction model and applies to the actions of pushing and piercing piecesof food on the dish: it provides a squeaking sound feedback that acts on the sensation of softness andelasticity of the materials.Bearing in mind that sound is the acoustic result of interactions with and between objects, the physicalmediation better emphasised the abovementioned relation, while focusing and exploiting the intention-ality and expressivity of the sound gesture. Furthermore, it allowed us to rethink the control mapping

2http://en.wikipedia.org/wiki/Wii Remote3http://en.wikipedia.org/wiki/Arduino4http://wiring.org.co/

76

towards a more natural and embodied link between gesture and sound, consistently with its actionanalysis. In such a way we avoid unexpected or weird sonic behaviours during the interaction.The achievement of the fork case encouraged us to free the physical mediation of the exact configurationof fork and dish and creatively play with any other object on the dish. With a high responsive controlmapping of the pressure sensed values on the friction control parameters, this resulted in the possibilityof having other consistent sonification of gestures, such as squeezing a pepper or scraping the surfaceof the dish with the nails.This leaded to a secondary line of our research: by thinking on the interactive qualities of the everydayobjects that surround us in terms of affordance and solmization, we started challenging the musicalpractice of the “found objects”, whose aesthetics focuses on the use of existing objects that havenot been designed for artistic purposes. Since the transformation of everyday objects into musicalinstruments is quite a common process (music raises from the employment of working-tools, domesticobjects, natural product of human activities or natural objects tout-court), we focused on creativelyexploring and pushing the boundaries of the sound design space, as suggested by the squeezing-peppercase.Even if in a design context, a glance on the artistic and musical practices is of great help when dealingwith sound. When embedded sound and (inter-)action are tightly coupled, the kinaesthetic and tactileexperiences establish a strong perceptive link between sound feedback and the artefact-source.From this point of view, the goal is to challenge the application of a solmization system to sonicallyinteractive everyday artefacts. The aim is to give birth to a new generation of Sonically AugmentedFound Objects [3].

77

5. The Sonic Palette

Parameter-settings identifying a number of timbral categories were provided. In this case, the synthesisalgorithms are activated by random variations of the external forces.This is a work in progress that, at the end of the project, will yield the widest possible coverage ofthe taxonomy as well as further sub-articulations of each timbral family into a variety of particularrealisations.Interpolation through the parameter settings of these different realisations provides a further way forthe enrichment of the timbral palette.

5.1. Low level models

5.1.1. Simple contact sounds [EX00 impact.mxb]

A study on materials has been undertaken, identifying the parameters for glass, wooden and metallicsounds. Also, a wider variety of timbres were obtained by means of interpolation of the specificmaterial timbres.

Glass sounds Roughly, glass sounds are characterised by higher frequency components than metal-lic sounds and lower decay time. With respect to glass sounds, a principal subdivision has to beconsidered for the qualities of the material and the shape of the glass: for instance a crystal glasssounds remarkably different from a glass tumbler. In this case, the immediate perception and mentalrepresentation of the object in relation to the sound is very interesting and effective.From a physical modeling point of view, crystal sounds have a high frequency spectrum, whose typicaltrait is the beat and longer decay time. Impact parameters for crystal glasses and glass tumblersespecially differ for the contact surface shape (alpha), which depends on the local geometry aroundthe contact area. Given an equal stiffness (k) value, glass tumbler sounds require lower alpha values,which is consistent with the usual shape of these glasses (see Figures 5.1 and 5.2 corresponding to thesound examples 02 impact crystal PR1.mp3 and 03 impact tumbler PR4.mp3, respectively).

Wood sounds Wood impact sounds have low decay times and relatively low frequencies. It is quitedifficult to obtain these sounds with only two modes of resonance, since the typical spectrum featuresreside in the attack transient. Indeed, the attack transient shows the presence of energy through thewhole spectrum with a concentration in the low frequencies.However for the sake of the cartoonification approach, wood sounds are quite simple to synthesise:starting from the glass presets, it is sufficient to work on the global frequency factor and the globaldecay factor, by reducing both control values in a range comprised between 0.1 and 0.2 (see Figures5.3 and 5.4). Thus by tuning the hammer mass and the contact surface shape parameters it is possibleto “cartoonify” different kinds of strikes (listen to the sound examples 04 impact WD PR7 8.mp3 and05 impact GL2WD PR5 9.mp3).

Metal sounds Metallic sounds are marked by higher decay times, rough attack transients and typicalbeatings of the not-exactly harmonic partials, as in bronze shapes like bells (see Figure 5.5 correspond-ing to the sound example 06 impact bronze bellPR14.mp3).

79

Figure 5.1: EX00 impact.mxb. Parameter values of the preset nr. 1, crystal glass.

Figure 5.2: EX00 impact.mxb. Parameter values of the preset nr. 4, glass tumbler.

80

Figure 5.3: EX00 impact.mxb. Generating a wooden sound starting from a glass timbre. From presetnr 5 to preset nr 9. Parameter values.

Figure 5.4: EX00 impact.mxb. Generating a wooden sound starting from a glass timbre. From presetnr 5 to preset nr 9. Graphical interface.

81

Figure 5.5: EX00 impact.mxb. Parameter values of the preset nr 14, bronze bell.

Hybrid impact sounds through interpolation of presets The possibility to dynamically interpolateeven non-adjacent presets gave interesting sonic results. Through a number of trials, we were ableto design other material sounds, as e.g. ceramic, to generate hybrid sounds of impacts between twodifferent materials, to observe how spectral fusion behaves when the involved timbres are relativelydistant. Presently, the interpolation is applied linearly, but further improvements will also include theavailability of custom non-linear mapping for each recalled parameter.

Interpolations between same materials The material perception is almost never contradicted. Rather,moving along the interpolation a sensation of exploring the object is given. This is especially effectivewhen the two extreme, i.e. the chosen presets, are physically consistent with the real world. On thecontrary, it gives also the possibility to sonically explore virtual shapes This is well represented in thesound example 07 impact wood interpPR8 11.mp3.

Interpolation between different material The pattrstorage framework allows an interpolation rangeof 0.-1. The behaviour gives noticeably different sound results depending on the material and itsinternal configuration.- Glass and wood: the interpolation tends to give ceramic-like timbres around half of the range 0.-1.Indeed, for other values the resulting timbre approximates towards the closest of the two extremes tothe point that one could argue which object is the striker and which the struck (listen to the soundexample: 08 impact GLWD interpPR4 7.mp3).- Glass and metal: the effect results in a quick perceptual displacement of the timbre towards ametal sound, at an average distance of 0.1 from the initial glass sound. For intermediate values thetimbre moves from glockenspiel-like sounds through different kinds of bells till the target timbre. Inthis case a non-linear interpolation will probably be more indicated.- Wood and metal: the hybridisation of the two mostly results in metallic bar sounds withdifferent configuration, along almost the whole range 0.-1 as one can hear in the sound example:09 impact WDMT interpPR7 12.mp3.Interpolation of sonic reference states can be effectively used as a starting point for designing hybridtimbres, that otherwise would be difficult to achieve. Moreover a parameter-dependent non-linear

82

interpolation could be of great help to improve sound sketching. Finally, each sound can be furtherrefined by tuning the various parameters.

5.1.2. Continuous contact sounds [EX00 friction.mxb]

Thanks to its high interactive and timbral features the friction model has been largely used in theGamelunch environment and other research activities at UNIVERONA (see CHAPTER 3). Startingfrom the reference basic event of rubbing, a deep survey of the friction model potentialities has beenperformed.The comprehension of the friction sound model has been accomplished by parallely considering theconfigurations corresponding to different materials, and the coupled gesture, namely the combinationof the external forces.

Rubbed glass As a thumb rule, a good starting point to generate a glass sound spectrum with thefriction model is to reduce of an octave the correspondent center frequency of the impact model andthen tuning the partial in order to have bright or whistled resonance. Reliable decay times varybetween a range of 0.03 - 0.12 in inverse proportion to the stiffness parameter. Settings the lattergives a primary rubbing character, as in wineglass rubbing (or a glass harmonica) for higher values(see Figure 5.6) or apple grater rubbing for lower values (see Figure 5.7).Interestingly, setting the glass into resonance is not a trivial task and requires some practice andcareful control, just as in the real world.Once set the stiffness or elasticity component, the following two parameters of the friction, namely thebristle viscosity and the linear viscosity, are used to tune the friction resonance. Since the two viscosityparameters respectively affect the sound bandwidth and the speed of timbre and pitch evolution, thecombination of the two shifts the timbre from whistled to more rough sounds: a fast timbre evolutiongives whistled sounds, whereas the bandwidth colours its roughness. Given this raw structure, thedynamic friction coefficient applies as a further bandwidth control, for sound quality tuning. Insteadthe static friction coefficient can be seen as a control of the nuance of the designed gesture, sincefrom a phenomenological point of view, it affects the smoothness of the attack. In the case of therubbed glass category, it represents the resonance response to the exerted external forces. On thecontrary, the effect of the break away coefficient is less evident, while the Stribeck velocity is usedas further tuner of the nuance of the gesture, in terms of sound attack. Finally, the noise coefficientaffects the perceived surface roughness, also providing a sensation of the size of the objects whencleverly combined with decay times(listen to the sound examples 10 friction GL harmonicaPR4.mp3and 11 friction GL graterPR4.mp3).Nevertheless, a main role for the consistency of the output sound hangs on the configuration of theexternal forces, as in the real world where the resonance of a wineglass largely depends on how fastwe move our moist finger around the wineglass rim and how much pressure we exert.

Glass wiping As subcategory of rubbed glass, wiped glass sounds are characterised by lower decaytimes, a greater bristle viscosity (wider bandwidths), and average linear viscosity and static frictioncoefficient (see Figure 5.8 for parameter details). On the contrary, greater and faster pressure differ-ences are required in order to have those typical partly rough and partly whistled resonances thatdistinguish those events such as wiping a window with a buff. Furthermore, from the gestural pointof view the external rubbing force requires a nearly linear and smooth flow. As in the previous case,the speed and the extent of the movement determines the duration of the resonance as one can hearin the sound example 12 friction GL wipingPR14.mp3.

Rubbed plastic As a rough starting point, rubbed resonant plastic sounds can be easily achievedby modifying a squeaking sound, thus reducing the center frequency of the resonator object andbroadening the extent of the external rubbing force over time. A rubbery sound texture is generated

83

Figure 5.6: EX00 friction.mxb. Parameter values of the preset nr 4, glass harmonica.

Figure 5.7: EX00 friction.mxb. Parameter values of the preset nr 1, scraped glass.

84

Figure 5.8: EX00 friction.mxb. Parameter values of the preset nr 14, glass wiping.

whose thickness can be tuned by adjusting the bristle stiffness, the linear viscosity and the staticfriction coefficient parameters. Longer decay times are used to adjust the sensation of the size of thevirtual object and whether it is hollow or solid (see Figure 5.9 corresponding to the sound example13 friction PL rubbingPR27.mp3).

Squeaking A typical everyday sound is the squeak produced by the hinges of a swinging doors. Theprovided examples show squeaking doors of different sizes with different behaviours. Of interest, inthe presets 24 and 25 the repetition pattern for the external rubbing force (set the mapping functionon “linear”) engenders completely different mental images: a wooden bar hung and driven by thewind in preset 24 and the hinges of a swing in preset 25. Figure 5.10 reports the parameters valuesnecessary to obtain the sound of a squeaking door of a cupboard, while Figure 5.11 represents theconfiguration for a sound of a squeaking hinge of a swing. A couple of sound examples are alsoavailable: 14 friction WD squeakingPR20.mp3, 15 friction MT squeakingPR25.mp3. The differenteffects are achieved with very little variations of the bristle stiffness, the linear viscosity and thehammer mass.

Cork screwing A more complex event is the cork screwing, that comes out from the derived squeakingprocess. In this case, the sound response is the result of different combinations of transient and con-tinuous sliding that are achieved by random variations of the external forces. The produced resonantsqueaks, with their typical glissando effect, distinguish the event of screwing the cork in the bottleneck.The provided examples show how it is possible to convey the sensation of the size of the bottle and thecork, from one side, and the reduction of the friction sound, due to the smaller portion of the cork andthe bottle that are in contact as illustrated by the sound example 16 friction GL corkscrewPR17.mp3.What is lacking in the provided example is a dynamic control of the bristle dissipation, as it normallyhappens in the real world. Moreover the perception of this sound event is time dependent, since aftera repeated listening one mostly tends to associate it to a kind of weird glass wiping sound . See Figure5.12 for the parameter configuration of this sound family.

85

Figure 5.9: EX00 friction.mxb. Parameter values of the preset nr 27, rubbed plastic.

Figure 5.10: EX00 friction.mxb. Parameter values of the preset nr 20, squeaking door of a cupboard.

86

Figure 5.11: EX00 friction.mxb. Parameter values of the preset nr 25, squeaking hinge of a swing.

Figure 5.12: EX00 friction.mxb. Parameter values of the preset nr 17, cork screwing.

87

Filing A further case derived from the rubbing basic event is the filing. Some examples of filinginvolving various materials are presented. Filing sounds are defined by very fast sound attacks, quicktimbre evolution, and high stiffness, so as to avoid whistled effects. This can be achieved by combiningthe static friction coefficient with the Stribeck velocity and adjusting the bristle stiffness and the linearviscosity (see Figure 5.13.In some cases the filing vaguely becomes closer to a sawing sound: however the regular pattern of thesound attacks is hardly able to simulate the crumpling behaviour that usually occurs in such events(listen to the sound example 17 friction WD filingPR30.mp3).

Figure 5.13: EX00 friction.mxb. Parameter values of the preset nr 30, filing wood.

5.2. High level models

A variety of sound timbres were produced by means of the high level models of the SDT. Hereunder,the sonic results of the impact-based high level models and of the liquid high level models are presented.

5.2.1. Complex contact sound

Discrete crumpling - [EX00 crumpling palette.mxb] New simulation examples have been addedto the crumpling palette. New addons concern both the design of sound events and materials:frying and popping, walking with wood slippers, different kind of rope stretching, crunching, tear-ing and tinware falling. The parameter settings of some of these sounds are reported in Figures5.14, 5.15, 5.16 and 5.17 and illustrated as well by the sound examples 18 crumpling fryingPR5.mp3,19 crumpling poppingPR6.mp3, 20 crumpling WDslipperPR18.mp3 and 21 crumpling tearingPR16.mp3,respectively.

Continuous crumpling – [EX00 continuous crumpling palette.mxb] In the continuous crumplingmodel, parameter conditioning applies to the exerted force and resistance.This model has been initially provided for the Gamelunch environment (see Chapter 6). Here, theconditioning also concerned the force stiffness and the contact surface shape. In the latter, conditioning

88

Figure 5.14: EX00 crumpling palette.mxb. Parameter values of the preset nr 5, fryingfriction.

Figure 5.15: EX00 crumpling.mxb palette. Parameter values of the preset nr 6, popping.

89

Figure 5.16: EX00 crumpling.mxb palette. Parameter values of the preset nr 18, wood slipper.

Figure 5.17: EX00 crumpling.mxb palette. Parameter values of the preset nr 16, strange tearing.

90

was used to add significant auditory information about the status of the process being sonified (pouring“sandy-like” liquid from a bottle in this case). Two graphical function editors have been added toshape both force and resistance for the sake of the sound sketching. The envelope of the exerted forcerepresents the action and its duration, while the second editor allows to shape the resistance of thematerial in correspondence with the exerted force for a more realistic effect.Among others, the presented palettes for crumpling sounds include fire-like, ice crushing and falling,walking on snow, and a moka spilling coffee-like sounds (listen to the sound examples22 cont crumpl ice crushPR16.mp3 and 23 cont crumpl mokaPR9.mp3).

5.2.2. Liquid sound

Bubblestream - [EX00 bubblestream palette.mxb]Parameters conditioning by random variation within specified ranges allowed to add a more realisticnuance to the sound palette of the bubblestream model. Moreover an interactive GUI, the object, isused to dynamically control the bubble frequency thus providing a direct sonification of the mousepointer on screen.New addons concern dripping sounds, bath filling and the garden hose flow illustrated by the soundexamples 24 bubblestream bath fillPR2.mp3, sound example and 25 bubblestream drippingPR5.mp3(see also Figure 5.18.

Figure 5.18: EX00 bubblestream palette.mxb. Graphical user interface for the preset nr 2, bath filling.

91

6. Interactive Sonification Examples

6.1. GUI-based interactive simulations

A number of examples of GUIs in form of tutorial illustrating the usage of the SDT for concreteinteractive sonification cases were implemented:• Spreading of jam on a toast• Salad spinner• Razor• Corkscrew• Window wiping.

6.1.1. Spreading of jam on toast - EX01 continuous crumpling-Spread Bread.mxb

Objectives: spreading of jam on toast, avoiding the break it. The sound feedback should be continuous,giving information about the exerted pressure on the slice of toast.Sound: continuous control crumpling model - [control contcrump]Possible implementation: flexor sensor if the utensil bends sufficienly, otherwise a pressure sensorplaced at the clutch of the blade in the handle.GUI and interaction: the user can choose one of the available sound from the preset palette. Themouse pointer is used as a metaphor of the spreading knife. The 2D interaction on screen is mappedonto the force parameter and the loudness of the sound feedback. One can move the pointer on thepicture, press the select button and start spreading and releasing. The elapsed time of the mousetracking mode is mapped on the force parameter of the continuous crumpling model according to thefunctions available at user’s choice. This results in the perceptive description of the smoothness of theaction. It is possible to explore the available mapping, from 1 to 5 (softer —> harder). The relativedisplacement (x, y coordinates) of the mouse pointer in tracking mode, over a period of 150ms, isthus mapped on the dynamics of the sound, as to increase the loudness according to the extent of thedisplacement. The GUI for this application is represented in Figure 6.1.

6.1.2. Drying the salad with the spinner - EX01 rolling-Salad Spinner.mxb

Objectives: drying the salad with a periodic and uniform rotary motion. The sound feedback shouldbe continuous, giving information about the status of the drying process.

Sound: rolling model - 4 voices.Possible implementation: accelerometer with memory.GUI and interaction: this example comes with a modified version of the rolling model. In this case,four rolling models are used in parallel, simulating the motion of four balls of different diameter andstiffness. For each ,is possible to configure the single parameters, the relative amplification and theirpanning on the stereo front, so as to have a better perception of the rolling bodies. On the right sideof the GUI, a dial is used as metaphor of the salad spinner. Available interaction settings includethe maximum velocity in the range 0-127 and six different velocity mapping, so as to have differentaccelerations. Once set the maximum reachable velocity, it is calculated as the mean value of the dialvalue over a period of 200 ms. Thus, this value is scaled within the range (0 - maximum velocity).At each velocity detection, a single random value in the range (0-4) is summed to the single velocities

93

Figure 6.1: Graphical interface for spreading jam on a toast based on the continuous crumpling model.

94

in ordert to add a little and reasonable variation. The interface is provided with the current velocitydisplay and the elapsed time during each spinning cycle, to monitor the constancy of the rotation inrelation to the combination of the maximum velocity and relative mapping settings. Adjusting thelatter, together with the size (diameter) and the shape (stiffness) of the balls, results in a differentperceived effort during the spinning. This also bears on the elapsed spin time. Moreover, in order todisplay the information about the drying process, the rolling sound feedback timbre evolves towards adryer sound, by intervening on the contact surface shape parameter of each underlaying impact model.The elapsed time value over an arbitrary range of 0-5000 ms is scaled in range of 0-100. At each cycle,the calculated value is summed to the prior, until the arbitrary sum of 700 is reached. Therefore, therange (0-700) is scaled so as to accordingly variate the contact surface shape parameter of the fourimpact models. A visual display is also provided, with a slider, whose colour gradually changes fromblue (wet) to green (dry), once done. The user can hold the select button of mouse and move the dialclockwise. The GUI for this application is represented in Figure 6.2.

Figure 6.2: Graphical interface for the sonification of a salad spinner based on the rolling model.

6.1.3. Shaving with a razor - EX01 friction-Screw Clean Scrub.mxb

Objectives: shaving with restrained pressure and speed. The sound feedback should be continuousand giving information about the pressure exerted and the speed.Sound: friction model - [friction 2modalb˜]Possible implementation: flexor sensor placed on the body of the razor.GUI and interaction: the user can choose one of the available sound from the preset palette and movethe cursor on the beard area, press the select button and start shaving and releasing. The elapsedtime of the mouse tracking mode is mapped on the pressure parameter of the friction model accordingto the functions available at user’s choice. This results in the perceptive description of the hardnessof the shaving. One can try the available mapping, from 1 to 5 (softer —> harder). The relativedisplacement (x, y coordinates) of the mouse pointer in tracking mode, over a period of 150ms, isthus mapped on the external rubbing force, so as to increase the speed according to the extent of thedisplacement. The GUI for this application is illustrated in Figure 6.3.

95

Figure 6.3: Graphical interface for shaving with the razor based on the friction model.

6.1.4. Wiping the window with a buff

In this situation, a GUI is provided to test on screen the glass wiping sound palette. Five presets areavailable at user’s choice. The user can move the pointer on the window, press the select button andstart wiping. The average relative displacement value of the pointer over a time of 200 ms is calculatedand mapped on the pressure parameter of the friction model, within a reasonable range of (0.1, 0.77).Instead, with regard to the external mapping force, the absolute displacement value of the cursor fromthe center of the window is scaled in the range (-1.58, 1.72). The GUI for this application is depictedin Figure 6.4.

6.1.5. Screwing the cork

A GUI is provided to test on screen the corkscrew sound palette. Five presets are available at user’schoice, simulating the cork slide in the bottleneck. The user can move the pointer to the needle of thedial and track it with different speeds (press the select button, move and release). The average relativedisplacement value of the needle (the dial value) over a time of 200 ms is calculated and mapped on thepressure parameter of the friction model, within a reasonable range of (0.59, 2.67). On the contrarythe pressure parameter is controlled by random variations within the fixed range (0.16, 0.94). TheGUI for this application is represented in Figure 6.5.

6.2. Gamelunch activities

The Gamelunch [2], [17], initially conceived during a CLOSED workshop at ZHdK in Zuerich in january2007, was chosen as a viable workbench for sampling and conditioning the research and developmentactivities at UNIVERONA.

96

Figure 6.4: Graphical interface for wiping the window with a buff based on the friction model.

Figure 6.5: Graphical interface for unscrewing the cork from the bottleneck based on the frictionmodel.

97

Accordingly to the joint approach in CLOSED to the kitchen and domestic scenario, the Gamelunchframework copes with the dining scenario for testing and improving the sound models, towards theprogressive development of the sound design tools (SDT).The Gamelunch, whose name is a crasis between the Balinese “Gamelan” orchestra and the word“lunch”, is a sonically augmented dining table: various sensing devices (force transducers, light andmagnetic field sensors) enable to capture continuous interaction between humans and everyday (din-ing) objects, in order to drive the control parameters of the physics-based sound models. With its setof immediate and natural gestures and actions, the dining scenario sets a fertile context for an inves-tigation of interactive sound generation. Simple everyday actions such as cutting, piercing, sticking,drinking, pouring, grasping, stirring, mixing, have been analysed in terms of source behaviour andgenerated sounds. Thus, the Gamelunch represented a profitable workbench to critically reconsideringthe sound models in terms of their effectiveness in shaping the sonic behaviour of objects.During its ongoing development the Gamelunch passed through different setups, in order to facedisparate issues related to effectiveness, robustness, stability and last but not least portability.

6.2.1. Basic sonic interaction design approach

Complementarily to a source behaviour approach to objects, the reflection also concerned the rela-tional links between objects and human interactions, in terms of interaction gestalts subtended by thecomplexity of the sensory and motor skills required, for instance, when eating with a fork, or pouringwater off a jug. In this point of view, action analysis, segmentation of the continuous coupling andpatterns highlighting found valuable and helpful tools. Moreover, the cultural and social aspects needto be considered, as they act as strong cognitive constraints towards an acceptable understanding ofthe multisensory feedback and relations that revolve around an artefact. Of interest, it was the situa-tion of the sounding fork, whose first sonic augmentation focused on the typical gesture one does whenbringing the pierced morsel to the mouth: the proposed configuration worked in conjunction with theknife, as when one pierces and slices food. In this case, however, the squeaking sound feedback wasapplied by contradiction to the abstraction of the articulatory movement of the elbow. Surprisinglythe effectiveness of the sonic response largely depends on the handling of the fork. This revealed to benot so obvious as contrarily expected. Different styles of handling were observed, supposedly depend-ing on the received education. Nevertheless, the sound feedback seemed to be effective in sharpeningthe difficulty of the task for less-skilled users.Therefore, around the workbench some major themes were highlighted and developed using a basicdesign approach:

• Continuous coupling in the interaction• Contradiction in sonic feedback,• Creation of a consistent hosting soundscape• Performance in cyclic gestures in everyday actions.

Continuous coupling in the interaction In this case, the speculation focuses on the perceptual dis-tance between the perception/action loop and the experience of a physical causality of the interaction,as the key issue for effective sonic interactive artifacts. From a sonic point of view, enactive interact-ing with embodied interfaces, i.e. everyday objects, can be analysed as a three stage process, namelytransmitting energy to produce sound, perceiving the state of the system and its affordances, commu-nicating, through action, a sonic content. Continuous manipulation is one of the key elements for theachievement of an embodied experience and, therefore, for highlighting the role of the perceived soundin terms of functional/informative, emotional and aesthetical qualities: performativity as a gradientof the expressive attributes of the interactive artefact. In the sonic moka exercise, for instance, thecontinuum of the screw connection is divided into three clear state stages of tightness, (i) loose, (ii)tight, (iii) too tight. The sound design implementation makes use of the metaphor of the Schellengdiagram for the different oscillatory behaviours of a bowed string. Thus, the proposed sound feed-

98

back gradually moves through the correspondent stages, by interpolating the timbral qualities of thethree sonic reference states, from a friction sound of glass harmonica through rubbed/wiped glass to asqueaking hinge-like sound. No direct mapping of the sensed force is applied to the control parametersof the sound model, rather it pertains to the shape of the interpolation, in such a way to consistentlyadjust the loudness and width of the resonating regions of interest. The experienced sound feedbackduring the interaction resulted to be very natural, while making the task more engaging and mean-ingful. Moreover, around the theme of the continuous coupling in sonic interaction, the localisation ofthe loudspeaker is another issue which is not to underestimate, since it further affects the perceivedphysicality of the emitted sound.

Contradiction in sonic feedback In the Gamelunch installation a wide set of interactive soundsis designed around the principle of contradiction, thus affecting the coordination of the differentsensory channel, in terms of perceived effort or perceived closed attributes. Following a basic designapproach, experimental sonic counter-actions have been realised for the sake of training experiences andperceptions, exploring the interplay between the sensory channels, developing a logic and a vocabularyfor the sound design tools. Therefore, functional, aesthetical and emotional attributes of the interactivesound are bent, by experiencing contradicting and unexpected sound feedback, in such a way tohighlight the importance of sound in everyday-life acts. By ensuring the energy consistency betweenthe gesture and the physics-based generated sound, the embodiment, i.e. the direct manipulation,is preserved. Nevertheless the distortion of the auditory display ends up to enactively affect theemotional response and expectations in the perception/action loop. Among the others, the goal ofthe Gamelunch is to preserve the feeling of the effectiveness of the gesture, while bending two otherperceptual dimensions, the perceived effort and the material of the objects involved in the interaction.In the jug sonification, the act of pouring water is considered. A solid friction sound is used to sonifyits inclination, therefore giving the feeling of a resisting force that opposes the natural effect of theliquid flowing and lightening of the jug. A case study complying with the contradiction of the materialis that of the action of stirring a sangria.The sonic feedback was similar to that of a tool mixing tinware, oppositely to the fact that nothingsolidwas present in the vessel. Analogously, in the fork realisation, a squeaking/rubber sound feedbackaltered the perceived consistency and quality of the pierced food.

Creation of a consistent hosting soundscape The Gamelunch installation is also provided with aconsistent soundscape, in order to enrich its whole ecologically founded environment. In continuitywith an ecological approach to auditory event perception and modeling, the soundscape organisationexploits by analogy the organisational principles of the Antiprimadonna, literally “anti-queen bee”, abasic design exercise conceived by the representative of the School of Ulm, Tomas Maldonado: theexercise envisages the formal organisation of seven vertical bands of variable width. Among the sevenbands, five can be of different colour, while the remaining two should have different black and whitetextures. The organisational framework requires that none of the bands plays the part of the queenbee. On the level of the sound design, the correspondent corpus of sound events is constituted by threekinds of impact sounds, i.e. wooden, glass, metallic, one burbling liquid sound, two friction sounds,i.e. squeaking, rubbing, and a fire-like aerodynamic sound.In addition to the housing of boards, sensors and cables, the tabletop acts as interface for the browsingof the soundscape: by means of two strip sensors, the exerted touch along the strips respectively controlthe density of the sound events generation and the spatial browsing of the sonic bands. The currentbrowsing is implemented with a simple spatialization based on intensity and direct-to-reverberantenergy ratio. Further implementations will be based on the new bidimensional Digital WaveguideMesh, for the spatialization in depth of sound sources.

99

Performance in cyclic gestures in everyday actions A survey of rhythmic and cyclic gestures inrepetitive tasks has been undertaken in the kitchen scenario with the case study of the chopping board.The considered task is the one of cutting carrots in rondelles. The objective is to support coordinationbetween the two hands, during the progression of the cut. The feedback should give information aboutboth the longitudinal translation of the carrot and the rhythmic action of cutting. For this purpose,the setup consists of three integrated sensing modules: a beat follower of the fracture sounds on thechopping board, a tracking of the vertical acceleration of the knife with respect to the chopping boardand a colour tracking of the section of the carrot being cut. A wide palette of physically-based impactsounds is provided for the rhythmic feedback, together with cross combinations of different materials(glass, wood, metal, hybrids), size and pitch configurations (from non-musical to musical intervals,e.g. the octave, the fifth). A rubbing friction sound is used to inform about the cyclic pattern andnuance of the gesture. Finally, the colour tracking is used to control the internal configuration of theabove mentioned friction sound model, providing a dynamic timbral transformation. The design ofthe continuous sound feedback applies to the interpolation of three reference sonic states, in such away to stress the status of the carrot being cut.The experience and patterns emerging from the Gamelunch can be profitably exploited in the devel-opment of the design scenarios that are being prepared to test measurement methods.

6.2.2. Chronology: Gamelunch set up and sensor mapping issues

Conception and initial mock-upJanuary, 15 - 19, 2007, HGKZ Zuerich, Sound in Interactionworkshop 1

This very first realisation, in the form of a pantomime, constituted the embryonic starting point forfurther speculations around the structuring of the workbench. A first important consideration arisenfrom this pre-alpha release concerned the closed loop emotion-action-sound. Conversely this realisationwas rather musically oriented, as far as the table top worked as interface for the generation of musicalsounds. Here-hence the coinage of the name “Gamelunch”, since the synthesised tones resembledthe Balinese gamelan instruments. In this early implementation, the tabletop was equipped withcontact microphones and worked as raw tangible acoustic interface (TAI). The exerted gestures werecaptured as sound pressure signals, where the amplitude tracking triggered the sound synthesis. As aconsequence, a relevant issue concerned the notion of causality whether to be tied up to a trigger-basedor continuous interaction. Some pictures of this early version are collected in Figure 6.6.

Figure 6.6: Gamelunch. Early setup in ZHdK, Zuerich.

Alpha releasesJune, 25 - 29, 2007, IUAV Venezia, Basic Design Lab 2

This alpha release was shown in the occasion of the closing social event of the Basic Design Lab, atthe IUAV University (see Figures 6.7 and 6.8. Improvements regarded the employ of the first releaseof the sound design toolkit for the sonification of each object.

1http://sonic.wikispaces.com2http://www.newbasicdesign.it/

100

Conversely the concept was refined with the adoption of the aforementioned principle of contradictionas guideline for the sound design.Set up

• The table: acted as TAI interface for hosting an ecological soundscape. Issues of latencies andhardware availability were noticed, seriously undermining the robustness of the configuration.Therefore, for the sake of the investigation the TAI approach was abandoned with the followingrelease.• The knife: the cutting velocity and the gravity were sensed by a wireless 3-axis accelerometer.The sound design made use of the friction sound model. However, because of the relative knowledgeof the model, the sensed data were linearly mapped on a 2 octave harmoniser. As a result, thecoupling of the harmoniser with the continuous movement affected the action, by forcing thegesture towards a uniform behaviour, in counter-phase with respect to the rising glissando.• The fruit bowl: by means of a light sensor, lifting and laying fruits are respectively sonified byconsistent tearing and footsteps sound. The sonification makes use of the crumpling model.• The olive-bones cup: by means of a contact microphone and attack detection, a dynamic splashingsound is fed back for each olive-bone thrown in the cup.• The water jug: the inclination of the jug on the 3-axis is sensed by a wireless accelerometer. Thestreams of data are mapped on the external forces and the bristle stiffness of the friction soundmodel. A braking sound feedback opposes the natural feeling of the jug balance during the liquidflowing and its relative lightening.• The tray: by means of a 3-axis accelerometer and the bubblestream model, continuous drippingand burbling sounds provide a feedback about the inclination of the tray, in the action of balancingit to serve beverages. Moreover, the displacement of the tray in the plane parallel to the soil issonified by a consistent rolling sound.

Sensor mapping issuesThis release stressed the difficulty of a reliable robustness of a TAI approach. We used A La kitchenKroonde board with wired sensors. Issues concerned the wiring limitations: this board is hardlyhideable, and only works with wired proprietary sensors. Additionally, the effectiveness of the soundinteraction was undermined by a linear mapping of the sensed data on the control parameters ofthe models, stressing the need for an in-depth analysis of the thresholds. On the software side, theavailable Max/MSP externals required a great CPU consumption. Three laptop were used to drivethe whole environment.

Figure 6.7: The Gamelunch in the Basic design workshop in Venice.

101

Figure 6.8: Gamelunch: The fruit bowl in Venice.

Beta 1 releaseNovember, 19 - 24, 2007, Grenoble, Enactive/07 3

The following release was shown during the conference Enactive 07, in the Enaction in Art section.A great effort was made to improve the software efficiency and control mapping. For this purposea lighter and more efficient version of the bubblestream was implemented. Also, a prototype of thecontinuous control crumpling has been provided. New objects were added. During the exhibition theimproved effectiveness of the whole environment was underlined by the attempts of various users, toplay the Gamelunch as a musical instrument. This leaded us to approach the embodied sound of theeveryday-life objects in terms of the available and natural set of gestural vocabulary, that introducedus to the complementary research on the Sonically Augmented Found Objects (SAFO): augmentedeveryday objects that become potential musical instruments [3].Set up

• The table: the wired sensors were hidden under the tablecloth. The loudspeakers and the sensorboard were placed under the table, to respectively localise the sound emission and hide as manywires as possible. On the other hand three objects, namely the salad bowl, the olive-bones cup,and the tray, were equipped with embedded small loudspeakers to improve the realistic effect ofthe provided sound feedback.• The knife: improvements concerned a proper sound design and exploitation of the friction model,by mapping of the acceleration data on the external forces and introducing the interpolation thereference sonic states as auditory display of the inclination of the blade around its Z axis: as thearm moves forward and backward and the wrist twists around its axis, the action of cutting issonified by plastic resonances on a wrinkled surface. The sound feedback affects the action bydistorting the perceived effort and the consistency of the material being cut.• The fork: a continuous friction/squeaking sound sonifies the action of lifting the fork while eating.Both the torsion of the wrist and the lift of the arm are bent by a sensation of the movement itselfas the one of a door hinge. The sensation of weight of the fork is warped, as well as a misleadingsensation of an elastic force applied at the upper end. Interestingly, the effectiveness of the soundfeedback largely depended on the ability of the single users. Less skilled users seemed to be moreawkward during the interaction.• The liquid salad bowl: by means of a light sensor and the bubblestream model, continuousdynamic sounds of dripping and boiling are coupled with the action of stirring and mixing thesalad. A sensation of softness and inconsistency is fed back, affecting the percept of the density

3http://acroe.imag.fr/enactive07/ and http://www.youtube.com/watch?v=-gbAjeLD7MY

102

and weight in the action of stirring. A small loudspeaker has been embedded in the false bottomof the bowl, to provide a greater enrichment of the interactive experience.• The crumpling sangria bowl: stirring and mixing actions are considered. The rotation of theladle is sensed by a magnetic sensor applied under the tabletop. Thus, the sonification makes useof the crumpling model. A dense granular sound, as the one of the sand, distort the perceptionwith a sensation of a heavier but shifty material.• The bottle set for vinegar and oil: the bottles are equipped with a 3-axis wireless accelerometer.The Y axis is mapped on the density of impact. The sonifications propose an electric/gravel soundfor the vinegar and an oily/metallic sound for the oil. Both sound feedbacks are generated by thecontinuous control crumpling model.

Sensor mapping issuesMajor issues concerned the stand-by states of the objects and the finding of solutions to avoide the un-desired feedbacks for the inconsistent manipulations. That was the case of the wireless accelerometersplaced in the mobile objects and the light sensor housed in the salad bowl. This issue introduced theneed to a strong conditioning of the control parameters of the sound models, in combinations with anefficient filtering or the gesture data profiles. Moreover, alternative strategies for the implementationof the whole set up have been in-depth reconsidered, to guarantee a greater robustness, effectivenessand portability. Also, the embedding of the sound diffusion systems has been taken in account inorder to improve the immersiveness and embodiment characteristics of the Gamelunch dining envi-ronment. In Figures 6.9, 6.10 and 6.11, one can see the reaction of the public, while interacting withthe Gamelunch.

Figure 6.9: Enactive07, Enaction in Arts, Grenoble. A visitor playing with the Gamelunch.

Beta 2 releases April, 1, 2008, Venezia, pesce d’aprile and April 5 - 10, 2008, Firenze, ACM CHI4

The beta release was shown in Florence, during the CHI 2008 conference, within the Interactivitysession. The whole environment has been extensively revised and debugged and the object designsensitively improved as depicted in Figure 6.12. The current set up includes a dedicated table, withinwhich we embedded force, light and magnetic sensors, two input boards, a five channel diffusionsystem (see Figure 6.13). A considerable work has been made on both interaction and sound design,and software development.Set up

4http://www.vimeo.com/879260April and http://vimeo.com/1337278

103



Figure 6.12: The Gamelunch sonically augmented objects as presented at the ACM CHI conference2008, Interactivity session in Florence

104

• The table: for the purpose of having a stable and portable configuration, a common table hasbeen hacked by embedding in the tabletop the sensors, the input boards and the sound diffusionsystem. Embedding the latter revealed to be a very valuable choice since the better localisationof the sound sources enhanced the embodied qualities of the Gamelunch. Further, this releasemarked the passage to use of hacked USB gamepads as input board for the sensors.• The fork: the sonic interaction focuses on the action of pushing and piercing. The squeakingfriction sound bends the percept, by giving a sensation of softness and elasticity in materials andactions. A more complex mapping concerned the conditioning of the acceleration data with thepressure exerted on the force sensor embedded in the tabletop, just under the dish. Thus it’sbeen possible to avoid the squeaking sound for any inclination of the fork when not in contactwith the dish. Conversely the acquired data have been cleverly coupled with control parametersof the friction model, namely the acceleration on the external rubbing force, and the sensed forceon rubber and the loudness. Finally, the rotation around the Z axis dynamically interpolates thetimbral qualities of three reference sonic states, with further enrichment of the fork expressivity.• The knife: the improvement regarded not only the control mapping but also the sound design.The perception of cutting is given by the oscillatory back and forth movement combined with thechanging of height. The Y axis acceleration data are mapped onto the external rubbing force,while the Z axis data drive the interpolation of three timbres. A sound as the one of filing agummy-plastic material affects the sensation of the resistance of the cut material. Moreover thedifferent timbres guide the inclination of the knife.• The water jug: the sound design enhanced the timbral evolution of the braking sound, by addingthe interpolation of two timbres on the Z axis, in such a way to better highlight the perception ofa force opposing the natural effect of pouring a liquid.• The soup bowl: the action of mixing a liquid encompasses the grasping of the ladle with a hand,the lift of the ladle, and finally the torsion of the wrist while the ladle is suspended. The perceptionof stirring is guided by the density and weight of the content. A hard tinware crumpling sound,where the density of the impacts is controlled by the ladle, feeds back a sensation of harder andfatiguing actions affecting the perception of the material consistency, in such a way to contradictthe liquid quality of the soup.• The salad bowl: different sets of mapping and sound presets have been produced to dynamicallyexploit the bubblestream model, towards a sensation of softness and inconsistency.• The bottles: improvements were made by conditioning the resistance and the contact surfaceshape parameters, according to the inclination. Free yet energetically consistent sonificationssynaesthetically involve the sound identities of the liquids. The continuous sound feedback informsquite accurately about the liquid flow, especially for fast pouring actions. Interestingly, the shapeof the bottles, their colouring, in conjunction with the proposed sound identities prompted varioususers to shaking and squeezing the bottles, as it was perceived more natural or immediate thanpouring.

Figure 6.13: The Gamelunch sonically set up as presented at the ACM CHI conference 2008, Interac-tivity session in Florence

105

7. Acoustic Depth: Rendering Auditory Distance

Many interesting interactions with everyday objects are based on the exploration of a one-dimensionalspace. When sound emission is embedded in the object itself, traditional spatial layouts, as achievedthrough panning, cannot be implemented. Human perception of acoustic depth, however, can be ex-ploited as it relies on cues that can be deduced from a monophonic sound source. The experimentson rendering acoustic depth, and measuring how ensembles of sounds can be sorted out by controllingtheir virtual distance from the listener, are aimed at making this spatial augmentation of objectspossible and reliable.

Recent studies have shown the ability of physical modeling of acoustic propagation to simulateacoustical environments. Of interest is the study conducted by Fontana and Rocchesso [8] whichhas underlined the effectiveness of a Digital Waveguide Mesh (DWM) modeling a rectangular paral-lelepiped to provide acoustic depth cues.

7.1. The Rectilinear Digital Waveguide Mesh

The proposed model for rendering distance cues of sound sources consists of a rectilinear two-dimensional mesh whose digital waveguides simulate acoustic wave transmission between each internaljunction. Each waveguide models the wave decomposition of a pressure signal p into its wave compo-nents p+ and p−, and each lossless node scatters 4 input signals coming from orthogonal directions,p+1 , ..., p+

4 into corresponding output signals p−1 , ..., p−4 (see Fig. 7.1). Reflections from the boundaries

Figure 7.1: Zoom on a node which is connected to other nodes via waveguides 1, 2 and 4. Waveguide3 leads to a partially absorbing boundary. Triangles filled in black represent oriented unit delays.

are modeled by first-order Digital Waveguide Filters (DWF), whose coefficients b0, b1 and a1 (see Eq.7.1) may be tuned to model specific reflective properties [8]. The filter function is implemented as adirect form II transposed structure such that the input-output description of this filtering operationin the z-transform domain is given by:

Y (z) =b0 + b1z

−1

1 + a1z−1X(z) (7.1)

The number of nodes along the mesh width and length can be converted into the correspondingmembrane dimension once the propagation speed of sound and the sampling frequency of the simulationhave been determined. Indeed the digital waveguide theory states that the sampling frequency fs of a

107

bidimensional mesh is determined as [10]:

fs =c√

2dW

(7.2)

where c is the nominal wave propagation speed (343 m/s for the speed of sound in air), and dW thespatial sampling interval or the length of each waveguide. Note that this relation is exact only fordiagonal propagation and at f = 0 because of the dispersion error (some spatial frequencies travelslower along the mesh). From Equation 7.2 we can then deduce the length dW of each waveguide.

In the following, we consider a DWM with the following properties:• sampling frequency: 22.05 kHz (a downsampling factor of 2 is used to reduce the CPU load)• dimensions: 2.64×0.132m2, corresponding to 121×7 nodes• coefficients of the filter simulating the two ends: [b0 = −1, b1 = 0, a1 = 0]• coefficients of the filter simulating the walls: [b0 = 0.95, b1 = −0.1, a1 = −0.1]

7.2. Acoustical Properties of the Membrane

In order to investigate the auditory distance cues inside the virtual environment, impulse responsesare computed at different distances on the membrane. The sound source is assumed to be point-wiseand is located on the y-axis of the mesh, on the fifth node. Due to some numerical artifacts in DigitalWaveguide Mesh simulations which affect the responses especially in the proximity of the mesh, it isunadvisable to use the first two lines at both ends of the mesh. Measurements of impulse responsesare carried out at different distances from the listening point, on the y-axis of the membrane. Referto Fig. 7.2 for the source and measurement positions. Simulation of the listening environment was

Figure 7.2: The virtual membrane with the sound source and the measurement positions. All sizes arein centimeters.

carried out in Matlab. Figure 7.3 shows the frequency responses up to 5 kHz, measured respectivelyat 0.55 m and 2.31m on the virtual membrane .

7.2.1. Overall Intensity Level

Figure 7.4 shows the variation of the total energy with distance. By comparing with the energydecrease in open space, characterized by the well-known 6 dB law, it can be seen that the overallintensity on the membrane decreases significantly less. This behavior allows to hear even the farthestsound sources at any location on the membrane. In addition, the volume can be manipulated by users,which might make the intensity cue unreliable for judging distance. Another reason for limitating theintensity cue for distance judgment is that the level of direct sound varies both with distance and withthe energy emitted from the sound source, so that the listener needs some a priori knowledge aboutthe sound source level in order to evaluate its egocentric distance.

108

102 103!80

!60

!40

!20

0

20

magnitude (dB)

frequ

ency

(Hz)

102 103!80

!60

!40

!20

0

20

magnitude (dB)

frequ

ency

(Hz)

Figure 7.3: Frequency responses up to 5 kHz on the membrane . Top: 0.44m. Bottom: 2.20m.

0 0.5 1 1.5 2 2.5!25

!20

!15

!10

!5

0

distance (m)

aver

aged

mag

nitu

de (d

B)

7eshReference open space

Figure 7.4: Average magnitude of the impulse response as a function of distance. Solid line: 2D mesh.Dashed line: Reference open space.

7.2.2. Direct-to-Reverberant Energy Ratio

The virtual acoustic space we have designed aims at rendering depth information mainly thanks tothe direct-to-reverberant energy ratio cue. For each impulse response, the delay of the direct sound isdeduced from the distance between the sound source and the listening point, and is removed from theimpulse response. Afterwards, the direct energy is integrated among the first 2.5ms of the delay-freeimpulse responses, which approximates the duration of Head Related Impulse Responses measuredin anechoic conditions and therefore captures the direct path of the sound signal [12]. Finally, thereverberant energy is calculated from the tail of the delay-free impulse responses. Figure 7.5 shows thevalues of the direct-to-reverberant energy ratios for different distances on the mesh. For comparison thedirect-to-reverberant energy ratio v was computed for a natural environment, modeled by Zahorik withthe function v = −3.64 log2(r) + 10.76 [26]. The two curves follow the same trajectory, suggesting thatthe direct-to-reverberant energy ratio in the virtual environment follows a natural behavior. Moreoverthe values of the ratios are much lower in the 2D mesh than in the natural auditory space, whichmeans that the amount of reverberation is exaggerated in the virtual environment.

The other known auditory distance cues, namely the Interaural Level Difference (ILD) and the spec-trum, are not provided by the present model. This is motivated by their relatively weak contributionto distance perception. First ILDs, which arise due to intensity differences between the two ears, arenull on the median plane and therefore will not provide any distance information for sound sourcesdirectly in front or behind the listener [20] as it is the case in our virtual environment. About the

109

0 0.5 1 1.5 2 2.5!15

!10

!5

0

5

10

15

20

distance (m)

Dire

ct!t

o!re

verb

eran

t ene

rgy

ratio

(dB)

Rectangular meshApproximation of a natural environment

Figure 7.5: Direct-to-reverberant energy ratio as a function of distance. Solid Line: 2D mesh. Dashedline: Approximation of a natural environment.

spectrum changes due to the air attenuation of high frequencies, they occur for very large distances(superior to 15 m) which lie beyond the available length of the mesh. As a result, the only piecesof information about the distance of a sound source in the mesh are the intensity and mostly thedirect-to-reverberant energy ratio.

7.3. Externals

7.3.1. Sampling rate

Due to the heavy computational load of the model, a downsampling may be required prior to thecomputation in the mesh, set by the downsampling factor. Inside the external, only decimation isperformed such that an anti-aliasing filter is required at the audio input(s). After computation inthe mesh, an upsampling is performed inside the external using the library libsamplerate (alsoknown as Secret Rabbit Code), by Erik de Castro Lopo (See the libsamplerate home page: http://www.mega-nerd.com).

7.3.2. Parameters of the DWM

The user controls the dimensions of the mesh, the listening position, and the sound source(s) posi-tion(s). Properties of the boundaries may also be modified by varying the coefficients [b0,b1,a1] of thefirst-order filters modeling the boundaries along the two axes of the mesh, implemented as

Y (z) =b0 + b1z

−1

1 + a1z−1X(z) (7.3)

7.3.3. Use

Two externals are provided:

1. mesh2D static∼ : suitable for rendering one or more static sound sources located at differentpositions from the listener.

2. mesh2D dynamic∼ : suitable for rendering one moving sound source along the depth dimension,i.e. one approaching/receding source. In this case, the computation in the mesh is made efficientby switching the sound source and the listening positions, such that the measurement positionvaries at runtime and not the position of the source.

Refer to Part 1 Section 3 for more details on the externals.

110

http://www.mega-nerd.com

http://www.mega-nerd.com

Figure 7.6: The Wii Remote is mounted on a trolley and is pushed horizontally to throw the ball.

7.4. Applications

7.4.1. DepThrow

DepThrow is an interactive audio game designed around the auditory perception of distance andthe use of physics-based models for simulations of the dynamics, the sound source, and the acousticalenvironment [15]. The game consists in throwing a virtual ball inside a virtual open-ended tube whichis inclined. The task is to keep the object inside the tube, in other words the user should adjustthe initial velocity applied to the object such that the latter does not fall out at the far end of thetube. The position of the object inside the tube is provided by continuous audio feedback, and userperformance is closely related to its ability to perceive the dynamic distance of the object in the virtualtube. Therefore, this game represents a potential tool for exploring the usability of auditory distanceinformation in interaction design.

The game prototype has been developed as a complex Max/MSP patch where three interactingphysics-based models enable to simulate, respectively, the dynamics of the environment (an inclinedplane), the sound source (a rolling ball), and to render the acoustical properties of the environment(an open-ended tube). The action of throwing is performed using a Wii Remote1 which provides a3D accelerometer, a vibration actuator, several buttons, and Bluetooth connectivity. As shown inFig. 7.6, the Wii Remote is mounted on a trolley. To throw the ball, the user pushes the trolleyhorizontally and presses the A button of the Wii Remote to release the ball The three physics-basedmodels are connected in a top-down chain structure. Figure 7.7 shows the main Max/MSP patch. Thethrowing force applied to the virtual ball is captured (gesture mapping) by means of the accelerometerand provides the input to the computation (integration) of the initial velocity with which the ball isgetting thrown inside the inclined tube. Then the initial velocity provides the input to the simulationof the inclined plane. The resulting velocity of the ball moving along the inclined plane drives thereal-time synthesis of a rolling sound, which constitutes the audio input to a bidimensional DigitalWaveguide Mesh (simulation of a tube), while the distance currently covered controls the distanceparameter of the mesh. When the object comes back to the user’s hand, the vibration actuator of theWii Remote provides tactile feedback, whereas a bouncing sound is simulated when the ball falls overat the far end of the tube (See also Part 1 Section 3.0.3 for the description of the Max/MSP patch).

7.4.2. Audio-Haptic Interface

Spatialization of sound sources in depth allows a hierarchical display of multiple audio streamsand therefore may be an efficient tool for developing novel auditory interfaces. An interaction tool

1The handed wireless controller for Nintendo’s Wii console.

111

Figure 7.7: The main Max/MSP patch of the audio game DepThrow.

based on rendering distance cues for ordering sound sources in depth was developed [6]. The userinterface consists of a linear position tactile sensor made by conductive material. The touch positionis mapped onto the listening position on a rectangular virtual membrane, modeled by a bidimensionalDigital Waveguide Mesh and providing distance cues. Spatialization of sound sources in depth allowsa hierarchical display of multiple audio streams, as in auditory menus. Besides, the similar geometriesof the haptic interface and the virtual auditory environment allow a direct mapping between the touchposition and the listening position, providing an intuitive and continuous interaction tool for auditorynavigation.An interface for audio browsing was implemented using the aforementioned tool [4]. By the touchposition on the ribbon, the user changes her/his relative distance to four equally spaced virtual speakersplaying simultaneously different audio samples. The virtual environment acts as an audio windowwhich can be moved along the whole playlist with the knob of a MIDI controller. The setup may beseen in Fig. 7.8. A user study shows that users succeed in the search task, and enjoy the interface.

7.4.3. A Peculiar DWM Geometry for Linear Auditory Distance Estimates

Auditory distance perception is affected by the so-called “auditory horizon” effect, defined as thetendency to underestimate far away sound sources. We have modeled a virtual listening environmentwhose objective is to provide a linear relationship between perceived and physical distance [5]. Thesimulated environment consists of a trapezoidal membrane, modeled by a bidimensional Digital Waveg-uide mesh, with specific absorbing properties at the boundaries. The physics-based model simulatesacoustic wave propagation in this virtual space and provides auditory distance cues, namely intensityand reverberant-to-direct energy ratio. Simulations predict the linearity of the psychometric functionrelating physical distance to perceived distance, computed from Bronkhorst and Houtgast’s model.A listening experiment was carried out, involving 9 subjects and using a procedure inspired by theMUSHRA test. This methodology was developed for the subjective evaluation of audio coding systemsand allows the comparison of high quality reference sounds with several lower quality test sounds [25].

112

Figure 7.8: Picture of the setup for the audio browsing user study.

In our experiment, the reference is defined by the sound source located at the shortest distance (1 m),and tests sounds consist of 8 stimuli at distances ranging from 1 to 8 m. Results show that linearityis mostly achieved for nearby distances, when changes in distance evoke noticeable variations in theintensity cue. This promising result suggests that the present virtual environment is a potential toolfor providing a direct mapping between perceived and physical distance in auditory interfaces.

113

Part III.

A Tool for Sound Designers

115

8. The High Level User Interface

8.1. Introduction

Being the SDT a set of physics-based synthesis algorithms, the parameters involved in the modelingshould in principle be easy to understand since they have a physical counterpart, and should be easyto control because our sensorial experience should mediate the action-reaction patterns to which theyrelate [7]. Nonetheless, as discussed in Part 2 of this deliverable, the SDT models are usually describedin terms of detailed physical relations and parameters that not always are accessible or understandableto the non specialists. Moreover, the nonlinear nature of the phenomena under observation maysometimes lead to numerical schemes that does not reflect the behavior of the real systems in the wholeparameter space. These problems were partially tackled by UNIVERONA during the previous year ofactivity as illustrated in Chapters 4 and 5. In this part of the deliverable two major improvements ofthe SDT in terms of front-end are presented, aiming at making SDT an easily accessible and flexibletool for sound designers: 1) a high level interface based on a taxonomical organization of the soundpalette offered by the SDT with many facilities in terms of external connectivity and polyphonic useof the models; 2)an auditory-based graphical tool for a perceptual oriented exploration of the SDTtimbral space. The first point is the subject of this chapter, while the latter one will be discussed inChapter 9.

8.2. GUI implementation and description

The SDT passed through an iterative refinement process in order to provide effective tools specificallyaddressed to interaction designers. The refinement process aimed at 1) providing a common front-end that takes into account the ecologically-founded framework of the SDT; 2) providing simpleand intuitive GUIs without requiring neither an in-depth knowledge of the Max/MSP environmentnor about physics-based synthesis techniques; 3) allowing an effective use of the SDT in real-timeinteractive applications.

8.2.1. Navigating the SDT package

The framework of the SDT is organized following a proposed taxonomy of everyday sounds, whichis based on Gaver’s studies [9]. Figure 8.1 shows the proposed taxonomy: the hierarchy is establishedfrom low-level sound events to more complex, patterned or compound ones. The corresponding low-level sound models are presented at the bottom of the graph, while the second level shows basic eventsand sound textures straightly derivable from them. Processes that can be related to temporal patternsof basic events and textures are presented at the third level. Lastly, the top level contains severalexamples of the implemented simulations, while dashed connections represent expected dependenciesfor the simulations still to be developed (see CLOSED deliverable 4.1 for more details).

When the SDT application starts, a front-end shows-up displaying the available sound models amonglow-level basic events and sound textures, while the derived processes can be obtained by means oftheir proper calibration and combination.

8.2.2. Clarity and intuitiveness

Usually, when first approaching a software application, users demand for clear and simple interfacesthat give effective feedback on the control parameters and the state of the system, and this is especially

117

Figure 8.1: The proposed taxonomy of everyday sounds.

Figure 8.2: High-level interface for the impact model: physical parameters, temporal control andtimbral palette sections.

important in real-time applications.Starting with the impact and the friction models, we defined a template for the GUI size, the

position of the control parameters, and the use of colors (see figure 8.2). Secondly, we rendered themeaning of the physical values more understandable and usable: sliders and dials have been scaledinto a straightforward and readable 0-100 range. Also, the nomenclature of the physical parameterswas further elaborated in order to stress the corresponding perceptual effect. This finalized the processalready introduced in Chapters 4 and 5.

A major intervention concerned the partition of the control parameters of the sound models betweenlow- and high-level interfaces. The high-level interface is intended to give access to the most relevantcontrol parameters that can be directly manipulated by means of external devices (e.g., sensors). Tothis end, high-level interfaces are now composed by three main sections: 1) physical parameters; 2)temporal control; 3) timbral palette. The physical parameters section encompasses the most direct and

118

Figure 8.3: The low-level interface for the impact model, gives access to all the available parameters.

effective parameters: for instance, in the case of the impact model, it includes the material elasticityand the shape of the contact surfaces (relative to the interaction), the hammer mass (relative toresonator 1), and global factors of frequency, decay time and gain (relative to resonator 2). Dependingon the sound model, the temporal control section offers sequencing tools that allow to simulate gesturesor temporal patterns. As an example, in the friction model this section refers to the velocity and therhythm of rubbing, and the duration of the exerted pressure. Finally, the timbral palette section allowsto manage, store, save, recall, delete or interpolate sound presets. On the other hand, the low-levelinterface gives access to all the available controls of the model, for further tuning and refining thesound (see figure 8.3). Among the others, it is possible to activate, desactivate or mute the modesof resonance and modify their individual frequency, decay time and gain. Since physics-based soundsynthesis generally requires the user to tune large sets of variables, the new GUI is, thus, intended toprovide him or her with a functional hierarchy of parameters that facilitates their intelligibility whileretaining the access to the low-level controls.

8.2.3. User-centered features

The aim of the SDT is to provide interaction designers with a sound design application which runson common real-time environments such as Max/MSP and Pd, and that can be easily exploited intheir sketching activities. In order to do this, we provided the patches with a set of dedicated side-toolsthat allow to easily manage projects when working with sensing devices.

For this purpose, a deep work of modularization was undertaken in order to clear some majorissues already highlighted, namely the choice of a global structure for the Max/MSP patches, the dataflow efficiency, and the synchronization of timing objects working at audio rate that is required bysome sound models. The modularization process concerned the package organization in folders andsubfolders, as shown in figure 8.4, in order to have a general and clear framework of sub-patches thatcould be easily accessed in view of further developments and updates. Every time the SDT front-end patch is started, a temporary search path is generated and automatically stored in Max/MSP’s

119

Figure 8.4: The SDT folders tree: Images, sub-patches, general preferences, presets and sound banksare respectively located in their subfolders of pertinence. The Library folder is further split into asmany sub-folders as the number of available sound models and contains all patches and abstractions.In the Presets folder, sub-folders that refer to each sound models contain separated lower level folderswhere Temporal control and Timbral palette presets are saved as XML file. Finally, in the Sound foldersound, samples can be recorded as AIFF files.

Figure 8.5: The main SDT window showing the taxonomical organization of the synthesis models.

file preferences: this way each basic module (i.e. resonator controls, physical parameters, temporalcontrols, etc.) located in folders and subfolders is recalled in the higher-level patches.

As a result, the new high-level framework of the SDT environment solves some important bottlenecksin terms of usability, by providing 1) a dynamic and polyphonic instantiation of each sound model,2) a dynamic control of the loaded sound model through MIDI and OSC protocols, and 3) a dynamicmanagement of the sound palette. Figure 8.5 shows the SDT front-end. Figure 8.6 shows the soundmodel manager patch, which is loaded for each sound model, once it is chosen from the SDT front-end.

The manager patch allows to activate the sound model. As shown in figure 8.8, multiple independentinstances of the same sound model can be loaded and added in the same working session, which meansthat a sound model can be used in a polyphonic, while connections, send and receive objects in thepatch are automatically and dynamically rearranged. Indeed, in the sound model high-level patches,all variables and connections are supplied as changeable arguments: once the model is added in themanager patch, a numerical value, according to the position of the loaded model in the ascendingorder of the manager menu, is automatically added to all the arguments of the loaded patch in orderto initialize it and build the local connection chain. Also, it is possible to remove or restore the soundmodels at user’s choice.

120

Figure 8.6: The manager patches allow some basic controls of the loaded instances: adding, removingor restoring a sound model instance, recalling a MIDI or OSC dedicated control patch for each singlemodel instance, accessing volume controls.

The manager patch comes provided with some additional features: independent MIDI and OSCcommunication patches, volume control and audio recording tools. These are automatically generatedand grouped in a separated window so as to have an indicator of the global mix. Figure 8.7 shows theMIDI and OSC control patch for the instance 1 of the friction sound model. In general, the availablecontrollable dimensions of a sound model correspond to the parameters shown in its high-level interface,since they have been chosen for their major effectiveness in shaping sonic behaviors.

Finally the management of the sound palette as presets makes use of the Max/MSP object pat-trstorage. Thus, sound configurations can be stored and saved saved as XML files, thus allowing themto be read, modified or generated also from other XML-compatible software. In addition, pattrstorageallows to make interpolations between presets (i.e. between values of parameters), providing furtherpossibilities for interactive manipulation of sounds and for a richer timbral palette.

Far from being exhaustive, these new features help solving a set of bottlenecks encountered andpointed out by the partners in the CLOSED project when making use of the sound models: patchesarrangement, control through a variety of sensors, sound mixing and so forth. The aim was to pro-gressively make the SDT as “plug & play” as possible to effectively exploit the sonic interactivecharacteristics of the physics-based sound synthesis paradigm.

121

Figure 8.7: MIDI and OSC control patches of instance 1 of the friction sound model. Controllabledimensions correspond to the parameters shown in the high-level interface, namely the rubber mass,the bristle stiffness, the bristle viscosity, the surface roughness, the global frequency, the global decay,and the global gain.

Figure 8.8: Three independent impact models.

122

9. Sonic Landmarks for the Exploration of theSDT Timbral Space

A graphical tool for timbre space exploration and interactive design of complex sounds by physicalmodeling synthesis is proposed. It is built around an auditory representation of sounds based on spikefunctions and provides the designer with both a graphical and an auditory insight. The auditoryrepresentation of a number of reference sounds, located as landmarks in a 2D sound design space,provides the designer with an effective aid to direct his search along the paths that lie in the proximityof the most inspiring landmarks.

9.1. The SDT explorer

We propose a graphical tool for the exploration of the SDT timbre space and the design of complexsounds that is intended to facilitate the sound designer. It is aimed at exploiting the auditory rep-resentation of sounds based on spike functions and provides the designer with both a graphical andan auditory insight that may be used in place of, or combined with, the set of low-level physical pa-rameters of the models. The graphical tool adopts a sonic landscape paradigm, in which the auditoryrepresentation of a number of reference sounds can be located as sonic landmarks in a 2D sound designspace, and provides an effective aid to direct the search along the paths that lie in the proximity inbetween of the most inspiring sonic landmarks.

The use of terminology and metaphors referring to the environment and landscapes has a ratherold tradition in the field of sounds perception, especially when referred to the perception of ecologicaland everyday sounds. In the late 70’s, the Canadian composer R. Murray Schafer introduced the term“soundscape”, defined as the auditory equivalent to landscape [19], and Barry Truax published hisHandbook for Acoustic Ecology [23]. Our use of the terms “sonic landscape” and “sonic landmark”however refers to a particular organization of sounds in a 2D sonic space.

The aim of the tool proposed here is to add to the tuning interfaces introduced in Chapter 8 aperceptual representation output layer based on spike functions and a tool to perform parametricinterpolations using reference auditory representations in the perceptual space. The representationof sounds based on the spike functions provides the designer with both a graphical and an auditoryinsight. The latter point is of extreme interest since it is based on a sort of ”sonic representation ofsound”. The peculiarity of the spike representation is, in fact, to be lossy. This aspect turns to bean advantage, whereas the reconstructed sounds maintain the temporal articulation of the originalones but they are decolored from their original timbral characteristics. Any reconstruction of a soundhas a “watery character”, so that all the sounds become timbrically similar, even if distinguishableaccording to their temporal structure. In this way, the sound designer has at disposal a kind of“auditory-radiography” of the sounds that can be compared as sonic archetypes, helping the designerin her/his sound space exploration.

9.2. Design and implementation of the Spike-based parameterinterpolation tool

The interactive interpolation tool presented is organized as a client-server distributed application, inwhich the client side hosts the graphical front-end based on the auditory spike sound representation,and the server side hosts the SDT audio synthesis engine. In the following, we provide some details

123

on the spike analysis framework used to graphically represent the sounds, and the properties of theinterpolation in the 2D space.

In a classical signal representation approach, overlapping discrete blocks are used. However, in thismethod, in particular for non-stationary signals, a small shift of a block can cause a large change inthe representation, depending on where an acoustic event falls within the block. A sparse, shiftablerepresentation method based on atom-like filter functions can solve the problem. Hence, following [21],a sound signal x(t) can be approximated as a linear combination of K filter functions, the so-calledspikes. In this study, we employed the Gammatone functions.

A Gammatone function is defined by its center frequency fk and its filter order. In a Gammatonefilter bank, the filter center frequencies are distributed across the frequency axis in proportion totheir bandwidth. The shape of the magnitude characteristics of the fourth order Gammatone filterapproximates the magnitude characteristics of the human auditory filter in a proper way [16]. Hence,these filter functions have a biological background. The spike analysis implementation was part of thework of NIPG (see CLOSED deliverable D5.1 for details).

Figure 9.1: For an impact sound, we display the sound wave and the spike code. This picture clearlyindicates how the spike code captures the skeleton of the sound.

Figure 9.1 shows the wave form and the spike code representation of an impact sound. In thisexample, a Gammatone filter bank of 256 filters is used for generating the spike code consisting of 32spikes. Note that salient areas in the wave of this sound is coded with more spikes than other areas.This indicates that the spike representation captures the signal characteristics properly.

124

9.3. The sonic space, sonic landmarks and GUI

The sonic landscape is organized as a 2D space in which reference sounds (sonic landmarks) arelocated. The organization of the reference sounds may rely on perceptual criteria, derived on astatistical basis (e.g., clustering), or may be arbitrarily decided by the user.

The sonic landmarks in the sonic space form a 2D scatter points set. The sound designer maychose to generate a new sound in the vicinity of a set of sonic landmarks inspired by their sonicproperties. Depending on the coordinates of the new position indicated by the user, the new set ofsynthesis parameters is generated through an interpolation scheme accounting for the neighboringsonic landmarks. Consider a set of landmark points

{(x1, y1, z1), (x2, y2, z2), . . . , (xn, yn, zn)}

where (xi, yi) is the position in the 2D space, and zi is the value of the synthesis parameters vector.A convenient interpolation scheme for this class of problems is based on the following steps (for sakeof simplicity, we consider a scalar parameter, z):• Compute a grid of triangles connecting the scatter points together. One possibility, which is usedhere, is the Delaunay triangulation.• For a new interpolated point (x, y, z), find the three vertices (xi, yi, zi), (xj , yj , zj) and (xk, yk, zk)among the landmarks that define the triangle in which the new point is confined and compute thecoefficients A,B, C, D for the equation of the plane Ax+By+Cz +D = 0 defined by the triangle.• Finally, compute the interpolated value through linear interpolation: z = f(x, y) = −A

C x−BC y−D

CFigure 9.2 gives an example of such interpolation for a small set of data points (circles) and a scalar

z parameter. The interpolated data are depicted as red stars.A graphical user front-end (GUI) was implemented in Matlab and is shown in Figures 9.3 and 9.4.

Its main frame represents the 2D sonic space with the sonic landmarks located in pre-defined spots(the figures show a configurations in which the landmarks were disposed as vertices of a rectangle).The location of the reference sounds can be arbitrarily decided and is specified in a configuration fileloaded at the beginning of each session. Once the landmarks sounds have been loaded and the auditoryrepresentation computed, the user can perform a number of actions, including:• Creating a new sound by choosing with the mouse a position in the proximity of the soniclandmarks with desirable characteristics.• Listening to landmarks and new sounds by clicking on the spike plot.• Deleting newly generated sounds which are not of interest for the user.• Saving the parametric setting of a new sound as presets in the XML-based format used by theSDT GUI.• Checking network connectivity with the Max/Msp or Pd server running the SDT sound synthesisengine.

The spike-based GUI is connected through an OSC network layer to a Max/Msp or Pd server runningthe SDT sound synthesis engine. When the user confirms the position of a new sound to be generated,the GUI starts the communication with the SDT server, proceeding through the following steps: first,the new interpolated parameters are sent to the server, which updates the controller values; whendone, the SDT synthesis engine is started and an audio file is generated; finally, the GUI loads theaudio file, generates the new auditory representation, and plot the result in the 2D sonic space.

9.4. Interpolation examples

This section presents the result of a sound design experiment in which six reference sounds (the soniclandmarks) where organized in the 2D space as shown in Figure 9.5 (the spike representation of thereference sounds, and the Delaunay triangulation generated with this configuration, are shown). Sixnew sounds were generated by choosing six interpolation positions (depicted as red stars). The synthe-sis sounds resulting from the interpolated parameters, converted into the spike-based representation,

125

Figure 9.2: Scatter points interpolation: Delaunay triangulation (dashed lines) on a set of data points(circles), and four interpolated points (stars).

126

Figure 9.3: Matlab GUI. Spike representation of four sounds (sonic landmarks) representative ofthe entire sound space generated by means of the SDT friction model. By clicking in the graphicalarea around a spike representation it is possible to listen to the corresponding sound (see the areahighlighted by circle 1). Circle 2 shows the control panel for the communication with the SDT modelin Max/MSP.

are represented in Figure 9.6.Perceptually, the results are in good agreement with the user’s expectations, and the dynamical and

spectral characteristics are recognized as actually deriving from the characteristics of the neighboringlandmarks. The audio files corresponding to the landmark sounds and to the interpolated sounds areavailable for download at the link provided in the footnote1.

1http://mordente.sci.univr.it/∼carlodrioli/SoMuCo09/Experiments.htm

127

Figure 9.4: Creation of new sounds. By clicking in any part of the Matlab GUI it is possible to startthe computation of a new sound, whose SDT synthesis parameters are calculated according to thegraphical position with respect to the four sonic landmarks (see circle 1). The interpolation of theparameters of the four sonic landmarks for the creation of a new sound is linear. The new parameterset is stored as an SDT preset in xml format and acquired by the SDT model in Max/MSP. Circle 2highlights the Spike representation of a new sound ready generated to be played in Max/MSP.

128

Figure 9.5: Interpolation example: the new sounds a), b) and c) are generated by interpolation ofpoints in the same triangle; the new sounds d), e) and f) are located in three different triangles

129

Figure 9.6: Interpolation example: new sounds ( a), b) and c) ) are generated by interpolation ofpoints in the same triangle; the new sounds d), e) and f) are located in three different triangles

130

References

[1] R. Bresin, S. Delle Monache, F. Fontana, S. Papetti, P. Polotti, and Y. Visell. Auditory feedbackthrough continuous control of crumpling sound synthesis. In Proc. Human Factors in ComputingSystems Conference, Florence, Italy, 2008. ACM/SIGCHI.

[2] S. Delle Monache, P. Polotti, S. Papetti, and D. Rocchesso. Gamelunch: a physics-based sonicdining table. In Proceedings of the International Computer Music Conference 2007 (ICMC2007),Copenhagen, Denmark, August 2007.

[3] S. Delle Monache, P. Polotti, S. Papetti, and D. Rocchesso. Sonically augmented found objects.In Proceedings of the New Interfaces for Musical Expression (NIME) conference, Genoa, Italy,June 2008.

[4] D. Devallez, F. Fontana, and D. Rocchesso. An audio-haptic interface based on auditory depthcues. In Accepted to ICMI 2008, Chania, Crete, Greece, October 2008.

[5] D. Devallez, F. Fontana, and D. Rocchesso. A virtual model for linear access to auditory distance.Submitted to Acta Acustica united with Acustica, 2008.

[6] D. Devallez, D. Rocchesso, and F. Fontana. An audio-haptic interface concept based on depthinformation. In Accepted to HAID’08, Jyvaskyla, Finland, September 2008.

[7] C. Erkut, V. Valimaki, M. Karjalainen, and H. Penttinen. Physics-based Sound Synthesis, volumePietro Polotti and Davide Rocchesso Eds., Sound to Sense, Sense to Sound. A State of the Artin Sound and Music Computing., chapter 8, pages 303–343. Logos Verlag, Berlin, 2008.

[8] F. Fontana and D. Rocchesso. A physics-based approach to the presentation of acoustic depth.In Proc. of the 2003 International Conference on Auditory Display, pages 79–82, July 2003.

[9] W. W. Gaver. What do we hear in the world? an ecological approach to auditory event perception.Ecological Psychology, 5(1):1–29, 1993.

[10] A. Kelloniemi. Room Acoustics Modeling with the Digital Waveguide Mesh – Boundary Structuresand Approximation Methods. PhD thesis, Helsinky University of Technology, 2006.

[11] N. Misdariis, R. Causse, L. Dandrel, J. Bensoam, and C. Vergez. Modalys, un outil pour le designsonore. In 1eres Journees du Design Sonore, Paris, 2002.

[12] H. Møller, M. Sørensen, D. Hammershøi, and C. Jensen. Head-related transfer functions of humansubjects. J. Audio Eng. Soc., 43:300–321, 1995.

[13] M. Muller, D. Charypar, and M. Gross. Particle-based fluid simulation for interactive applications.In Proc. ACM SIGGRAPH/Eurographics 03, pages 154–159, 2003.

[14] S. Papetti. Sintesi audio in tempo-reale mediante modelli waveguide di risonatori e modellinon-lineari di contatto. Master’s Thesis in Computer Engineering, University of Padua, A.A.2004/2005.

[15] S. Papetti, D. Devallez, and F. Fontana. DepThrow: a physics-based audio game. In Proceedingsof the International Conference on Auditory Displays, Paris, France, June 2008.

131

[16] R. D. Patterson and J. Holdsworth. A functional model of neural activity patterns and auditoryimages. Advances in Speech, Hearing and Language Processing, 3-B:547–563, 1992.

[17] P. Polotti, S. Delle Monache, S. Papetti, and D. Rocchesso. Gamelunch: Forging a dining expe-rience through sound. In Proceedings of the ACM CHI 2008, Florence, Italy, April 2008.

[18] D. Rocchesso and F. Fontana, editors. The Sounding Object. Mondo Estremo, 2003. Freelyavailable from http://www.soundobject.org/.

[19] M. Schafer. The Soundscape: Our Sonic Environment and the Tuning of the World. Rochester,Vermont: Destiny Books, 1994.

[20] B. Shinn-Cunningham. Learning reverberation: Considerations for spatial auditory displays. InProceedings of the International Conference on Auditory Displays, Atlanta, GA, 2000.

[21] E. Smith and M. S. Lewicki. Efficient coding of time-relative structure using spikes. NeuralComputation, 17:19–45, 2006.

[22] J. O. Smith. Principles of digital waveguide models of musical instruments. In M. Kahrs andK. Brandeburg, editors, Applications of DSP to Audio and Acoustic, pages 417–466. KluwerAcademic Publishers, 2002.

[23] B. Truax. Handbook for Acoustic Ecology. ARC Publications, 1978.

[24] K. van den Doel. Phisically-based models for liquid sounds. In Proc. ICAD 04, Sydney, July2004.

[25] E. Vincent. Mushram: a Matlab interface for MUSHRA listening tests, 2005.

[26] P. Zahorik. Direct-to-reverberant energy ratio sensitivity. J. Acoust. Soc. Am., 112(5):2110–2117,Nov. 2002.

132

http://www.soundobject.org/

report version: 1 classiﬁcation: pub. deliverable no. 2.3...

Documents