OPTIMAL POSITIONING IN LOW-DIMENSIONAL CONTROL SPACESUSING CONVEX OPTIMIZATION

Peter Kassakian & David WesselCenter for New Music and Audio Technologies (CNMAT)

University of California Berkeley{kassak, wessel}@cnmat.berkeley.edu

ABSTRACT

A music analysis and control system for use in liveperformance is presented and demonstrated. Musical fea-tures, such as harmony, rhythmic patterning, or melodicstructure, are extracted and automatically placed at appro-priate locations in a control space. The control space is oflow dimensionality, usually only two dimensions, whereinperceptual dissimilarity is represented as distance. Thesystem can be viewed as a listening assistant that aids aperforming computer musician with usable informationabout the evolving musical context so that the material un-der control can be adjusted to respond in interesting waysto that context. The method developed to position thepoints is based on a semidefinite programming relaxationof a nonconvex positioning problem and is both novel andelegant. Another novel feature of our work is an inter-process communication implementation that provides fortwo-way traffic between Max/MSP and MATLAB in real-time. We demonstrate the system with a harmony spaceand a melodic process space. The method is robust, rapid,and not plagued by problems with local minima. This suc-cessful use of convex optimization suggests that semidefi-nite programming may have broad application in the com-puter music field.

1. INTRODUCTION

When performing with others it would seem important toknow what they are doing; what rhythms they are play-ing, what harmonies they are employing, what melodicprocesses they are gesturing upon, what timbres they use,and what auditory environments they inhabit. We offera real-time method for mapping machine generated per-ceptions of such aspects of a musical scene to low dimen-sional gestural control structures that afford a variety ofmusical reactions to past and present contexts. Our goalis to develop a rudimentary real-time music analysis sys-tem that aids in the mapping of a low-dimensional controlstructure to both stable and evolving features of the musi-cal scene. A rich variety of control spaces is presented in[8].

The general setup for our analysis problem is as fol-lows. We assume the control space contains a set of fixedpoints, called anchors, with which we can compare new

musical data. We specify a way to evaluate similarity be-tween the new element and the anchors, and position theelement in a manner that optimally reflects the dissimilar-ities as geometric distances. The result is a representationof the new data as a positioned point in the control space,useful both in providing information, and for subsequentcontrol.

The optimization problem associated with this posi-tioning problem is solved by an interior point method –a type of method for solving convex optimization prob-lems that since the early 1990’s has been an area of in-tense research interest. The interior point methods havefavorable computational complexity, and are very efficientin practice. We present the optimization problem in detail,showing how it’s possible to form a convex approximationto the nonconvex localization problem that reliably findsthe global minimizer of our error function. We empha-size the value of convex optimization in computer musicapplications, where nonconvex and peculiar optimizationproblems abound.

To demonstrate the analysis-to-controller link we presenta two straight-forward examples, one in harmonic analysisand another melodic analysis.

Finally we discuss an implementation of the completesystem that links Cycling74’s Max/MSP[1] and the Math-works MATLAB[7]. We have built a Max/MSP exter-nal object that directly links, via dynamic C libraries, tothe MATLAB Engine, an interface designed to allow pro-grammers call MATLAB from within other programs writ-ten in C. This project represents one of many computermusic projects that involve complicated numerical rou-tines that are conveniently handled in MATLAB, but aredifficult to implement directly in the Max/MSP environ-ment or in C. In our case, we call an interior point solverSeDuMi[11] from MATLAB, called from within Max/MSP.

2. ALGORITHM

2.1. Introduction and Motivation

In this section we describe the method for positioning pointsrepresenting newly gathered musical elements. We as-sume that the musician already has constructed an abstractspace, defined by existing points, now held fixed for thepurposes of locating an incoming point. The dissimilari-

ties of the musical element in relation to the fixed elementsare evaluated, and thought of as a set of desired Euclideandistances between the new point and the existing points. Itis unlikely that the evaluated dissimilarities are consistentin the sense that they unambiguously and perfectly deter-mine the coordinates of the point in the control space. Thisis especially true if the number of fixed points is large, andthe new point represents a unique type of musical elementnot encountered previously.

It is because of this likely inconsistency that we seekthe coordinates that minimize the sum of the squared er-rors between desired dissimilarities and achieved distances.The important attribute of the evaluated dissimilarities ishow they are related relative to each other so we calculatean optimal global rescaling factor, in effect working withdistance ratios instead of absolute distances.

2.2. Formulation

We now turn to the mathematical details of the optimiza-tion problem and show how a convex relaxation of theproblem can be obtained and solved with interior pointmethods. Interior point methods differ from gradient andNewton methods in that they incorporate global informa-tion about the geometric structure of the optimization prob-lem, and find solutions by progressing through the interiorof the feasible region. Gradient and Newton methods arelocal in the sense that they only query the description ofthe problem to obtain derivatives of the objective func-tion. They require a suitable starting point, can find onlylocal minima, and can be inconsistent in terms of num-ber of iterations required to obtain a precise solution. Incontrast, the formulation we present is mathematically in-formative, centered on duality and convexity. The use ofinterior point methods in numerically calculating the so-lution is efficient and consistent. In all problem instanceswe have encountered and checked, the method has yieldedthe globally optimal solution, even in cases where thereare many local minima.

Let ri ∈ Rn, i = 1,2, ...,m be the locations of m fixed

points in the space. Let di ≥ 0, i = 1,2, ...,m be the evalu-ated dissimilarities between the new element and the fixedelements. We want to find x? ∈R

n, and α? ∈R+ that solve

p(r,d) = minx,α

m

∑i=1

(‖x− ri‖−αdi)2. (1)

It will be convenient to express the objective function ina different form. Referring to Figure 1, we see that eachterm in (1) can be expressed as the optimal value of a sub-problem:

(‖x− ri‖−αdi)2 = min

bi∈Rn‖x−bi‖

2 (2)

s.t. ‖ri −bi‖2 = α2d2

i .

Combining (1) and (2), and writing in matrix form, we

bi

x

|‖x− ri‖−αdi|

‖x− ri‖

riαdi

Figure 1. Geometric intuition for the subproblem (2).

have

p(r,d) =minx,α,b

‖Jx−b‖2 (3)

s.t. ‖ri −bi‖2 = α2d2

i , i = 1,2, ...,m,

where b ∈ Rmn is definedb ≡ [bT

1 ,bT2 , ...,bT

m]T

and J ∈ Rmn×n is definedJ ≡ [I, I, ..., I]T .

The strategy for solving this nonconvex problem is to re-lax the constraints via duality. The dual problem that arisesthrough Lagrange multipliers is always convex, and be-cause of the algebraic form of the problem (quadratic), itcan be analytically derived, and solved using interior pointmethods. We will solve the “dual of the dual” which isalso convex, and has an interpretation as a relaxation ofthe primal problem (3). This problem is

minZ,z,v

Trace(WZ) such that (4a)

‖ri‖2 − vd2

i +n

∑j=1

2(ri) j zki j +Zki j ,ki j = 0, (4b)

ki j = (i−1)m+ j, i = 1,2, ...,m,[

1 zT

z Z

]

is positive semidefinite, (4c)

v ≥ 0, (4d)

where Z ∈ Smn×mn,z ∈ R

mn,v ∈ R

and W ≡ I − J(JT J)−1JT.

The problem can be solved with semidefinite program solvers,for example SeDuMi, developed by Joe F. Sturm [11]. Be-cause the convex program is a relaxation of (3), the vari-ables do not directly correspond to the variables in ouroriginal problem (3), though they are related. In fact, in-cluding an additional (nonconvex) contraint zzT = Z in(4) yields an alternative formulation of the original primalproblem (3). Hence the dual relaxation can be directly de-rived by reformulating (3), and ignoring a constraint of theform zzT = Z. To obtain a primal feasible solution fromthe relaxation, i.e., a solution that indeed satisfies zzT = Z,we use a randomized method first reported by Goemansand Williamson in [3] and generalized by Y. Nesterov in[10].

A more conventional method for optimizing this ob-jective function, for example a gradient method, could re-sult optimal solutions, especially if time is allocated forrepeated solution using several starting points. In our ap-plication though, we are pleased with the interior pointapproach because of its consistent speed and high-qualitysolutions. Convex optimization and interior point methodsform a powerful framework both in terms of mathemati-cal modeling and numerical complexity; for a referenceon the subject see [2]. They are promising methods formany computer music applications.

3. EXAMPLES

In this section we consider two simple examples of theidea and solution method. The first uses the key spacediscussed in [5] as a way to track implied tonality of asoloing musician. We analyze four choruses of CharlieParker’s “Blues for Alice”, transcribed by Jamie Aeber-sold and Ken Slone, published in [4]. The second ex-ample involves a melodic contour space, where similaritybetween melodies is evaluated as a function of the typeof intervallic motion encountered. This example serves toshow that the optimization method finds globally optimalpoints, even when the objective function possesses severalstationary points.

3.1. Key Space Example

Consider the four dimensional key space discussed in [5].Each of the 24 keys is represented by a point, and in sucha way that the distance between any two points reflectsthe dissimilarity between the keys, as measured by sub-jective evaluation. The technique used to calculate the lo-cations of these points is called multidimensional scaling.Our problem is related, but simpler. We wish to use thelocations of the 24 keys in the space as anchors, and posi-tion an arbitrary histogram of tones within this structuredspace. This would allow us to track tonality in a system-atic way.

A normalized histogram of notes is called a tone pro-file. For each of the 24 keys there exists a canonical toneprofile, also described in [5]. We will use the natural Eu-clidean metric to measure similarity, i.e., if r1,r2 ∈ R

12

are tone profiles, then the dissimilarity between them isd = ‖r1 − r2‖2. Hence, presented with an unfamiliar toneprofile, we can calculate its dissimilarity relative to eachof the 24 keys, and position it appropriately using the out-lined method.

A representation of the space in two dimensions can befound by projecting the points onto a torus, and unwrap-ping the torus onto a square in R

2.Here we have compiled tone profiles for each of 12

measures in Charlie Parker’s “Blues for Alice”, as tran-scribed in [4]. The tone profiles were gathered over fourchoruses of the song including the melody. By plottingtheir optimal locations in the key space, we can gain in-sight into the song’s harmonic structure as interpreted by

Parker. See Figure 2.

C

C#

D

Eb

E

F

F#

G

Ab

A

Bb

B

c

c#

d

eb

e

f

f#

g

ab

a

bb

b

F

Em7b5−A7Dm−G7

Cm−F7Bb7

Bbm−Eb7Am

Abm−Db7Gm

C7

F7

Gm−C7

Figure 2. Charlie Parker’s solo on Blues for Alice

3.2. Melodic Process Example

In this example, we form a two-dimensional melodic con-trol and analysis space. Anchors in the space are melodieswith motion consisting of only small steps, only large leaps,leaps up with stepwise motion down, leaps down withstepwise motion up, and several other variants. Dissim-ilarities are computed in the following way. Building onKrumhansl’s interpretation [6] of Narmour’s melodic pro-cess [9], we first associate a Markov transition matrix witheach melody that explains the empirical frequencies of oc-currence of steps downward followed by steps upward,steps downward followed by steps downward, leaps up-ward followed by steps downward, etc. The dissimilaritybetween two melodies in this space is evaluated by com-puting the sum of the squared differences in the entries ofthe respective Markov matrices. Figure 3 shows the lay-out of the space, with graphical depictions of the melodiesassociated with the points.

The contours in the figure are contours of the objectivefunction for the optimization problem. We see that thesolution computed by the semidefinite relaxation methodfinds the global minimum, even though there is more thanone stationary point of the function. This is a very desir-able characteristic of the method.

4. MAX/MSP-MATLAB-SEDUMIIMPLEMENTATION

To use the proposed idea in a performance setting, wecombine the live performance convenience of Max/MSP,and the numerical computation convenience of MATLABand SeDuMi. We have installed all three software pack-ages on a Macintosh running OSX, and have written aMax/MSP external C program that dynamically links tothe MATLAB Engine. The MATLAB Engine is an in-terface designed to give programmers the ability to call

Figure 3. Melodic Process Space The contours of theobjective function possess two local minima, yet we areable to locate the global minimizer.

functions and fully interact with a MATLAB session fromanother C program, in our case, our Max/MSP external.This allows the musician to perform complicated or spe-cialized computation in a very convenient way. In ourcase, we call, via MATLAB, the semidefinite optimiza-tion software package SeDuMi, the numerical result ofwhich appears in the Max/MSP environment. Work isbeing done to design a more general Max/MSP object toconnect to MATLAB that would enable musicians to con-veniently call complicated numerical routines in the midstof a live performance.

5. CONCLUSION

We described a method for the analysis of musical contentin terms of placement within a defined control space. Themethod relies on numerical optimization by interior pointmethods, a relatively new methodology for solving convexoptimization problems.

A detailed treatment of the underlying optimization prob-lem was presented. The analysis of the problem in termsof duality and convexity is mathematically more informa-tive than optimizing via traditional methods. This is be-cause we are able to obtain an analytical formulation of aconvex approximation to the nonconvex positioning prob-lem. Having this representation allows the use of interiorpoint methods which are efficient and well-behaved. Wehave applied the method to generate examples in harmonicand melodic analysis.

Finally, we mentioned an implementation of the systemconveniently linking Max/MSP and MATLAB, throughwhich we call the semidefinite program solver.

6. ACKNOWLEDGMENTS

The authors would like to thank Meyer Sound Laborato-ries, Rimas Avizienis, Michael Zbyszynski, Adrian Freed,Laurent El-Ghaoui, and Lieven Vandenberghe for theirhelp and advice.

7. REFERENCES

[1] Cycling 74. Max/MSP. See www.cycling74.com fordetails.

[2] Stephen Boyd and Lieven Vandenberghe. ConvexOptimization. Cambridge University Press, 2004.

[3] M. Goemans and D. Williamson. Improved ap-proximation algorithms for maximum cut and satis-fiability problems using semidefinite programming.Journal of Association for Computing Machinery,42(6):1115–1145, June 1995.

[4] Michael H. Goldsen, editor. Charlie Parker Omni-book. Atlantic Music Corp, 1978.

[5] Carol L. Krumhansl. Cognitive Foundations of Mu-sical Pitch. Oxford University Press, 1990.

[6] Carol L. Krumhansl. Music psychology and musictheory: Problems and prospects. Music Theory Spec-trum, 17(1):53–80, 1995.

[7] The Mathworks. MATLAB. Seewww.mathworks.com for details.

[8] Ali Momeni and David Wessel. Characterizing andcontrolling musical material intuitively with geomet-ric models. In Proceedings of the 2003 Conferenceon New Interfaces for Musical Expression, Mon-treal, Canada, 2003.

[9] E. Narmour. The Analysis and Cognition of Ba-sic Melodic structures: The Implication-RealizationModel. University of Chicago Press, 1990.

[10] Yurii Nesterov. Semidefinite relaxation and noncon-vex quadratic optimization. Optimization Methodsand Software, 9:141–160, 1998.

[11] J. F. Sturm. Using SeDuMi 1.02, a MATLAB tool-box for optimization over symmetric cones. Opti-mization Methods and Software, 11:625–653, 1999.