CORRECT AUTOMATIC ACCOMPANIMENT DESPITE MACHINELISTENING OR HUMAN ERRORS IN ANTESCOFO

Arshia Cont, José Echeveste, Florent Jacquemard, Jean-Louis Giavitto

MuSync Team-Project (Ircam, Inria, CNRS)Ircam-Centre Pompidou, Paris, France.

name.lastname@ircam.fr

ABSTRACT

Automatic accompaniment systems are comprised of re-altime score following systems that in reaction to recogni-tion of events in a score from a human performer, launchnecessary actions for the accompaniment section. Whilethe realtime detection of score events out of live musi-cians’ performance has been widely studied in the litera-ture, score accompaniment (or the reactive part of the pro-cess) has been rarely addressed. In this paper, we exposethe respective literature concerning this missing consid-eration. We show how its explicit design considerationswould allow correct accompaniment despite machine lis-tening or human errors introduced during score following,and furthermore how it enables a more elaborate authoringof time and interaction for mixed live electronic pieces.

1. INTRODUCTION

Automatic accompaniment is the act of delegating the in-terpretation of one or several musical voices to a computerin interaction with a live solo (or ensemble) musician(s).The paradigm was first put through disjointly by Dannen-berg and Vercoe in [4, 9] where a computer would provideautomatic accompaniment out of incoming symbolic sig-nals (MIDI) from a musician. The most popular form ofsuch systems is the automatic accompaniment of an or-chestral recording with that of a soloist in the classicalmusic repertoire (concertos for example) as described in[6]. In a larger context, these systems became popular forinteractive computer music repertoire [7], where the as-sociation of live musicians with computer generated pro-cesses becomes crucial. Figure 1 shows the score of a sim-ple interactive computer music piece where the top staffcorresponds to the part for human musician and the lowerstaves correspond to automatic accompaniment sectionsthat should be run synchronously to the first staff duringlive performance. Note here that the accompaniment com-mands depend exclusively on the nature of the computermusic process in question. They can be symbolic com-mands, continuous curves or sequences written relative tothe live instrument section of the score. Within this con-text, the case of automatic accompaniment for classicalmusic repertoire is thus a special case of this larger con-text where accompaniment commands are replaced by ei-ther symbolic MIDI messages or a live rendering of an

audio recording of the orchestral part.

Figure 1. Score example for an interactive piece, thetop staff contains the instrumental section whereas lowerstaves correspond to specific computer music commands.

The best starting point in the design of an accompa-niment system is to observe the human counterpart. Mu-sical accompaniment among musicians is a combined actof listening and coordination governed by music scores.While human listening plays a crucial role in accompani-ment, it is (computationally speaking) fallible. Moreover,actions produced by musicians in an ensemble can contain(perceivable or unperceivable) errors. Despite such incon-sistencies, the overall musical output should stay more orless intact and error-free. This is to say that despite theimportance of the recognition (or listening) phase in musi-cal accompaniment, the coherence of the overall musicaloutput is to a great extent covered by the ability of hu-man musicians to coordinate and anticipate their actionsand adopt the best synchronization strategies in realtimeto achieve the best musical output.

Automatic accompaniment systems in general are com-prised of two main components as illustrated in figure 2:A score follower that is capable of decoding the live in-strument position as well as necessary musical parametersin realtime given the instrumental score (for the top staffof fig 1); and a second component in charge of launchingthe accompaniment commands synchronously to the livemusician (for lower staves in fig 1). In realtime controltheory [8], the score follower is an interactive system withtransactions at its own speed with the environment (usu-ally governed by signal processing and machine learningtechniques). Execution speed is formally abstracted to beperceptively close to zero for such processes. The ac-

companiment block is however a reactive system, reactingcontinuously to the external environment and at the speedimposed by the environment (the human performers).

Interactive System Reactive System

Score FollowerElectronic Score

Executor (qlist, messages, etc.)

Sound Generation (MaxMSP / PureData)

Figure 2. Architecture of an automatic accompanimentsystem comprising of interactive and reactive systems.

Considering both human and computer accompanimenttogether, the “healthiness” of the musical output of anaccompaniment system requires as much considerationsfor the coordination/reaction phase as for the recognitionphase. The general score following and accompanimentliterature has however underestimated the first, focusingon robustness in the listening phase; leaving the actionphase to hand-engineered and most often preliminary con-siderations for the accompaniment actions. This lack ofconsideration for action coordination would in return re-duce the paradigms of interaction between human and com-puter mediums for interactive pieces, leading to severelysimplified programming and authorship for mixed instru-mental and live electronic pieces as discussed in [3].

In this paper, we aim to illustrate the importance ofthe reactive system architecture in automatic accompani-ment and showing its usefulness within two contexts: (1)the ability to automatically handle machine listening orhuman errors with no propagation to the accompanimentoutput; and (2) to enrich the vocabulary of live electron-ics accompaniment by providing explicit access to the au-thoring of time and interaction in the interactive computermusic repertoire. We therefore assume throughout this pa-per that an interactive listening system is readily availableand focus on handling and authoring accompaniment ac-tions. This paper discusses the integration and employ-ment of the proposed paradigms within the Antescofo1 [1]software coupling both listening and coordination aspectsof such systems. In section 2 we discuss previous worksthat explicitly handle accompaniment actions integratedwithin a score following systems. We showcase and moti-vate problems with existing approaches in section 3 andpropose an approach to exposed problems by adoptingsynchronization strategies during authorship and run-timein section 4. We conclude the paper by illustrating severalexamples of this approach for automatic accompanimentas well as live electronic pieces using Antescofo.

2. PREVIOUS WORKS

In this section, we overview two existing and popular sys-tems that handle both machine listening and online ac-companiment. For the scope of this paper, we focus onthe way automatic accompaniment is transcribed and han-dled in such systems. Therefore, we do not provide any

1http://repmus.ircam.fr/antescofo

discussions on the machine listening aspects and do notoverview all existing score following systems.

2.1. Music-Plus-One

Music-Plus-One is a musical accompaniment system de-veloped by Christopher Raphael for the classical musicrepertoire and destined for a soloist in a concerto-like set-ting. The system is decomposed into three modules: onefor a realtime score match using hidden Markov models, asecond for coordinating audio playback of the existing ac-companiment using a phase-vocoder, and a third for link-ing the two using a Kalman filter model for tempo adjust-ments [6]. Whereas the listening and decoding parts ofMusic-Plus-One is well documented, there are no explicitdocumentation on the authoring and handling of the coor-dination for the accompaniment part (at least at the time ofwriting this paper). However, demos of the software areavailable on the web2. Here we aim to study the makingof the accompaniment part in order to understand the be-havior of the system in an online accompaniment setting.Music-Plus-One’s application is destined for the classicalmusic repertoire and concerto-like settings. It has beenhowever rarely used for contemporary interactive musicrepertoire. Understanding the synchronization and coor-dination strategies are important for the creation of newpieces for the system. The discussions hereafter are thusby no means a criticism of this approach.

Music-Plus-One uses the following minimal text datato undertake automatic accompaniment: a musical scoreof the solo part, an audio recording of the orchestral oraccompaniment part, a set of trained parameters for thelistening model (trained offline and used during realtimedetection), and some timing data in charge of associatingthe solo score to the accompaniment audio for the render-ing phase. Among these, the timing data is of outmostimportance since it creates the necessary binding betweenthe soloist score and the accompaniment audio that willbe employed during realtime rendering. Figure 3 showsan excerpt of Music-Plus-One text input for illustratingthis point. On the left, is a text description of the soloistmusic score, in terms of relative position in a measure,MIDI pitch numbers and other necessary information forthe listening module. On the right (a separate text file),each line contains the accompaniment audio onset timefor the specific event in the solo score. This correspon-dence is (most probably) obtained by a rigorous segmen-tation of the accompaniment audio with regards to the soloscore, to provide coherent musical phrases to be used dur-ing live accompaniment rendering. Note that the coordi-nation points on the right, do not necessarily correspond tothe solo score. This is normal since an optimal segmenta-tion of accompaniment onsets should be based on musicalphrasing rather than a one-to-one correspondence.

Within the structure described in the above example, itis not clear how Music-Plus-One handles machine listen-ing or human errors during live performance. Depending

2http://music-plus-one.com/

1+0/1 0/1 94 77 681+3/8 3/8 94 77 01+3/8 3/8 94 79 771+13/32 13/32 94 79 01+13/32 13/32 94 77 711+7/16 7/16 94 77 01+7/16 7/16 94 76 691+15/32 15/32 94 76 01+15/32 15/32 94 77 761+1/2 1/2 94 77 01+1/2 1/2 94 81 881+11/16 11/16 94 81 01+11/16 11/16 94 79 721+3/4 3/4 94 79 01+3/4 3/4 94 82 84

start_pos = 1+0/1end_pos = 103+5/8firstnote = 0lastnote = 1022atm_1+0/1 197.000 0atm_1+1/4 249.000 0atm_1+1/2 299.000 0atm_1+3/4 345.000 1atm_2+0/1 391.000 0atm_2+1/4 436.000 1atm_2+1/2 476.000 1atm_2+3/4 521.000 1atm_3+0/1 568.000 0atm_3+1/4 611.000 1atm_3+1/2 653.000 0

Figure 3. Excerpts of Music-Plus-One input. Left is thesoloist score, Right is the coordination score. Extract fromBeethoven’s Romance 2 for violin and orchestra, Op. 50’.

on the musical context, and in presence of machine lis-tening or human errors (missing events for example), onemight want to skip an accompaniment phrase or launchit with varying speed. Moreover, preparing such accom-paniment/coordination score seems to be a heavy phasein preparing an accompaniment and not very suitable forcompositional purposes.

2.2. Antescofo

Antescofo [1] is another polyphonic score following sys-tem capable of handling accompaniment actions and in-tegrated within the same system. The listening machineof Antescofo is documented in [2] and some aspects ofits accompaniment language in [1, 3]. The main musicalparadigm addressed by Antescofo is that of mixed instru-mental and live electronic (or interactive computer mu-sic) pieces, where the “accompaniment actions” can rangefrom a simple concerto-like setting to triggering of liveelectronic processes as common in interactive computermusic. Antescofo requires no training for its listener andaccept a single text as score input.

An Antescofo score is composed of both instrumen-tal (soloist) score and the accompaniment actions withinone integrated score. Figure 4 shows an example of asimple Antescofo score with both instrumental and ac-companiment actions. The semantics of the instrumentalscore allows the construction of complex events such astrills, glissandi, improvisation boxes and also continuousevents. The (accompaniment) action semantics is entirelybased on message-passing and provides constructions forgrouping of events in order to create polyphonic phrases,as well as loops and continuous trajectories. The timingof actions can be relative (to the score tempo) in floatingpoints or rational numbers, or in absolute time. Graphi-cal representation and authoring of such scores is possiblethanks to its integration within the NoteAbility score edi-tor. Figure 1 is in fact an illustration of an Antescofo scorefeaturing three discrete-sequence groups (green boxes), acontinuous trajectory and a symbolic action group (lowerstaff) as accompaniment actions, all living within a single

score framework. In its original text format, composersare also able to create nested hierarchies within electronicphrases (groups inside groups), employ macro expansionsand use data flow functionals (see [3] for descriptions).

Loop 1

Loop 2

a11 a12

a21 a22 a23

BPM 60!!NOTE 60 1.0!! Group1{!!0 action11!!½ action12!

}!!NOTE 67 1.0!! Group2{!!0 action21!!½ action22!

}!

Performer'events'

Electronic''ac1ons'

a21' a22'

Group2'

a11' a21'

Group1'

Figure 4. Sample score diagram and the correspondingsimple Antescofo score

The role of Antescofo in realtime is to decode the posi-tion and tempo of the performer within a synchronous re-active system to best interpret the accompaniment section.Compared to previous approaches in interactive computermusic, the accompaniment actions can be described asrelative to the performers’ tempo and thus dynamicallyrescheduled in realtime. Each action’s starting point isrelative to its backward closest event in the instrumentalscore. The phrasing schemes available in Antescofo allowthe scope of such electronic phrases to go beyond inter-onsets of the instrumental score.

While the ability of authoring parallel phrases (as op-posed to segmented and chopped actions) makes the actof authoring more appealing, such timing overlaps createimportant issues both for realtime coordination of eventswith the live performer and also their authoring. For ex-ample, for a simple concerto-like accompaniment setting,one might want to specify the accompaniment part as onesingle (and non-chopped and non-segmented) electronicphrase. This appealing tendency creates interesting chal-lenges for synchronization strategies as well as error han-dling of the virtual accompaniment interpreter which aredescribed in the next section.

3. MOTIVATIONS

3.1. Synchronizing electronic with live musicians

For compositional purposes, it seems more natural to beable to express accompaniment actions as phrases as op-posed to small segments within instrumental note onsets.However, such additions require explicit and dynamic strate-gies for handling synchrony between accompaniment ac-tions when their scope goes beyond that of its starting in-strumental event. The need for such strategies becomeeven more crucial when tempo changes occur after thephrase’s launch. Figure 5 attempts to illustrate this withina simple example: Figure 5(a) shows the ideal perfor-

mance or how actions and instrumental score is specifiedto the system. In this example, an accompaniment phraseis launched at the beginning of the first event from thehuman performer. The accompaniment here is a simplegroup consisting of four actions that are written paral-lel to subsequent events of the performer in the originalscore. In an ideal setting (i.e. correct listening module)the action group is launched synchronous to the onset ofthe first event. For the rest of the actions however, the syn-chronization strategy depends on the dynamics of the per-formance. This is demonstrated in figure 5(b) where theperformer hypothetically decelerate the consequent eventsin her score. In this case, the delays between the actionsand their corresponding instrumental event will naturallyincrease. Such asynchrony is despite correct decoding oftempo from the listening machine. We note however, thatthis behavior ensures a smooth synchronization with per-former tempo changes, but without any guarantee for po-sition synchronicity. Although this behavior is desired insome musical configurations, it seems natural to proposean alternative strategy where the electronic actions wouldbe synchronous to the events detection.

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

Performer'events'

Electronic''ac1ons'

delay'

delay'

a1' a2' a3' a4'

Ac#ons'Group'

(a) Ideal performance

Ac#ons'Group'

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

Performer'events'

Electronic''ac1ons'

delay’’'

delay’’'delay'

a1' a2' a3' a4'

(b) Slower performance

Figure 5. The effect of tempo-only synchronization foraccompaniment phrases. Illustration for different tempi.

3.2. The case of machine listening or human errors

The musical output of an automatic accompaniment sys-tem should not solely depend on its listening module oreven to the human performer at specific circumstances.This is in analogy to human coordination for ensembleperformance: A live music performance output should notbe at stake in presence of any error in realtime.

In a live performance situation different errors may beencountered: the listening module could confuse an eventwith another, miss an event, or produce a false-alarm. Ad-ditionally, musicians might introduce performance errors

that can affect the accompaniment results. In all cases,we expect not only that the system continues to work,but also that it reacts as musically as possible. Parts butnot all of these errors can be handled directly by the lis-tening modules (such as false-alarms and missed eventsby the performer). The critical safety of the accompani-ment part can thus be reduce to handling of missed events(whether missed by the listening module or human per-former). The natural question to ask in this case is whatthe system should do in case of a missed event? Shouldthe associated actions be performed or not? The answer tothis question seems more musical than technical: In someautomatic accompaniment situations, one might want todismiss associated actions to a missed event if the scopeof those actions does not bypass that of the current event atstake. On the contrary, in many live electronic situationssuch actions might be initializations for future actions.

This discussion shows that while such considerationscan be addressed automatically in special cases, error han-dling attributes should be first-class citizens in any speci-fication language for automatic accompaniment and inter-active computer music pieces.

4. PROPOSED APPROACH

The musician’s performance is subject to many variationsfrom the score. There are several ways to adapt to thismusical indeterminacy based on specific musical context.The correct synchronization and error handling strategiesis at the composer or arranger’s discretion. To this end,we propose explicit synchronization and error handlingstrategies as attributes for the composer to choose, tak-ing into account performance variations and to managethe errors of the musician and the recognition algorithm.In this paper, we provide a verbal description of the pro-posed approach. The formal and semantical definition ofthese concept are described in [5].

Accompaniment phrase and loop constructs are usedto start a sequence of actions from a trigger event in the in-strumental score. Once a sequence of relatively-timed ac-tions is launched, its synchrony by default is governed bydynamic rescheduling following changes of tempo fromthe musician. However, as seen in section 3.1, knowledgeof realtime tempo is not sufficient for precise synchroniza-tion with events played by the musician as shown in Fig-ure 5. While this loose synchrony is useful in some mu-sical context (loose phrasing of electronics for example),it is not always desirable. For a finer synchronization, weprovide the composer the ability to assign a tight at-tribute to a phrase block. If a block is tight, its insideactions will be dynamically analyzed to be triggered notonly using relative timing but also relative to the nearestevent in the future. This new feature evades the composerfrom segmenting the actions blocks to smaller segmentswith regards to synchronization points and provide a high-level vision during the compositional phase. A dynamicscheduling approach is adopted to implement the tightblock behavior. During the execution, the system attempts

to synchronize the next action to be launched with the cor-responding event using a hybrid strategy employing bothtempo and future-event positions.

The problem of error handling, as discussed in sec-tion 3.2, boils down to the ability of attributing scopes toaccompaniment phrases and blocks. Despite their spaceoccupancy in the score, a block is said to be local if dis-missible in the absence of its triggering event during liveperformance; and accordingly it is global if it should belaunched in priority and immediately if the system recog-nizes the absence of its triggering event. Once again, thechoice of an entity being local or global is given tothe discretion of the composer.

The combination of the synchronization strategy at-tributes (tight or loose) and error handling attributes(local or global) for a group of accompaniment ac-tions give rise to four distinct situations. Figure 6 attemptsto showcase these four situations for a simple hypotheti-cal performance setup. In this example, the score is as-sumed to demand for four distinct performer events (e1 toe4) with grouped actions whose two actions are initiallyspecified to occur on e1 and e3. The figure demonstratesthe simulation of the system behavior in case e1 is miss-ing during live performance for the four configurationsdiscussed above. Naturally, in our realtime setup, e1 isreported as missed once e2 is detected.

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

0 1 2 3 4 5 6

1 2 1 2 1 2

group loose

action-loose 1 action-loose 2 action-loose 3 action-loose 4

group tight

action-tight 1 action-tight 2 action-tight 3 action-tight 4

7 8

1 Pulsations en fonction du tempo estimé

Temps absolu(secondes)

Évènements joués de la partition du musicien

= 1 seconde

Performer'events'

Electronic''ac1ons'

e1' e2' e3' e4'

a1'

Ac#ons'Group'global'loose'

a1' a2'

Ac#ons'Group'global'#ght'

a1' a2'

Ac#ons'Group'local'#ght'

a1' a2'

Ac#ons'Group'local'loose'

a2'

Figure 6. Accompaniment behavior in case of missedevent for 4 synchronization and error handling strategies.

It is worth to note that each combination correspondsto a specific musical situation encountered in authoring ofmixed interactive pieces:

local and loose: A block that is both local and loosecorresponds to a musical entity with spatial independencewith regards to its counterpart instrumental events, andstrictly reactive to its triggering event onset (thus dismissedin the absence of this event).

local and tight: Strict synchrony of inside actionswhenever there’s a spatial correspondence between eventsand actions in the score. However actions within the strictvicinity of a missing event are dismissed. This case corre-sponds to an ideal concerto-like accompaniment system.

global and tight: Strict synchrony of correspondingactions and events while no actions is to be dismissed inany circumstance. This situation corresponds to a strongmusical identity, strictly tied to the performance events.

global and loose: An important musical entity withno strict synchrony once launched. Such entity is similarto musical phrases that have strict starting points with ru-bato type progressions (free endings, tempo-synchronous).

5. SAMPLE RESULTS

The above proposal has been integrated in Antescofo’sformal language and adopted in various new music pro-ductions involving live instrumental and electronic music.The best test for such systems is to see and use them in ac-tion. For the sake of completeness we discuss two sampleresults: one on a simple automatic accompaniment set-ting and another in case of a contemporary music pro-duction. Curious readers are referred to our website forfurther videos and events3.

In the first example, we attempt to reconstruct the per-formance of a four-voice Fugue by J.S. Bach (in B-minor)where one voice is performed by a musician and othersby automatic accompaniment. Figure 7 shows the scorefor this performance. In this example, and on purpose,the three accompanying voices are written as three distinctgroups of actions (for MIDI accompaniment in this case).Each block corresponds to the accompaniment actions foreach voice and the top staff represents the instrumentalpart. We choose tight synchronization strategies to en-sure precise synchrony with a performer and also betweendifferent accompaniment voices. To ensure musical con-sistency, local strategies are assigned to the groups. Inthis way, missed event’s corresponding actions will be dis-missed without altering the overall performance. Despiteits intuitive nature in run-time, the way the accompani-ment voices are written and handled in realtime provideease of authoring for composers willing to use such sys-tems for their compositions.

The second example is an excerpt score from the be-ginning of “Tensio” (2010) for String Quartet and liveelectronic by French composer Philippe Manoury as illus-trated in Figure 8. The left side of Figure 8 corresponds tothe hand-written manuscript (excerpt) containing both in-strumental (string section, four staves in bottom) and liveelectronic scores (top staves), and the right side of the fig-ure correspond to its Antescofo equivalent used during liveperformance. The Antescofo score describes three parallelgroup actions entitled arco, h1-trans and Pizzicaticorresponding to the left column. The two groups arcoand Pizzicati contain discrete and atomic sequences

3http://repmus.ircam.fr/antescofo

Figure 7. Antescofo score of Bach’s B-minor fugue inautomatic accompaniment mode, in NoteAbility Pro.

(defined with the keyword GFWD) whereas the h1-trans(defined as CFWD) contains a continuous sequence. Thecontinuous process h1-trans correspond to the top staffin the left column. During realtime performance theirpolyphonic synchrony is assured despite any variations inthe performance. These groups are notated as globaland loose (the default behavior of Antescofo) and cor-respond as discussed to integral musical phrases launchedin parallel to the instrumental world and influenced by theenvironment’s tempi.

BPM$ $72$TRILL $$($8500$)$$0.5 $ $IA...EVT41$GFWD $ $arco$${$$

$syn_voices$1 $pgm$40$ $bnk$7 $vol$122 $$$syn_voices$1 $56 $$120$ $@b(2)$$syn_voices$1 $50 $$120$ $@b(2)$$syn_voices$1 $46$ $120 $@b(2)$

2.0 $$syn_voices$1 $53$ $$120$ $@b(0.36)$$$syn_voices$1 $52 $$120$ $@b(0.36)$$syn_voices$1 $58 $$120$ $@b(0.36)$

0.36 $syn_voices$1 $69 $$120$ $@b(1.5)$$syn_voices$1 $63 $$120$ $@b(1.5)$$syn_voices$1 $59 $$120$ $@b(1.5)$

1.5$ $syn_voices$1 $66 $$120$ $@b(0.8)$$syn_voices$1 $65 $$120$ $@b(0.8)$$syn_voices$1 $62$$$$$$$$$$$$$120$$$$$$$$@b(0.8)$$

}$CFWD$$$$$$$$$$$$$$h1_trans $@grain$30ms${$$ $0$$$$$$$$$$$$$$0$$$$$$$$$$$$$$$

$2.0$$$$$$$$$$$300$$0 $0$$0.46 $800$$0 $0$$1.5 $4500$

}$GFWD $0.5 $PizzicaV${$

$Bsyn_voices$ $4$ $vol$127 $$1/5.5 $Bsyn_voices$4 $85 $127$ $100 $$1/5.5 $Bsyn_voices$4 $85$ $127 $100$1/5.5 $Bsyn_voices$4 $85$ $127$ $100$1/5.5 $Bsyn_voices$4 $85 $127$ $100$1/5.5 $Bsyn_voices$4 $85 $127$ $100$1/5.5 $Bsyn_voices$4 $85 $127$ $100$1/5.5 $Bsyn_voices$4 $85$ $127$ $100$}$NOTE$ $0$ $0.5$TRILL$ $(8100)$ $0.5$NOTE$ $0$ $0.5$TRILL$ $($7500)$ $0.5 $ $ $IA...EVT42$

Figure 8. Score manuscript of the first bars of “Tensio”(left), and the Antescofo counterpart (right).

6. CONCLUSION

In this paper we attempted to formalize the problem ofsynchronization and coordination of actions in an auto-

matic accompaniment setting by considering them as re-active synchronous systems on top of classical score fol-lowing and machine listening. With such considerationswe aimed at addressing the error handling and intelligentsynchronization strategies despite human or machine lis-tening error imposed to the system during a live perfor-mance. This is in contrast to most existing approacheswhere the integrity of the musical output is highly depen-dent on the healthiness of the artificial listening modulesat work. This becomes possible by studying the languageconstructs of both music composition and performancewithin a computer setting.

The solutions proposed in the paper try to cover vari-ous musical situations that correspond to concerto-like ac-companiment settings as well as compositional and per-formative aspects of interactive computer music pieces.They are integrated within the Antescofo software and havebeen employed in various music productions.

The coupling of high-level computer language con-structions with that of low-level machine listening, at thecore of this paper, is a necessity in musical practices ofinteractive computer music and requires more research inthe computer music community. We believe that researchon those lines could address complex problems with sim-ple and elegant solutions useful for both music composi-tion and live performance.

7. REFERENCES

[1] A. Cont, “Antescofo: Anticipatory synchronizationand control of interactive parameters in computer mu-sic,” in ICMC 2008.

[2] ——, “A coupled duration-focused architecturefor realtime music to score alignment,” IEEETransactions on Pattern Analysis and MachineIntelligence, vol. 32, no. 6, pp. 974–987, 2010.

[3] ——, “On the creative use of score following and itsimpact on research,” in SMC 2011.

[4] R. B. Dannenberg, “An on-line algorithm for real-time accompaniment,” in ICMC 1984, pp. 193–198.

[5] J. Echeveste, A. Cont, J.L. Giavitto, F. Jacque-mard, “Antescofo: a Domain Specific Language forreal time musician-computer interaction,” in JDEDS2012, submitted.

[6] C. Raphael, “The informatics philharmonic,” Com-mun. ACM, vol. 54, pp. 87–93, 2011.

[7] R. Rowe, “Interactive music systems: machinelistening and composing”. MIT Press, 1992.

[8] R. Shyamasundar and S. Ramesh, “Real time pro-gramming: languages, specification and verification”.World Scientific Pub Co Inc, 2009.

[9] B. Vercoe, “The synthetic performer in the context oflive performance,” in ICMC, 1984, pp. 199–200.