Automatic synchronisation between audio and score musical description layers

Antonello D’Aguanno, Giancarlo VercellesiLaboratorio di Informatica Musicale

Università degli Studi di Milano

Speaker: Antonello D’Aguanno

Introduction

• Contemporary digital music archives consist of huge collections of heterogeneous documents

• The heterogeneity of music information makes retrieval hard to accomplish

• Synchronization is one of the open problems in Music Information Retrieval (MIR)

Automatic synchronisation between audio and score musical description layers 2 /15

Synchronisation

3 /15

Stream Audio

Score

Synchronized score

Synchronization

Synchronisation means that for a given event in a music score we can determine the timing of the corresponding audio events

Automatic synchronisation between audio and score musical description layers

Related Works

• Many algorithms have been proposed in literature that deal with synchronisation

• All the algorithms proposed use MIDI score representation and PCM audio

4 /15Automatic synchronisation between audio and score musical description layers

Related Works

• The unsolved problems of synchronisation are not limited to finding a suitable algorithm

• How can this result generalised to use it in other applications?

• No answers in literature…

• MX can solve the problem

5 /15Automatic synchronisation between audio and score musical description layers

The linking structure in MX

6 /15

Notational Layer

Spine Sub-Layer

Performance Layer

Audio Layer

Automatic synchronisation between audio and score musical description layers

The COMSI algorithm

Three phases:

• Score Analysis

• Audio Analysis

• Recursive Decisional Matching

7 /15Automatic synchronisation between audio and score musical description layers

The three phases

• Score Analysis: The MX score is read in order to extract all relevant musical events

• Audio Analysis: The PCM audio signal is analysed to identify all possible musical notes

• Recursive Decisional Matching: Relates the event at the score level with the same event at the audio level

8 /15Automatic synchronisation between audio and score musical description layers

MX score analysis

1. For each measure, only the notes with strong accent are selected

2. For every strong accent, the verticalisation of the score is computed

9 /15

Observation:

COMSI Algorithm can manage conveniently even MIDI score

Automatic synchronisation between audio and score musical description layers

Audio Analysis

1. The audio signal is filtered with a notch filter centred on the frequency of the note examined

2. The audio signal in time-windows is split at 100ms

3. For each note, a possible attack-time is the audio segment which has an energy value above a threshold, obtained from the average energy of the filtered signal

10 /15Automatic synchronisation between audio and score musical description layers

Execution Events

The execution events are all the audio events having an energy above the threshold

The set of these execution events is named pseudo-score

11 /15Automatic synchronisation between audio and score musical description layers

The Decisional Matching

12 /15Automatic synchronisation between audio and score musical description layers

Measure Synchronisation

Recursive research of attack-time related to the first event of each measure in the score

Musical Event Synchronization

Sequential research of each musical event contained in a measure

Results

13 /15

Track Time Signature

Correct Measure

One Error Measure

Event

Twice Error Measure

Event

Chopin 3/4 66% 24% 10%

Beethoven 4/4 71% 18% 11%

Beethoven Faster 4/4 59% 36% 15%

Fiore di Maggio 4/4 80% 17% 13%

Cavatina 6/8 43% 48% 9%

Aria 6/8 61% 31% 8%

Automatic synchronisation between audio and score musical description layers

Conclusions

In this work we have described algorithm dedicated to score and audio alignment using the MX / IEEE P1599 formatThis algorithm allows alignment of an MX score and its execution, coded in PCM formatIt produces an output for the MX Spine that contains synchronisation between notes and audio signal

14 /15Automatic synchronisation between audio and score musical description layers

Future works

• A new test infrastructure is being developed, which will use the MX capabilities

• More tests in a shorter time!

• Will take into account every musical event

Automatic synchronisation between audio and score musical description layers 15 /15

Thanks!

daguanno@dico.unimi.it

Titolo della presentazione 16 di 15