performing expressive music using case-based reasoning ramon lópez de mántaras iiia - csic...
Post on 20-Dec-2015
216 views
TRANSCRIPT
![Page 1: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/1.jpg)
Performing expressive music using Case-Based Reasoning
Ramon López de MántarasIIIA - CSIC
[email protected]/~mantaras
![Page 2: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/2.jpg)
Outline
• Reminding CBR & Introducing Saxex main components
• Case representation– The musical knowledge
• Retrieval using perspectives
• Reuse– Fuzzy combination
• SaxEx Results
• TempoExpress
• Conclusions and future work
![Page 3: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/3.jpg)
Case-based reasoning (CBR)
Solving problems by means of examples of already solved similar problems
(reasoning from precedents)The task of our system is to infer, via CBR
and musical knowledge, a set of expressive transformations to be applied to the notes of inexpressive musical phrases given as input
The precedents are examples of expressive human interpretations
![Page 4: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/4.jpg)
Saxex Components
Noos
SMSanalysissynthesisScore
CasesCBR methodMusicalmodels
.snd.sndInexpressivephrase Expressive phraseInput Output
.sms .sco.midAffectiveLabels
![Page 5: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/5.jpg)
SMS Snapshot
![Page 6: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/6.jpg)
Saxex-CBR
Saxex-CBRRetrieveReuseIdentifySelectConstructperspectivesRetrieveusingperspectivesRankprecedentsusing persp.and pref.
Applyexpressivetransform.Memorizenew solvedcaseRetain
SearchIdentify&SelectProposeexpressiveperformancesRevise
![Page 7: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/7.jpg)
Outline
• Reminding CBR & Introducing Saxex main components
• Case representation– The musical knowledge
• Retrieval using perspectives
• Reuse– Fuzzy combination
• SaxEx Results
• TempoExpress
• Conclusions and future work
![Page 8: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/8.jpg)
Case representation
• Score• Musical knowledge
– implication-realization, metrical structure, time-span reduction & prolongational reduction
• Performance representation (solution description)
• sound transformation operations: – eg: high dynamics, medium rubato, very legato, etc.
SOLUTION
![Page 9: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/9.jpg)
Transformations• Transformations (for each note)
– Dynamics (5 possible values)– Rubato (5 possible values)– Vibrato (5 possible values) -----> 1250
possibilities– Articulation (5 possible values)– Attack (2 possible values)
Vibr. Vibr.
Din.
Rub
Art.
![Page 10: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/10.jpg)
Score
![Page 11: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/11.jpg)
Musical knowledge• Implication/Realization model (Narmour)
– Basic structures:
– Melodic direction, durational cumulation
• GTTM theory (Lerdahl & Jackendoff)– Metrical structure (metrical strength of notes)– Time-span reduction (relative importance of
notes within phrases or sub-phrases)– Prolongational reduction (tensions, relaxations)
• Jazz Theory– Harmonic Progressions (duration, harmonic
stability)
![Page 12: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/12.jpg)
Implication/Realization Model
![Page 13: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/13.jpg)
GTTM Theory
![Page 14: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/14.jpg)
Performance
![Page 15: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/15.jpg)
Outline
• Reminding CBR & Introducing Saxex main components
• Case representation– The musical knowledge
• Retrieval using perspectives
• Reuse– Fuzzy combination
• SaxEx Results
• TempoExpress
• Conclusions and future work
![Page 16: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/16.jpg)
A Retrieval Perspective
![Page 17: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/17.jpg)
![Page 18: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/18.jpg)
Case Memory
Problem
Identify Search Select
Retrieval Example
![Page 19: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/19.jpg)
Outline
• Reminding CBR & Introducing Saxex main components
• Case representation– The musical knowledge
• Retrieval using perspectives
• Reuse – Fuzzy combination
• SaxEx Results
• TempoExpress
• Conclusions and future work
![Page 20: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/20.jpg)
Saxex-Reuse• Transformations
– Dynamics– Rubato– Vibrato– Articulaction– Attack
• Criteria– Most similar
– Majority
– Minority
– Continuity
– Random
– Fuzzy combination (DEFAULT)Vibr. Vibr.
Din.
Rub
Art.
![Page 21: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/21.jpg)
Problem
Din.
Rub
Art.
Single case retrieved
Din.
Rub
Art.
Saxex-Reuse Example
![Page 22: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/22.jpg)
Saxex-Reuse (Fuzzy Combination)
20 320 Tempo0
1
Very VeryLow Low Medium High High
The notes in the human-performed musical phrases are qualified by means of five ordered linguistic values. Those for rubato are:
Assume that SaxEx has retrieved and selected two notes whose rubato values are72 and 190 respectively. The fuzzy combination followed by a defuzzification gives the rubato value to be applied to the input note:
72 123 190
0.90.7
COALow Medium
![Page 23: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/23.jpg)
Outline• Reminding CBR & Introducing Saxex
main components
• Case representation– The musical knowledge
• Retrieval using perspectives
• Reuse– Fuzzy combination
• SaxEx Results
• TempoExpress
• Conclusions and future work
![Page 24: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/24.jpg)
Saxex Results
Autumn Leaves
InexpressiveInput phrase
ExpressiveOutput phrase
SaxEx
![Page 25: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/25.jpg)
Affective Labels
• Three orthogonal dimensions– Tender-Aggressive– Sad-Joyful– Calm-Restless
• Relating to notions such as– activity– tension vs. relaxation– Brightness . . .
![Page 26: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/26.jpg)
InexpressiveInput phrase
SaxEx Results
SaxEx
Aff. values
Joyful
Sad
All of me
![Page 27: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/27.jpg)
• Reminding CBR & Introducing Saxex main components
• Case representation– The musical knowledge
• Retrieval using perspectives
• Reuse– Fuzzy combination
• SaxEx Results
• TempoExpress
• Conclusions and future work
![Page 28: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/28.jpg)
Goal:– Changing the original performing tempo of a
melody, preserving expressiveness, in the context of jazz standards.
Application:
Audio editing softwareVideo / Audio post-production (video constrains audio)
Why not applying uniform time stretching to the audio?Timing of notes w.r.t. beat may have to changeOther expressive phenomena (e.g. ornamentations, consolidations, fragmentations) may have to change as a function of the tempo
TempoExpress
![Page 29: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/29.jpg)
Musical explanation: Expressivity is a result of the conception of the music by the performer, and this conception changes with tempo [Desain & Honing, 1994]
Original tempo (180 ) Transformed tempo (90)
Uniform time stretchingMelody: “Up Jumped Spring”
Recording
TempoExpress
![Page 30: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/30.jpg)
Some basic music performance concepts and their relations
Expressive Transformations
![Page 31: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/31.jpg)
Onset deviations at different tempos(Body and Soul A1)
![Page 32: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/32.jpg)
• “Hand crafted”– Let a music expert formulate rules for music performance
(Friberg, CMJ 1991, Friberg et al. CMJ 2000)
• Machine learned– Derive expressivity rules automatically from examples
(Widmer, ICMC 2000, JNMR 2002)• Eager approach: Builds a model based on many training
examples and uses the learned model to solve new problems
– Imitate expressivity using examples of concrete human performances by means of CBR (Arcos & Lopez de Mantaras, JNMR 1998, Lopez de Mantaras & Arcos, AI Mag 2002)• Lazy approach: Take the solution of the training example
that resembles most to the new problem, and adapt it to solve it
“That an expressive effect is applied only once does not mean it is insignificant” (Sundberg, MP 2001)
Approches to expressive music generation
![Page 33: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/33.jpg)
TempoExpress Architecture
Desired Tempo
![Page 34: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/34.jpg)
Performance Annotation
Expressivity in jazz is more than timing / dynamics deviations. It is alsospontaneous note ornamentations, fragmentations, etc.
To model this, we define a set of Performance Events:
And we use them as edit operations to obtain an edit-distance-based alignmentbetween the score and the performance
![Page 35: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/35.jpg)
Goal of the annotation process– Automatic case base acquisition
Comparing Score vs recordings
![Page 36: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/36.jpg)
Body and Soul
Once I Loved
Examples
F C C
I I
![Page 37: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/37.jpg)
• Goal: Assessing the distance between two sequences <S1 , S2
>– Calculated as the minimal cost of
transforming S1 into S2
– Requires:• Edit operations• Cost functions
Edit (Levenshtein) distance
![Page 38: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/38.jpg)
€
di, j = min
di − 1, j + w(ai,∅ ) (deletion)
di, j − 1 + w(∅ ,bj) (insertion)
di − 1, j − 1 + w(ai,bj) (replacement)
di − 1, j − k + w(ai,bj − k + 1,...,bj),2 ≤ k ≤ j (fragmentation)
di − k, j − 1 + w(ai − k + 1,...,ai,bj),2 ≤ k ≤ i (consolidation)
⎧
⎨
⎪ ⎪
⎩
⎪ ⎪
RRR I
Edit (Levenshtein) distance
![Page 39: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/39.jpg)
T T F
Case
Annotation examples (I)
![Page 40: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/40.jpg)
TTT CTCT
Case
Annotation examples (II)
![Page 41: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/41.jpg)
TTTTI
Case
Annotation examples (III)
![Page 42: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/42.jpg)
• Rationale: the expressivity of a performed note is not just determined by the note itself.
• Ergo: Some representation of the melodic context of the note is needed
• We use the Implication / Realization model of melodic structure (Narmour, 1990)– It captures the pattern of fulfillment / violation of expectations
created by the melodic surface– Groups notes based on gestalt principles
Representing melodic context
![Page 43: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/43.jpg)
Repeated for eachtempo
Case Representation
![Page 44: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/44.jpg)
• 1. Filter cases by tempo: keep cases containing performances at relevant tempos (one of the tempos is similar to the original tempo of the target melody and there is another performed tempo similar to the desired tempo to which the target melody has to be transformed)
• 2. Rank the cases that passed the previous filter by I/R similarity to the score of the target melody (using edit-distance)
• 3. Partition the phrases of the most similar cases into segments using the I/R parser or any other melodic segmentation algorithm (for instance Temperley, 2001)
• 4. Form a “new” case base containing the obtained segments (space of partial solutions) as cases
Retrieval
![Page 45: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/45.jpg)
• Solutions for the target melody are generated segment-wise via a best first search through the space of partial solutions (segments)
• Procedure:1. Retrieve best matching segment (using edit-distance)2. Align target melody and retrieved segment3. Transfer performance events
For aligned notes T and R, let Ti(R) -----> To(R) represent the tempo transformation of note R; use the annotations differences between Ti(R) and To(R) to generate the solution To(T) from Ti(T)
4. For non-aligned target notes use UTS to transform Ti(T) into To(T)
Reuse
![Page 46: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/46.jpg)
TempoExpress overall view
![Page 47: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/47.jpg)
Uniform time stretching
CBR
Human
55 100 bpm
Example of TempoExpress Result
![Page 48: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/48.jpg)
• Four jazz standards recordings by a professional musician (12 tempos for each: 48 recordings)
• 14 different phrases containing a total of 64 different melodic segments
• More than 8000 tempo-transformation problems in the case base
Experimental comparison to UTS
![Page 49: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/49.jpg)
TempoExpress vs. UTS as a function of the ratio of original tempo to transformed tempo.The lower plot shows the probability of incorrectly rejecting the hypothesis (that there is no difference between TempoExpress and UTS) for the Wilcoxon signed-rank test.
![Page 50: Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es mantaras](https://reader036.vdocument.in/reader036/viewer/2022062407/56649d4c5503460f94a2b09a/html5/thumbnails/50.jpg)
Conclusions & Future• CBR is a powerful technique to imitate human solutions
(performances): Human-like output• SaxEx successfully retrieves relevant cases• Fuzzy combination increases output variation• SaxEx as a pedagogical tool:
– Users can experiment with the system– Helps understanding how to use the different expressive resources
• TempoExpres: an application to audio post-production that clearly outperforms UTS
• Further TempoExpress experimentation with fast tempos (more example cases at fast tempos are needed)
• Add within-note descriptions:– Energy envelpe features: attack, sustain, decay, tremolo– Pitch envelope features: vibrato, glissando
• Add between-notes descriptions:– Articulation (legato,, staccato)