Machine Discoveries: A few Machine Discoveries: A few Simple, Robust Local Simple, Robust Local Expression PrinciplesExpression Principles

Written by Gerhard Widmer presented by Siao Jer, ISE 575b, Spring 2006

Presentation OverviewPresentation Overview

General Overview Introduction Training Data Target Classes Experimental Results Quantitative Validation Conclusion Future Research

Gerhard WidmerGerhard Widmer

Head of the Department of Computational Perception at Johannes Kepler University Linz, Austria

Head of Machine Learning, Data Mining, and Intelligent Music Processing Group at the Austrian Research Institute for Artificial Intelligence

Numerous publication, awards, projects

General OverviewGeneral Overview

Discovering rules of expressive music performance

Inductive machine learning Experiments with large data sets Simple and general principles Robust with surprisingly high level of

accuracy

IntroductionIntroduction

What do performers do to make music “come alive?”

Studies done through a few classical approaches

Proposal of inductive machine learning No preconceptions and expectations Huge data sets allowed for more validity

IntroductionIntroduction

Previous work Success in ability of machine learning (Widmer

2000) Extremely complex Attempt to find a complete model

Current goals Testing new learning algorithm based on partial

models Learn rules of timing, dynamics, articulation Testing degrees of fit over various styles and

performers

Training DataTraining Data

13 complete Mozart piano sonatas Performed by Roland Batik On computer monitored grand piano

MIDI format Includes hammer speed, impact times, pedal

movements measured & xform’ed Written score coded into computer format Timing, dynamics, & articulation computed 106,000 total notes Melody restriction limits us to 41,000 notes

Target ClassesTarget Classes

Objective: find note-level rules Limit predictions to categorical decisions Timing Dimension: note N is considered

lengthened If the note is lengthened relative to the

instantaneous tempo over the previous note If lengthened relative to local tempo over the

last 20 notes Analogous to this is a note shortened

Target ClassesTarget Classes

Dynamics: louder if Louder than previous note And louder than average level of piece Analogous to this is softer

Articulation Staccato if played duration ratio (PDR) is less than 0.8 Legato if greater than 1.0 Portato otherwise, but study only concerned with staccato

and legato Pedaling not taken into account for articulation

Notes do not necessarily have to fall into one of these classes

Learning Partial Learning Partial Rule-based ModelsRule-based Models

No expectation to cover and describe all instances Describe parts and define in meaningful terms PLCG algorithm developed with these ideas in

mind Goal to come up with rules that covered lots of

cases with good accuracy

Learning Partial Learning Partial Rule-based ModelsRule-based Models

General Steps Separation into subsets Learning partial rules within subsets Merge all rules Clustering of rules One generalization per cluster Optimize trade-offs (coverage vs. accuracy)

Result: 383 specialized rules narrowed to 17 general rules

Experimental ResultsExperimental ResultsTiming: Lengthening NotesTiming: Lengthening Notes

"Lengthen the middle note in a “cummulative” 3-note rhythm situation (ie, given 2 notes of equal duration followed by a longer note, lengthen the note that precedes the final, longer one).” Most important one as it has highest prediction value

“Lengthen a note if it is followed by substantially longer note (ie the ratio between its duration and the duration of the next note is < 1:3)”

“Lengthen a note if it preceds an upward melodic leap of more than a perfect forth, if it is in a metrically weak position, and if it is preceded by (at most) stepwise motion” 2 cases above have atleast 70% prediction rate

Experimental ResultsExperimental ResultsTiming: Lengthening NotesTiming: Lengthening Notes

“Lengthen a note if it preceds an upward melodic leap of more than a perfect forth, if it is in a metrically weak position, and if it is preceded by (at most) stepwise motion” More of a “tendancy” than a rule

Interesting Note: previously observed But not over such a large data set

Experimental ResultsExperimental ResultsTiming: Shortening NotesTiming: Shortening Notes

Difficult, but understandable No strong rules, but a few tendencies “Shorten a note in a sequence PN-N-NN if it is

longer than its predecessor and longer than its successor.”

“Shorten a note in fast pieces in 3/8 time if the duratio ratio between previous note and current note is larger than 2:1, the current note is at most a sixteenth, and is again followed by a longer note.” Example of a specialized rule Correlation with Gabrielsson 1987

Experimental ResultsExperimental ResultsDynamics: Stressing NotesDynamics: Stressing Notes

Clear rules emerge, low coverage Interesting note: relating stress to

melodic contour Upward melodic movement

Observation by previous research as well “Stress a note by playing it louder if it is

preceded by an upward melodic leap larger than a perfect fourth.”

Experimental ResultsExperimental ResultsDynamics: Stressing NotesDynamics: Stressing Notes

“Stress a note by playing it louder if it forms the apex of an up-down melodic contour and is preceded by an (upward) leap larger than a minor third.”

“Stress a note by playing it louder if it at least twice as long as its predecessor, is reached by upward motion, and is in a quite strong metrical position.”

Experimental ResultsExperimental ResultsDynamics: Attenuating NotesDynamics: Attenuating Notes

Difficult to predict “Attenuate a note by playing it softer if it is less

than 1/5 the duration of its predecessor.” “Attenuate a note by playing it softer if it is

preceded by a downward leap larger than a major third, is metrically weak, and is preceded by a note at least 1/3 of a beat long.”

“Attenuate a note by playing it softer if it is preceded by a downward leap larger than a perfect fifth and is metrically weak.”

Observation: linking metrically weak notes reached by downward leaps

Experimental ResultsExperimental ResultsArticulation StaccatoArticulation Staccato

Most easily predictable, 4 strong rules “Play a note staccato if the note is marked with a

staccato dot in the score.” “Play a note staccato if it is followed by a note of

the same pitch (ie the interval between the note and its successor is a unison).”

Observations: Combine for +90% accuracy & 6,000 cases Previously observed in KTH Rules (Friberg 1995) Physical reasons and explanations

Experimental ResultsExperimental ResultsArticulation StaccatoArticulation Staccato

“Insert a micropause after a note if it precedes an upward leap larger than a perfect fourth and is metrically weak.”

“Insert a micropuase after a note of it is reached by downward motion and is followed by a note more than twice as long (ie the ratio between its duration and duration of the next note is < 0.4).”

Observations: Correlation to lengthening rules Supported by “Cumulative Rhythm” (Nramour 1977) Articulation Staccato 30% of expression observed

Experimental ResultsExperimental ResultsArticulation LegatoArticulation Legato

Most difficult to predict A LOT fewer instances vs. staccato No markings on score

Low prediction rate (53.7%) “Play a note legato if it is not marked

staccato in the score, if it forms the apex of an up-down melodic contour, if it is quite short (<1/3 of a beat), and is metrically quite strong.”

Observations: Melodic peak legato?

Quantitative Validation:Quantitative Validation:Generality IGenerality I

Different Performer (Philippe Entremont) Same pieces No significant degradation in coverage and

accuracy Exception of “softer”

Higher coverage in “lengthen” “louder” “staccato”

Quantitative Validation:Quantitative Validation:Generality IIGenerality II

Testing on Different Styles & Artists 2 Chopin pieces 22 skilled pianist from Univ. of Music in Vienna

Surprising Results “softer” and “legato” unpredictable “louder” high % of positive examples, but

high level of false predictions too “lengthen,” “shorten,” & “staccato”

extremely good Need more diversity of pieces

ConclusionConclusion

Small Step Basic & simple rules Robust model of local expression principles Observations from other researchers Autonomous discovery Large data sets Possible foundation

Further ResearchFurther Research

Further evaluation of rules different performers Different types of music

Extension to other dimensions (e.g. Harmony)

Going beyond note level (e.g. phrase structure)

Comprehensive multi-level model