annotating music and lyrics

Annotating Music and LyricsKristine Monteith

CS 652 - Research ProjectJune 11, 2009

Project GoalFind a suitable song for a

given situationApplications

Indexing songs by topic and mood

Soundtracks for movie scenes

Music therapy groups

What do we want to label? Lyrics

Themes Moods

Music Moods Energy Emotional evocativeness Genre

Song Ontology

How to Extract Features:Lyrics Current state: Keyword search

Find all occurrences of the search term in the song

Find all occurrences of search term synonyms (Using WordNet synsets)

Future work: Extend searches to phrases Determining mood

Supervised Learning

Classifier

Features of a Musical Selection

Prediction of target label

Labeled Training Data

Input Features Bag of Words

Words appearing on page and word counts

Looking for other methods to analyze documents and collect input features

Output LabelsDerived from questionnaireHand-labeled by researchers

How to Extract Features:Music Target labels to predict

Moods (labeled by subject or expert) Energy (determined by direction of

change in biofeedback responses) Emotional evocativeness (determined

by extent of change in biofeedback responses)

Genre (labeled by subject, expert, or clustering)

Input Features:Acoustic Properties

Spectral Centroid Spectral Rolloff Point Spectral Flux Compactness Spectral Variability Root Mean Square Fraction of Low Energy Windows Zero Crossings Strongest Beat Beat Sum Strength of Strongest Beat Strongest Frequency Via Spectral Centroid Strongest Frequency Via FFT Maximum

Input Features:Acoustic Properties

MFCC LPC Method of Moments Partial Based Spectral Centroid Partial Based Spectral Flux Peak Based Spectral Smoothness Relative Difference Functions Area Method of Moments

Input Features:Symbolic Features

Tempo Key Mode Musical form Rhythmic structure Vocalization Instrumentation Melodic contour Harmonic patterns

Output Labels:Questionnaire-based Responses

Output Labels:Physiological Responses Heart rate Breathing rate Perspiration Skin temperature

Each subject will listen to one minute segments separated by one minute of silence

Conclusion Demo: Music Therapist Assistant

Any Questions/Suggestions?