predicting the emotions expressed in music - ida universe · pdf fileusing machine learning to...

30/11/20161 Cognitive Systems, Technical University of Denmark

Using machine learning to decode the emotions

expressed in music

Jens Madsen

Postdoc in sound project

Section for Cognitive Systems (CogSys)

Department of Applied Mathematics and

Computer Science (DTU Compute)

Technical University of Denmark

Website: www.jensmadsen.com

Twitter: @cogniemotion

Email: [email protected]

http://www.jensmadsen.com/


Ubiquity of music


Spotify (2006) 36+ million songs, (100+ million users)

Apple music (2015) 37 million songs (17 million users)

Deezer (2007), 40 million songs (16+ million users)

TiDAL (2010), 30+ mio tracks (4+ million users)

YouTube (2005), 30+ mio tracks (1+ billion users)

Globally, digital revenues of music account for 45% of total sales in 2016

Denmark, digital revenues of music account for 74.3% in 2014

Music is with us everywhere


Major selling points have been

Catalogue

Quality

User Interface

Functionality

Streaming services


Design of a music system

SystemUser Interface

30+ mio tracks


• Exploration

– Search (artist, title, genre, emotion, …)

– Interfaces (visual, auditory, tactile representations,…)

• Recommendation

– This is what other people are listening to

– Demographic, Collaborative, Content-based, Context-based filtering

• Hybrid

– Similar artist, tracks, …

• Editorial

– Playlists, events, etc.

Celma, Oscar. Music recommendation and discovery: The long tail, long fail, and long play in the digital music space. Springer Science & Business Media, 2010.

Music system interfacing


Does recommendation give us what we actually want?

Do people want to explore large music collections?

Design of a music system


Why do people listen to music?


1. Regulate their emotional state

2. Self-awareness and finding identity

3. Social bonding/relatedness

Thomas Schäfer, Peter Sedlmeier, Christine Städler and David Huron” The psychological functions of music listening” in frontiers in

Psychology 13 august 2013

Juslin, Patrik N., and Petri Laukka. "Expression, perception, and induction of musical emotions: A review and a questionnaire study of

everyday listening." Journal of New Music Research 33.3 (2004): 217-238.

Rentfrow, Peter J; ,The role of music in everyday life: Current directions in the social psychology of music,Social and Personality Psychology

Compass,6,5,402-416,2012,Blackwell Publishing Ltd

Scientific answer to why people listen to music


Mechanisms for emotion regulation

Expressed

emotion

Perceived

emotion

Induced

emotionPerformance

/ expression

Audience

Episodicmemory

Emotionalcontagion

Visual imagery

Evaluativeconditioning

Musical expectancy

Brain stem reflexes


Audio signal

Annotations (tags, genre, etc.)

User metadata (age, country, etc.)

Lyrics

Building a music system regulating emotions

Episodicmemory

Emotionalcontagion

Visual imagery

Evaluativeconditioning

Musical expectancy

Brain stem

reflexes

Primary mechanisms

of induced emotions


Using emotional contagion for emotion regulation

(Mis)matching the emotions expressed in

music with the emotional state of the listener

AudiencePerformance

Expressed

emotion

Emotional

state


We need to know what emotions are

expressed in 30+ mio tracks!

Too much data to annotate!

Technical challenge


Create mathematical models that can

predict the emotions expressed in music

Overall goal of research


Predictive model of emotions expressed in music

Elicitation of emotions

Modelling framework

Audio representation

ModelPredictions

Mo

de

lin

g

fra

me

wo

rk

User

Internal

representation

of emotion

Eli

cit

ati

on

of

em

oti

on

s

User interface

Decision

Music database

Feature extraction

Feature representation

Au

dio

re

pre

se

nta

tio

n


ModelPredictions

Mo

de

lin

g

fra

me

wo

rk


30+ mio tracks

User

Internal

representation

of emotion

Eli

cit

ati

on

of

em

oti

on

s

User interface

Decision

Music database

Feature extraction


Au

dio

re

pre

se

nta

tio

n


Feature extraction

Music database

Feature extraction


Au

dio

re

pre

se

nta

tio

n

Pitch

Timbre

Beat

Melody

Chords

Harmony

Rhythm


Features describing music

Music database

Feature extraction


Au

dio

re

pre

se

nta

tio

n

Pitch/Chroma (100 ms) 12 300

Mel-frequency cepstral coefficients (23 ms) 20 1290

Loudness (23 ms) 24 1400

Beat/tempo (10s) 1 15

Dimensions No. time windows




Modelling framework


User

Model

Core affect

Eli

cit

ati

on

of

em

oti

on

s

Predictions

User interface

Mo

de

lin

g

fra

me

wo

rk

Decision

Music database

Feature extraction


Au

dio

re

pre

se

nta

tio

n

User

Model

Internal

representation

of emotion

Eli

cit

ati

on

of

em

oti

on

s

Predictions

User interface

Mo

de

lin

g

fra

me

wo

rk

Decision

Music database

Feature extraction


Au

dio

re

pre

se

nta

tio

n


Elicitation of emotions expressed in music

User

Internal

representation

of emotion

Eli

cit

ati

on

of

em

oti

on

s

User interface

Decision

• Internal representation of emotions

• Define the question to the user

• Define users response format (decision)

• Scale? Rank?


Elicitation of emotions expressed in music

User

Internal

representation

of emotion

Eli

cit

ati

on

of

em

oti

on

s

User interface

Decision


Internal representation of emotion

Valence and Arousal / Core affect

Valence

Arousal

Active

PositiveNegative

Passive


Valence and arousal / Core affect

Valence

Arousal

Active

PositiveNegative

Passive


Predictive model of emotions expressed in musicUser

Eli

cit

ati

on

of

em

oti

on

s

User interface

Decision

Va

len

ce

Arousal


Response format for quantifying valence and arousal model

Excerpt

A

Excerpt

A

Excerpt

BExcerpt

K

RelativeAbsolute




Modelling framework


ModelPredictions

Mo

de

lin

g

fra

me

wo

rk

Music database

Feature extraction


Au

dio

re

pre

se

nta

tio

n

User

Eli

cit

ati

on

of

em

oti

on

s

User interface

Decision

Va

len

ce

Arousal


Modelling framework

𝒳 = 𝑝(𝐗|𝜽𝑛) |𝑛 = 1:𝑁 ∧ 𝐗 ∈ ℝ𝐷𝑥𝑇

𝒴 = 𝑦𝑚|𝑚 = 1:𝑀 ∧ 𝑦 ∈ −1,1

𝜃κ~ℎ𝑎𝑙𝑓𝑠𝑡𝑢𝑑𝑒𝑛𝑡 − 𝑡(ν, η)

k 𝑝(𝐗|𝜽), 𝑝(𝐗|𝜽′) = න𝑝(𝐗|𝜽)𝝆𝑝(𝐗|𝜽′)𝝆d𝑿

𝐟|𝒳, 𝜃κ ~𝒢P 𝟎, k 𝑝(𝐗|𝜽),⋅ 𝜃κ

𝜋𝑚|𝐟, 𝜎 = 𝛷 𝑦𝑚𝑓𝑢𝑚 − 𝑓𝑣𝑚

𝜎2∀𝑚 = 1:𝑀

𝑦𝑚|𝜋𝑚~𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖∓1 𝜋𝑚 ∀𝑚 = 1:𝑀

Discrete choice model

ObservationsUser

Decision

User Interface

Feature Extraction

Experimental Design

Music Database

u

v

Mo

de

l


Au

dio

re

pre

sen

tati

on

Vale

nc

e

Arousal

Likelihood

Decision Making

Latent Function

Model of Core affect


Visualizing the latent function (A-V space)

No. Song name

1 311 - T and p combo

2 A-Ha - Living a boys adventure

3 Abba – That’s me

4 ACDC - What do you do for money honey

5 Aaliyah - The one I gave my heart to

6 Aerosmith - Mother popcorn

7 Alanis Morissette - These r the thoughts

8 Alice Cooper – I’m your gun

9 Alice in Chains - Killer is me

10 Aretha Franklin - A change

11 Moby – Everloving

12 Rammstein - Feuer frei

13 Santana - Maria caracoles

14 Stevie Wonder - Another star

15 Tool - Hooker with a pen..

16 Toto - We made it

17 Tricky - Your name

18 U2 - Babyface

19 UB40 - Version girl

20 ZZ top - Hot blue and righteous

J. Madsen, J. B. Nielsen, B. S. Jensen, and J. Larsen,“Modeling expressed emotions in music using pairwise comparisons.” in 9th International Symposium on Computer Music Modeling and Retrieval (CMMR) Music and

Emotions, 2012.


• Defined a psychological based approach of designing of music system

• Investigated elicitation methods of emotions expressed in music

• Designed a predictive model of emotions expressed emotion

Conclusion


Thank you!

Website: www.jensmadsen.com

Twitter: @cogniemotion

Email: [email protected]

http://www.jensmadsen.com/

predicting the emotions expressed in music - ida universe · pdf fileusing machine learning to...

Documents