Music and Text Generation “in the style of”

François Pachet

SONY CSL & LIP6, UPMC

Constraints and composition

Constraints and text writing

Palindromes:- « A man, a plan, a canal: Panama »- Georges Perec’s palindromes (1,247 words)

Lipogrames- G. Perec “La Disparition” without voyel “e”- “Les revenentes (Perec, texte)”, with only voyel ”e”

« Telles des chèvres en détresse, sept Mercédès-Benz vertes, les fenêtres crêpées de reps grège, descendent lentement West end Street et prennent sénestrementTemple Street vers les vertes venelles semées de hêtres et de frênes près desqelles se dresse, svelte et empesé en même temps, l'Evêché d'Exeter.

Creativity often arises from playingwith styles

Ghedini et al. The Flow Machines project. Ijcai 2013 and AAAI 2013 best video award

Imitative Sequence Generation

That sounds/reads/looks like 𝐶AND

enforce specific properties:domain-dependentuser defined

Given a corpus, i.e. a set of finite sequences

𝐶 = 𝑆1, 𝑆2, … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1, 𝑆𝑖2, … , 𝑆𝑛

𝑘

Generate 1 sequence / the best

Generate all sequences

Generate a representative subset of sequences (≈ sampling)

Imitative Sequence Generation

That sounds/reads/looks like 𝐶AND

enforce specific properties:domain-dependentuser defined

Given a corpus, i.e. a set of finite sequences

𝐶 = 𝑆1, 𝑆2, … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1, 𝑆𝑖2, … , 𝑆𝑛

𝑘

Generate 1 sequence / the best

Generate all sequences

Generate a representative subset of sequences (≈ sampling)

Optimization

Imitative Sequence Generation

That sounds/reads/looks like 𝐶AND

enforce specific properties:domain-dependentuser defined

Given a corpus, i.e. a set of finite sequences

𝐶 = 𝑆1, 𝑆2, … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1, 𝑆𝑖2, … , 𝑆𝑛

𝑘

Generate 1 sequence / the best

Generate all sequences

Generate a representative subset of sequences (≈ sampling)

Statistical inference

Optimization

Imitative Sequence Generation

That sounds/reads/looks like 𝐶AND

enforce specific properties:domain-dependentuser defined

Given a corpus, i.e. a set of finite sequences

𝐶 = 𝑆1, 𝑆2, … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1, 𝑆𝑖2, … , 𝑆𝑛

𝑘

Generate 1 sequence / the best

Generate all sequences

Generate a representative subset of sequences (≈ sampling)

Statistical inferenceCSP & Global constraints

Optimization

Imitative Sequence Generation

That sounds/reads/looks like 𝐶AND

enforce specific properties:domain-dependentuser defined

Given a corpus, i.e. a set of finite sequences

𝐶 = 𝑆1, 𝑆2, … , 𝑆𝑛 𝑤ℎ𝑒𝑟𝑒 𝑆𝑖 = 𝑆𝑖1, 𝑆𝑖2, … , 𝑆𝑛

𝑘

Generate 1 sequence / the best

Generate all sequences

Generate a representative subset of sequences (≈ sampling)

Statistical inferenceCSP & Global constraints

Information geometryOptimization

Style and Markov chains

Markov Hypothesis

𝑃 𝑠𝑖 𝑠1, 𝑠2, . . , 𝑠𝑖−1) = 𝑃 𝑠𝑖 𝑠𝑖−1)

Random walk

Generating Markov Sequences

𝑃 𝑠𝑖 𝑠1, 𝑠2, . . , 𝑠𝑖−1) = 𝑃 𝑠𝑖 𝑠𝑖−1)

Random walk:generate X1 with prior P(X1),then X1with P(X2 X1then find the longest prefix 𝑘 ≤ 𝑚𝑎𝑥 for which

there are at least 𝑝 continuations (𝑝 ≥ 1) and drawP(Xn Xn−k, Xn−k+1, ⋯ , Xn−1

𝑋1 𝑋2

𝑋1 𝑋2 𝑋3 𝑃 𝑠𝑖 𝑠1, 𝑠2, . . , 𝑠𝑖−1) = 𝑃 𝑠𝑖 𝑠𝑖−1, 𝑠𝑖−2)

Order 1

Order 2

Variable order with max bound

Markov and Music Improvisation: Continuator

Pachet, F. The Continuator: Musical Interaction with Style J. of New Music Research, 2003, best paper awardPachet, F. Music Interaction With Style in SIGGRAPH 2003 Abstracts and Applications, San Diego, 2003

A Veenendaal

Alan Silva

Bernard Lubat

Interactive Markov chains for stylistic imitation

Continuator

Continuator with children

Addessi, A.-R. and Pachet, F. Experiments with a Musical Machine: Musical Style Replication in 3/5 year old Children. British J. of Music Education, 22(1):21-46 March 2005

2 days later, the childinvents a new style,

which sounds like Bach arpeggios …

Composition

Arranging

Accompaniment

Style?

Improvisation

Problem: control is incompatible with Markov

=

𝑃 𝑠𝑖 𝑠1, 𝑠2, . . , 𝑠𝑖−1) ≠ 𝑃 𝑠𝑖 𝑠𝑖−1)

Long-range correlationsGlobal constraints

Because of

control ?

Unary, binary, nary constraints (music « rules »)

Max order (avoid plaggiarism)

Meter (essential !)

Cardinality (grains of salts)

Alldiff (ensure diversity)

Spreading and distributions (natural properties, e.g. 1/f)

Etc.

Unary, binary, nary constraints (music « rules »)

Max order (avoid plaggiarism)

Meter (essential !)

Cardinality (grains of salts)

Alldiff (ensure diversity)

Spreading and distributions (natural properties, e.g. 1/f)

Etc.

.. impossible with random walk in Markov chains

Constrained Markov sequences: a new class of problems

General Solution for optimization problems:Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences, Constraints, 2011.

Unary constraints solved in linear time:Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011

Meter in pseudo-polynomial time:Roy, P. and Pachet, F. Enforcing Meter in Finite-Length Markov Sequences. AAAI, 2013

MaxOrder (= enforcing novelty) in linear timePapadopoulos, A., Roy, P., Pachet, F. Avoiding Plagiarism in Markov Sequence Generation, AAAI, 2014 http://www.flow-machines.com/maxOrder

Distribution (= spectrum, 1/f) Pachet, F. Roy, P. Sakellariou, Generating 1/f noise sequences as a CSP, IJCAI, 2015

Virtuoso phrase example:Stefano di Battista, « Night in Tunisia »

The Physiological Perspective on Virtuosity

– Strong human motor and perception limits (seeextreme drumming)

– Virtuoso = 10,000 hours of practice (Sloboda et al.)

– Virtuosos dont make mistakes, because theysuppress the slow monitoring functions of the brain:

Virtuosos avoid the speed traps of their pre-frontalcortices1

In other words: they don’t think

1 Justin London, The Psychology and Neurobiology of Musical Virtuosity, Whitehead Lecture in

"Cognition, Computation, and Culture," given at Goldsmiths College, U. London, 2010.

The AI View on Virtuosity

• Virtuoso do extraordinary things (super experts)

• Compilation of know-how in the body is a wayto « solve problems »

• From the viewpoint of Markov chains:

Well-defined, apparently hard problem

« Virtuosos are NP-hard problem solvers »

Pachet, F. Bebop Virtuosity Explained McCormack & d'Inverno, Eds. Computers and Creativity, Springer, 2012

Unary constraints can be solved in real-time

If one considers only:

– Unary (1 variable) constraints at any position in the sequence, or

– n-ary between n consecutive states (n < order)

Then ∃ a Markov chain M’ s.t.

– M’ generate only the sequences satisfying the constraints

– M’ and M are statistically equivalent

=> virtuosity problem solved

Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011

Example

Generate from this example

Generate all 4-notes melodies ending with D with theircorrect probabilities (sampling)

– Reformulation as CSP with 4 variables

– Arc-consistency of the network with Markov constraint

– Retro-normalisation

(was ½ 1/6 1/3)

Papadopoulos, Pachet, Roy, Sakellariou, Exact sampling for regular and Markov constraint with belief propagation, submitted

Virtuoso, with Mark d’Inverno (U. of London)

Markov and meter

General Solution for optimization problems:Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences, Constraints, 2011.

Unary constraints solved in linear time:Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011

Meter in pseudo-polynomial time:Roy, P. and Pachet, F. Enforcing Meter in Finite-Length Markov Sequences. AAAI, 2013

MaxOrder (= enforcing novelty) in linear timePapadopoulos, A., Roy, P., Pachet, F. Avoiding Plagiarism in Markov Sequence Generation, AAAI, 2014, http://www.flow-machines.com/maxOrder

Distribution (= spectrum, 1/f) Pachet, F. Roy, P. Sakellariou, Generating 1/f noise sequences as a CSP, IJCAI, 2015

Meter

• In the style of M. Legrand

• Total duration = 8-bars

• No note spanning bar lines

• Total duration is 𝑖=1𝑛 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛(𝑛𝑜𝑡𝑒𝑖)

• Duration until 𝑛𝑜𝑡𝑒𝐾 is 𝑖=1𝐾 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛(𝑛𝑜𝑡𝑒𝑖)

Impossible to explore all chains

• For n=23, there are 13 millions paths• Exponential growth with sequence length

…

Additive number theory

• Counting integer points:– Polytopes

– Sumsets ℎ𝐴 = 𝑎1 + 𝑎2 +⋯+ 𝑎ℎ| 𝑎𝑖 ∈ 𝐴

– Naively: ℎ(𝐴) ≈ ( 𝐴 /2)ℎ

• Khovanskii theorem: the number of integer points in a convex polytope is a polynomial:– ℎ(𝐴) = 𝑃(ℎ) with P polynomial (for h big enough)

– Degree of P is < 𝐴

Khovanskii, A. 1992. Newton polyhedron, Hilbert polynomial, and sums of finite sets.Functional Analysis and Its Applications 26(4):276–281

After 4 steps

• N = 4 => more than 130 paths

• But only 3 unique path lengths: 4, 7 and 10

We explore the set of all path lengths

• n=23, 13 millions paths

• Only 9 unique lengths !

Capturing the style of Ray d’Inverno

Original recording of Ray’s comping on Girl From Ipanema

Constraint: Giant Steps score

Basic rendering

Ray’s reaction

Hi Francois, It is fantastic. Well done.

It is very realistic - you can even hear the wrong notes in some of the chords !

Keep up the good work.

Cheers, RealRay

Pachet, F. and Roy, P. Beyond minus ones: Virtual Band Siggraph talk and demo, Los Angeles, 2012

Giant Steps by Wagner

Capture the style of Take 6!

One of the best a capella jazz groups. 10 Grammy awards

Rich harmonisations, veryrecognisable

Very hard to imitate, even to transcribe!

Reaction of composers: Ivan Lins’ The Island

Grammy-winning Brazilian songwriter. His hit "Love Dance" is one of the most re-recorded songs in musical history (Wikipedia)

The Island harmonized by Take 6Rio, 2013

Pachet & Roy, Non-Conformant Harmonization: The Real Book in the style of Take 6, Int. Conf. on Comput. Creativity, 2014.

Beyond random walk: principled generation

General Solution for optimization problems:Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences, Constraints, 2011.

Unary constraints in linear time:Pachet, Roy & Barbieri, Finite-Length Markov Processes with Constraints, IJCAI 2011

Meter in pseudo-polynomial time:Roy, P. and Pachet, F. Enforcing Meter in Finite-Length Markov Sequences. AAAI, 2013

MaxOrder (= enforcing novelty) in linear timePapadopoulos, A., Roy, P., Pachet, F. Avoiding Plagiarism in Markov Sequence Generation, AAAI, 2014, http://www.flow-machines.com/maxOrder

Distribution (= spectrum, 1/f) Pachet, F. Roy, P. Sakellariou, Generating 1/f noise sequences as a CSP, IJCAI, 2015

The Max Order problem

Max OrderDef: The max order of a generated sequence is the maximum length of replication in the original corpus.

Example: an order 1 Markov sequence with max order 7

The MaxOrder automaton

Villeneuve, D., and Desaulniers, G. 2005. The shortest path problem with forbidden paths. European Journal of Operational Research 165(1):97–107.Aho, A. V., and Corasick, M. J. 1975. Efficient string matching: An aid to bibliographic search. CACM 18(6):333–340.Gilles Pesant: A Regular Language Membership Constraint for Finite Sequences of Variables. CP 2004: 482-495

L 𝑀𝑎𝑥𝑜 𝑚𝑎𝑥 = 𝐿 𝑀 ∩

𝑖=1, 𝑐𝑜𝑟𝑝𝑢𝑠 −𝑚𝑎𝑥

𝐿 𝐴 𝑛𝑜𝑔𝑜𝑜𝑑𝑖

Our contribution:• Build this automaton quickly• Naive approach not polynomial, Polynomial construction algorithm inspired by 1, 2

• Automaton is fed to the regular constraint to create ALL length-L sequences

The MaxOrder automaton

Papadopoulos, Roy & Pachet, Avoiding Plagiarism in Markov Sequence Generation, AAAI, 2014

Papadopoulos, Roy & Pachet, Generating non-plagiaristic Markov sequences with max order Sampling, Creativity and Universality in Language, Degli Esposti, Altmann, Pachet Eds, Springer, Morphogenesis series, 2015

Same with Leadsheet generation

Pachet, F. and Roy, P. Imitative Leadsheet Generation with User Constraints, ECAI 2014

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

Distribution of chunk size

Order min 1

Order min 2

Order min 3

Order min 4

Order min 3max 10PlagiarismJunk Sweet spot

MIN ORDER = 3 & MAX ORDER = 10

MaxOrder Constraint

Leadsheet Generation

Pachet & Roy, Imitative LeadsheetGeneration with User Constraints, ECAI 2014

LSD: http://lsdb.flow-machines.com

Pachet et al, A Comprehensive Online Database of Machine-Readable Lead Sheets for Jazz Standards, ISMIR 2013

www.flow-machines.com/leadsheetgeneration

Flow Composer

Without production Original (produced)

There is a huge difference between a SCORE and actual MUSIC

Without production Original (produced)

Without production Original (produced)

Music Synchronization (with M. Marchini)

Revisiting Ode to Joy (Beethoven)

From Chi Mai (Ennio Morricone )

From Penny Lane (the Beatles)

From Prayer in C (Lilly Wood & The Prick)

Marchini, Pachet, Roy, Synchronization Constraints: An Audio-content Based Method for Generating Multi-Instrumental Music, submitted Ismir 2015

Brazil guitar style capture

Style Capture in Brazil

Listening to Generation

Generation examples

ReChord generation

Insensatez by Geber Ramalho Blue in Green

Arrastao by Giordano Cabral Canto Triste (Edu Lobo)

Pop chords by Mathieu Ramona Michelle

Friends to go by Benoit Carré All of me

Original played by various musicians Automatic generation on arbitrary leadsheets

Ramona, Pachet Groove preserving concatenative synthesis, submitted

Benoit Carré Case StudyFrench Pop Artist

Original version by Benoit Carré

Version by Enzo Enzo, popularartist in France

Original guitar accompaniment by Benoit Carré

Michelle

Blue in GreenReChord generation

Markov Constraint applied to text« Yesterday » by the Beatles

Barbieri, Pachet, Roy & Degli Esposti, Markov Constraints for Text Generation with Style, ECAI 2012

Yesterday … by Bob Dylan

Innocence of a story I could leave today

When I go down in my hands and pray

She knocked upon it anyway

Paradise in the dark side of love it is a sin

And I am getting weary looking in

Their promises of paradise

Now I want to know you would be spared this day

Wind is blowing in the light in your alleyway

Innocence in the wind it whispers to the day

Out the door but I could leave today

She knocked upon it anywayKhatchatourov, Pachet et al. Action Identity In Style Replication Systems , submitted to Music Perception

Yesterday by the Beach Boys

Anyone to call after night and start today

Back home let watch the long time I pray

Here we surround you anyway

Heavenly skies and rock like that in my mind hey

Walk a summer summer summer day

You realize a deity

At a time for a change of the while I say

Guys I wonder love I have took a runaway

Anyone to me yeah I over heard you say

Moved her so you will love like today

Here we surround you anyway

Yesterday by ACDC

Fortunate if you wanna take you out to playStage I'm big and I want to sayI get enough and I can playSatellite blues yeah yeah yeah yeah yeah you shook meToss off buddy she's gotta seeA fireball in the back onceStage I'm in the high I said it's wayToss off buddy she's got it and I'm heyFortunate if you ain't nothin I can playPlayed all the time you just keep away

Yesterday by Johnny CashTheater of your liquor I don't take pay

Gone in the clouds how to live this way

Hit him again and when my way

Theater of your plans I'm down and tell me

Played the boogie in the saddle he

A wanderer a wandering

Play for you and I know what's the way they

Played the boogie in the harp with the key the way

Honeycomb and live in the water and bread they

Play in the sun and I rode away

An interesting and complex global property: Palindromes

• Palindromes are extreme display of human creativity / virtuosity

• DOC NOTE: I DISSENT. A FAST NEVER PREVENTS A FATNESS. I DIET ON COD(Peter Hilton: “one full sleepless night”)

Palindrome Generation

• Goal: construct palindromic sequences from a corpus of N-grams (e.g., google n-grams, text, any sequences, etc.)

• Major difficulty:– Two levels are inter-related: LETTERS and WORDS

• No algorithm to solve this problem– Brute force– Stochastic, HMM, Monte-Carlo, Metropolis– Double recursion– Automata– Graphs– Constraints

The Palindrome Graph(with A. Papadopoulos and J.-C. Régin)

CONJUNCTION graph: Gf x Gb= Tensor product AND encodes the “same character” relation

Papadopoulos, Roy, Régin, and Pachet, Generating all Possible Palindromes from Ngram Corpora, IJCAI 2015

Palindromes: Examples

• ‘Evil on an olive’,

• ‘To lay a lot’,

• ‘Born a man, rob’,

• ‘Till I kill it’,

• ‘God all I had, I hid: a hill a dog’,

• ‘God, a sin: a man is a dog’,

• ‘Sworn in us at a sun in rows’

• ‘Drawn in war, died. I set a gate side, I, drawn in ward’

• ‘Never a way. By a war even’

• ‘Evil as a witness is sent, I was a dog; God as a witness is sent, I was alive’

• ‘Et on a là, la baraba, là, la note.’

• + very very long ones (80,000 words)

Other Games on Words

• Ambiphrases (each direction in a different language)

– El, a Roma se dedica -> Acide de sa morale

– No delay -> Y a le don

– Âme de Roy a le don -> No delay or edema

• More to invent…

See interactive demo at Ijcai 2015, Buenos Aires

Conclusion

• Style as a computational object: an ingredientfor creativity enhancing tools,

• New research problems in sampling fromstatistical models under global constraints

• Fruitful combinations of ideas from discretedomain combinatorial optimization, machine-learning and statistical inference

• Industrial potential in entertainmenteconomy