Collaborative Jazz Improvisation as an example of Cognitive and

Network Extension

David Mott, John Ibbotson, IBM

12th May 2009

Structure

• Introduction– extended cognition– jazz playing

• Jazz as collaborative extended cognition

• Jazz as an networkable activity

• Experiments

• Relevance to ITA

Introduction

What is Cognitive Extension?

• Coupling with external artefacts– counting on fingers

• Socially extended– language to share thoughts, refine concepts, execute

plans

• Sense of self– affected by interaction with others– confidence as an improver of performance

[The Extended Mind, Clark and Chalmers]

Examples that convince me of Cognitive Extension

• Driving a car:– the car “drives itself”– I have the same “feeling of vulnerability” when

I turn that I have with my own body

• Playing jazz music– (on a good day) the piano “plays itself”

• Consider the jazz example …

Leaving artefacts in the environment

• augment memory– memory == existence of an entity in a place– keys in same place– (plant triggers in memory)

• reminders/requests to do things– notes, paper trails

• for self and others

Using tools

• notepads

• calculators

• hoey [cognition in the wild]

Is jazz cognitive and capable of cognitive extension?

Jazz group

• instruments are tools

• need to collaborate

• need to problem solve

Use of patterns

• Some aspects of performance is realtime– no time to construct totally new melody lines– people use patterns previously learnt– but not necessarily exact note-for-note, so

cognition is still required

• Other aspects are less realtime– who is doing next solo– this can be done by problem solving

Is playing jazz NEC?• Is it Cognitive?

– a complex problem solving activity involving sequencing, tempo, structure (you only have abstract chords for guidance), in real time

– Improvisation requires more than just muscle memory• Is it network?

– must synchronise, respond, and lead other members of the group (different roles for same player at different times in an ad hoc way)

– If you cant hear, you cant play– If the “groove” is good, you play better (mismatch of different player’s rhythms can destroy

playing)– If the “cross-musician feeling” is good you play better (eg take up of others improvisation and

rhythmic ideas)• Does the tool (the piano) assist?

– out of tune means you cant play– good key action leads to better playing– good sound quality leads to better playing– there is an intimate connection between tool and mind

• Do the tools interact?– bad matching (eg guitar and piano in same “tonal register”) can destroy playing

• Therefore playing jazz is definitely NEC!– The sum is significantly greater than the whole, and the network and tools are essential– (I am talking here about quality of playing - achieving a “jazz feel”, not basic note work)

Do jazz performers leave artefacts?

• Does one performer leave something for later use?– to another performer– to himself– to the “environment”

• e.g. start a song with a certain rhythm, tempo?– really setting up a pattern

Instruments as tools

• Does the particular layout of the instrument enhance the cognitive extension?

• Most jazz tools have a visual component (keyboard “lays out” chords spatially but different transpositions have different patterns)

Do performers “play” other instruments (use other tools)

• Does piano player “play” the drums?– piano player does play rhythmically– if the piano is “heard” as drums then this is

like playing a drum?– eg the passing drum rhythm to drummer

• Does piano player “play” the bass?– e.g. if bass is lost then piano can play the

bass line to resynch

Jazz performance - Tactical and Strategic

• Strategic– Book the hall and the group– Advertise

• Tactical– perform a piece

Can jazz be network extended?

Utility of the Network

• To achieve the quality of face to face?– when communication MUST be distributed, with

reduced communication, can the network rise to the same quality as face to face?

• To improve over face-to-face?– Are there situations where the network can enhance

performance eg• training• direction• better awareness of what others are doing• better awareness of holistic view of performance

What does NEC give you?

• People playing jazz together play better than on their own

• Other players contribute to creativity

• But group must also be disciplined for this to happen– some “maverick” behaviour (strange chords,

rhythms, phrases) may add to creativity, but too much is disruptive

Confidence

• Confidence is key to quality of playing and integration

• Easy to lose confidence if group not working together, not synchronised

• Can the players help increase confidence

• Can the network help increase confidence

• Might a bad network reduce confidence?

• Rhythmic stability seems very important

Rhythmic collaboration (1)

• Ending a song• Actions

– looked at the drummer and played a rhythm– taken over by drummer– subsequently mutual ending

• Needed– visual communication– music communication– shared schema/pattern

Rhythmic collaboration (2)

• Guitar lost synchronisation– Save the Last Dance for Me has unusual 5-bar

phrase lengths– Guitarist changed chord too early– Pianist emphasised new chord at the right place and

guitar resynchronised

• Needed– music communication (both ends)– musical “validation” ability– shared memory of song

Schemas

• Schemas are typical patterns/stories we expect to happen– “eating at restaurant”– “dodgy accountants” – enemy movements?

• Musical Schemas– 4 bar phrases– “Surprise Symphony” [play example]– jazz structure: tune, solo, tune

What is communicated?• Organisational

– who is to solo next– when a solo is finished– timing of piece– when to complete a piece

• Leadership– leadership passes between the players, semi ad hoc

• Musical Patterns– Tempo, eg change to double time– Style, eg “bebop”, “latin”– Phrases, specific phrases may be echoed, or modified– Harmony, eg player may lay down a chord substitution

and this is taken up by others

Communication of Information

• Rhythmic Metre (beat), the gross timing of the piece sync the “tapping of feet”

• Rhythmic Style, the gross “type” of piece, eg Latin, Swing, Rock selection of accompaniment pattern

• Rhythmic Confidence/Stability, whether the drummer is reliable personal confidence need to “lead the rhythm”? sparseness of accompaniment

• Harmonic Progression, roots sync the place in the sequence more space to play richer “higher voicings”

• Harmonic Progression “where I am”???

Communication of Information

• Rhythmic Metre (beat), the gross timing of the piece sync the “tapping of feet”

• Rhythmic Style, the gross “type” of piece, eg Latin, Swing, Rock selection of accompaniment pattern

• Rhythmic Confidence/Stability, whether the drummer is reliable personal confidence need to “lead the rhythm”? sparseness of accompaniment

• ?? “easier to play if the piano sound is good” ??

Other’s tools

• Is it possible to extend the mind into others instruments?

• In a genuinely collaborative situation can this achieve a deeper common understanding?

• Would need intimate feedback between other tool and ones brain, via network

Key collaborative performance aspects

• Synchronised rhythm– style (accents, patterns) varies (swing, latin)– key aspect for a good performance– key for confidence

• Harmonic structure– where are we in the chord sequence?

• Organisational structure– when do we end and change solos?

Creating jazz NEC

How do we create a Jazz NEC?• MIDI instruments:

– keyboard– midi drums– midi guitar (or use keyboard)

• MIDI is a stream of digitised events that can be communicated, instrumented and recorded

• Setup a MIDI “group” across the network with remote players, and play!– connect via sensor fabric– build OpenSim module to communicate MIDI?

NEC over the Sensor Fabric

Fabric Node

Sensor Client

Sensor ClientSensor

Client

SynthSynth Synth

Plugins

Dashboard

Control

Data

Control

Experiments

How do we evaluate a Jazz NEC?

• Evaluation is “quality of jazz feel”– must be done by other musicians (or keen listeners)

subjective feel– BUT I believe you will get a good degree of

agreement (you can tell when a band is “together” or “tight”)

• Could you analyse the MIDI stream mathematically?– Not sure, you might pick up “gross defects” (mismatch

of rhythm, complete collapse of group), but for the “jazz feeling” I suspect this is elusive

What can be measured?

• Degree of “music playing itself”– subjective judgement by player

• post hoc listening and recall (there I was playing well, there I got lost)

• additional effector (foot button?) pressed to indicate “im now in the flow”

• Emotional State– Emotive-like devices, measuring emotion, mood, physiological

factors

• Group review by players– post hoc listening and recall– discussion of what was going on, issues (I couldn’t hear the

guitar, the drums were lagging, I felt the group was not together)

• …

Experiments

• Effect of network quality– can artificially reduce MIDI signals between players

• reduce volume so cant hear drummer• simulate bandwidth reduction by dropping packets

• Effect of face-to-face presence– playing over headphones vs being in visual contact

• Effect of injecting information– put drum “double time rhythm” onto the network to

see if it is taken up by others

• If things go wrong, can the group recover?

QOI

• Quality of Information measures could be applied to performance?

Measuring vs changing

• Could just measure a performance

• Could introduce a change and see the effects

• It may be premature to experiment with the effects of introduced change

Simple intervention – play to click

• Rising to f-f:– prioritise click over full drums, given reduced

bandwidth– agent to approximate drums and send to players

• Improving f-f– already used in real performances– would click give confidence to new players– intervene with double time click?– auto quantise drums – would this increase

confidence?

BeatFinder

• Locate “beats” in music– 1, 2, 3, 4, 1, 2, 3, 4– a complex task, lets assume we can do it

• Show coloured screen for “in time”, “early” and “late”?

• Detect individual accuracy of beats• Detect collaborative accuracy of beats• Work out and display where in the music

we are

Displaying place in tune

• use beat tracker to calculate place in tune– jazz navigational structure usually fairly

simple– would need to visualise place in context?

• would this improve confidence in new player?

• or take away the skills learning?

Experiments with beat finder

• Surely groups already synchronise their playing?– NO, not if nervous or multiple versions of rhythm and

not clear where beat is

• Possible experiments:– For individual practise– Dashboard view of collaborative performance– See other players beats (all, or selected)– One player (drummer, trainer) pushes a

synchronisation pulse to other players

• What effects would this have?

Using network for training

• transcription/copying is very important for learning

• could a trainer play examples to copy?– in the context of a networked performance– note-for-note copy or approximate patterns

• could a trainer give real-time cues?

The relevance to ITA

What is the point of jazz in the ITA?

• An example of extended mind– well understood, easy to relate to (for some)– an articulation of the abstract cognitive

concepts

• NATO uses it as an example of C2– ????

• Realistic test bed of some complex experiments

• Stimulation of new ideas

Metaphors

• (Lakoff) Cognitive metaphors bridge from concrete to abstract– LIFE IS A JOURNEY– TIME FLOWS FORWARDS– EMOTIONS GO UP AND DOWN

• (Saslow) Musical Metaphors– MUSIC IS A PATH– MUSIC IS A FORCE

The FORCE Metaphor

MUSICAL EXAMPLE Military TERMSDEFENCE, SEIZE

Jazz as extended mind

• coupling with external artefacts– finger positions on keyboard to calculate harmonic

progressions

• socially extended– other people as memory of tunes, styles, rhythms,

harmonic progressions– synchronisations of rhythms

• sense of self– jazz playing in social context affects sense of self