Music ColoringMusic Coloringfor Biofeedbackfor Biofeedback

Presented ByPresented By

Eyal BallaEyal Balla

ballaeya@tau.ac.ilballaeya@tau.ac.il

The Demo System

EKG Monitor

PC with A/Dand music coloring software

Patient

Original Music

Colored Music

Music Coloring What is it?

Music coloring is changing of music in different aspects (such as pitch, rhythm, timber, volume, reverberation and other aspects of the music) in order to embed data into the music.

Music Perception in Humans

Recent research shows that several distinct neural functions exist in music perception: Volume Pitch Rhythm Timber Direction of source of music

This introduces the possibility of multidimensional display with audio.

Moreover music has been shown to have effect on human breathing and heart rate

Music Perception in Humans

Visual DisplayAuditory Display

Number of dimensions is limited to 3D

Able to percept many dimensions such as pitch, rhythm, timber etc.

Uses full attention of human using it.

Uses partial attention. Other tasks can be done simultaneously.

Previous Works in Auditory Display

Sonification Mappings: sMax: A Multimodel Toolkit for Stock Market Data

Sonification. Maps stock market data into multi-streams of audio. User has control of mapping. Requires training. Mapping is through pitch, pulse speed, brightness and attack time of different instruments

MarketBuzz. A sonification of real-time financial data. 4 behaviors were identified and sonified in beeps and pitch sonifications. Requires training.

Previous Works in Auditory Display

Polyrhythm in the Human Brain. Sonification of EEG brain waves in four different streams according to EEG wave types.

Listening To the Mind listening. Sonification of ECG heart rate and two streams of EEG with difference between the two temporal sections (above ears) were used as stereo positioning.

Previous Works in Auditory Display

Feedback Systems: Auditory Feedback of Human EEG for Direct

Brain Computer Communication. A spelling machine for the handicapped was built. User would use SCP EEG brain waves to control it. Learning was through auditory instruction. They also found other EEG waves controlable using multi-stream audio teaching.

Previous Works in Auditory Display

Previous Works in Auditory Display

Data Analysis through Auditory Display: Applications in Heart Rate Variability. The work deals with creating a auditory system for heart rate variability analysis. The auditory display created is made of several streams each representing a different aspect of heart condition. The work mentions the importance of using pitch and volume.

Conclusion of Current Works

Several orthogonal features of music have been proved as distinguishable by humans in experiments

Sonification is a good way of displaying multidimensional data.

Works usually create new music according to data streams extracted from the raw data.

Current Directions

Last year’s workshop created an interface which inserts samples of EKG into MAX/MSP environment using a A/D sampler.

The workshop showed that online real-time analysis of this data is possible using the MAX/MSP development environment.

Current Directions

We are looking for auditory “Axis” which can be precept orthogonally.

We are using already build components which color music to learn the reaction graph of humans to change in different music features.

References

The structural Components of music perception. Herve´ Platel, Cathy Price, Jean-Claude Baron, Richard Wise, Jany Lambert, Richard S. J. Frackowiak, Bernard Lechevalier and Francis Eustache

Max/Msp 4.5 Documentation. http://www.cycling74.com/products/dldoc.html

sMax: A Multimodel Toolkit for Stock Market Data Sonification. Fabio Cifariello Ciardi

MARKETBUZZ: Sonification of real-time financial data. Petr Janata and Edward Childs

References

Polyrhythm in the Human Brain. Thomas Hermann, Gerold Baier, Markus Muller.

Listening To the Mind listening. Hans van Raaij

Listening to the mind listening concert. http://www.icad.org/websiteV2.0/Conferences/ICAD2004/concert.htm

Auditory Feedback of Human EEG for Direct Brain Computer Communication. Thilo Hinterberger,

Gerold Baier, Jürgen Mellinger, Niels Birbaumer Data Analysis through Auditory Display:

Applications in Heart Rate Variability. Mark Ballora

MAX/MSP Introduction

MAX/MSP is an object oriented development environment for music synthesis and creation.

The MAX part of the environment handles MIDI format manipulation.

The MSP part of the environment handles signal processing and manipulation.

MAX/MSP Example

The following example outputs via audio several 2 dimension data items using 2 streams of data for pitch and volume.