cost sid - interactive music 2.0 - short term scientific mission work plan

8/8/2019 COST SID - Interactive Music 2.0 - Short Term Scientific Mission Work Plan

http://slidepdf.com/reader/full/cost-sid-interactive-music-20-short-term-scientific-mission-work-plan 1/11

Interactive Music 2.0 User-driven ad hoc dance clubs Short Term Scientific Mission: COST-IC0601 - WG3Art (Interactive Art and Music)

Candidate: Yago de Quay

Host: fourMs Laboratories, University of Oslo – Norway

Dates & Budget: 1st of August – 30th of September 2010; 2500€

Mission:

Investigate how gestural sonic design can be applied in multi-participatory dance

environments.

Introduction:

Much has been learned in the area of interactive music up to now, spawning new areas such as music

information retrieval, generative music, adaptive scores, virtual instruments and the like. However1,2

group movement descriptors and their music information remain un-investigated. This mission

proposes to fill in this research gap, disseminate the work, and promptly implement the results in

public locations.

Alexander Jensenius has been working in the field of musical gesture analyses for quite some time

and his department has materials to support this STSM. In my Home Institution, Carlos Guedes has

been working extensity with generative music, Fabien Gouyon with music information retrieval, and I

1Rolf Inge Godøy, Alexander Refsum Jensenius. Body Movement In Music Information Retrieval. ISMIR 2009

2Camurri, A., Canepa, C., and Volpe, G., Colet ta, P., Mazzarino, Ferrari, N. Social Active Listening and Making of Expressive Music:

The Interactive Piece The Bow is bent and drawn. DIMEA 2008



2

Interactive Music 2.0 Mission|Work Packages|Budget|Timeline

with sound and interactive systems design. Combined, these diverse backgrounds can help usher

interactive systems into popular public venues.

In contrast to existing work, I intend for the output of this mission to be made accessible, inexpensive,

and easy to implement in multi-participative environments. I plan for the final interactive system to

be scalable and permit interplay between users. Furthermore, using social web instruments, I hope to

raise awareness and help the exchange of knowledge this field of research.

Figure 1 - Mission Goals

The work plan will be divided into 5 work packages. Work Packages 2 & 3 will have routine qualitativeand quantitative testing's on available users, and will be inspired by online community feedback.

run

simulation assemble motion

capturing system developgesture

analysis

define

interactive

music

Mission Goals

GDIF generative

composition

music

information

retrieval

design

interaction

model

software

hardware

user –

generated

content web 2.0

approaches

and techniques

beta

testing alpha

testing

usability

testing define user

mappingtechniques

collaborators professors

sample

groups

draftquestionnaire

collect data



3

WorkPackage 1 initial dissemination

Wordpress blog

Facebookprofile

Microblogging

FriendFeed

Advisory Board

WorkPackage 2 theoretical &

methodological

framework

Mappingtechnique

Musiccomposition

strategy

User testingcommunity

feedback

WorkPackage 3motion capture

system

List of hard &software

Installation site

Motion capturesystem

Questionnaireresults

WorkPackage 4final dissemination

Articles

Videos

Images

Talks


WP 1 Where: Home Institution Funding: No

The first package will focus on preparing the dissemination so that we take full advantage of

techniques such as viral marketing and social learning to bring in individuals as well as communities to

influence and increase the reach of this mission’s results, tools, practices and insights.

WP 2 Where: Host Institution Funding: Yes, partial (1650€) Work package 2 will determine the theoretical and methodological modus operandi to be followed

throughout the mission. It will decide the gesture mapping and music composition strategy.

WP 3 Where: Host Institution Funding: Yes, remainder (850€)

This stage concerns the implementation of the motion capturing system, the simulation, and the

questionnaire.

WP 4 Where: Home Institution Funding: No

Work Package 4 is concerned with the analysis and dissemination of the results, the publication of a

paper, and exhibitions.

Figure 2 - Overview of internal and external deliverables



4

Interactive Music 2.0 Mission|Work Packages|Budget|Timeline - WP 1

Work Package 1: Initial Dissemination

_Week 1 _Home Institution _No Funding

“The term "Web 2.0" (2004–present) is commonly associated with web applications that facilitate

interactive information sharing, interoperability, user-centered design, and collaboration on the3World Wide Web.” This STSM will not build a web application, but will mobilize and interact with on-

line communities using Web 2.0 technologies to foster social interaction and exchange between

stakeholders both on-line and off-line.

My strategy is to empower researchers with the necessary knowledge and tools to effectively

implement the methodology themselves, and foster massive participation of user-generated content

from the scientific community. It will include Blogs, group-related or personal online journals, where

people can post ideas, images, and links to other web pages or sites related with their role in this

STSM as well as micro-blogging tools, like Twitter, in a stream of constantly-updated information. Andin addition, other tools include social bookmarking, shared scholarly references and groups, socia

news, and social publishing. This collaborative environment, where users are expected to co-create

the solutions, leads to a natural acceptance by the stakeholders who will feel empowered to test,

evaluate and report their own experience with the new solutions in a controlled social environment.

Centralized development approach

Technology-driven BroadcastTop-Down

Low communication effectiveness

High costs of content production

Low participation

Limited growth

Mission loosely-coupled integration

Demand-driven ParticipationBottom Up / Edge to Core

High communication effectiveness

Very low costs of content production

Exponentially high participation

Unlimited growth

3Wikipedia, http://en.wikipedia.org/wiki/Web_2.0

BroadcastUser Centric



5


Figure 3 - Broadcast vs. User centric approaches

This mission is positioned closer to the demand of new content and services related with technology-

enhanced learning where a significant percentage of the content will be generated by intermediate

and end users/researchers. As an example, most of my projects are made publicly available and count

with a growing community of users and contributors.

Figure 4 - (Starting from top, clock-wise) Ethnomusicological Research (yagodequay.blogspot.com); Beatpick Social Tagging

(paginas.fe.up.pt/~mm09011/SDI/index.html); Reason Producers Community (reasonproducers.ning.com); Porto’s Home-lessStories (paginas.fe.up.pt/~mm09021/); Blog (yagodequay.com)

The Advisory Board is a project management instrument that provides an

additional layer of quality by having external experts to the consortium assess

the progress and research outputs of the project. However, the advisory board

has been carefully selected with individuals with strong networks that will assist the project in

fomenting the growth of communities and in identifying early adopters amongst universities and

enterprises.

Advisory Board



6


Work Package 2: Theoretical and Methodological

Framework

_Week 2-5 _Host Institution _Partial Funding (1650€)

This initial research step will define the research strategy to be adopted. Considering the various

current forms of music interaction and motion capture, use cases and community feedback will help

steer the research practice. Work Package 2 will figure out the best way to merge the areas of gesture

analysis, music composition and dance. This is important not only for the purpose of this mission, but

for the scientific community at large, for there has been limited research on music information1,2

retrieval from single and, to a lesser extent, group gestures. Furthermore, the theoretical and

methodological model hopes to investigate the following issues: First, the limitation in the method of

music creation. Engineered music, contrary to interactive music, only allows the expert to have a say

on the overall creative direction of a song. Second, non-interactive systems miss out on the plethora

of knowledge and inputs that other users and experts can provide. Third, competition and innovation4,5

in this field barriers to entry imposed by high financial cost lack of an open knowledge community.

This research procures solving these problems by offering an interactive music model for multi-

participative environments. This interactive model is not intended to substitute performers, experts

or engineered content, but intended to add a tool for public interactive media, a tool that can have

benefits on various on education, art and the scientific community.

Below are photos of two interactive music systems that I presented at formal exhibits and danceclubs. They were designed to be inexpensive, intuitive and open for all. The result is a creative outlet

for users and accessibility for other researchers.

4Paul Chin. The Value of User-Generated Content. Intranet Journal. July 3, 2006.

5Tim O'Reilly. What Is Web 2.0, O'Reilly Network. http://www.oreillynet .com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-

20.html. August 2006.



7


Figure 5 - (Left) Electro-Gymnasts, Mostra UP – Porto, Februray 2010; (Right) iDance System, Factor J – Vale de Cambra, March

2010.

Work Package 3: Motion Capture System

_Week 6-7 _Host Institution _Remainder Funding (850€)

Considering the formerly mentioned methodology, we will at this point assemble the motion capture

system. Dancers can choose how they interact with media by selecting between body-oriented and

environment-oriented, or a mixture of both input approaches. Currently, body-oriented systems

involve extracting data directly from the dancer such as brain waves, heart rate and skin6

conductivity. Environment-oriented methods position the dancers in a 3D movement sensitive grid

by using motion tracking devices such as video cameras, sonar, and laser beams. Because dance is

mostly involved with fluid body movement, the trend goes towards environment-oriented and non-

intrusive techniques that remove the burden of the dancers carrying wires and devices attached to

their bodies.

In the past I have used rather simple setups using infrared cameras, black light, wiis or simple black

and white cameras to capture motion. Below are pictures of two motion capture system that

designed which respond, with sound effects, to motion. These have systems were presented at

academic institutions and discothèques.

6Robert Wechsler http://www.palindrome.de/



8


Figure 6 - (Left) iDance 2.0 – single user feature tracking; (Right) iDance 1.0 – multi-user feature tracking w/ black lights.

List of materials:

Hardware

CCD Camera

Frame Grabber

Laptop

Projector

IR Light

Software

Eyecon

Eyesweb

Max/MSP

Ableton Live

Processing

Data Collection & Analysis:

Questionnaires and opinion surveys will rate music appreciation as well as dance appreciation. The

nature of data will be both qualitative and quantitative.

Qualitative data

• Videos of users interacting with motion capturing system (MOCAP);• Recorded music material created by users;

• Performance tests will evaluate the degree of sensitivity between the subjects interact ion and the

_.affected music. This will fine tune parameters in the music software.

Quantitative data

• Behaviour observation checklists will be used to evaluate dancers' movements and provide

_.information on what movement to capture;

video capture

interaction design; audio procesing

image processing



9


• Questionnaires and opinion surveys will rate music appreciation as well as dance appreciation;

• Likert scale will allow participants to self-rate how much control they had on the music.

After the motion capturing system has been implemented, the testing will proceed as follows:

Two random samples will be created from a population of club goers. Demographic questionnaires

will filter subjects to a sample representative of club goers. The sample would then be split in to two

conditions. Under these conditions participants will repeat various sets. Each set has a length of

10min and the music played will have one of three levels of interaction assigned randomly: Non-

interactive, interactive and very interactive.

Under condition A, participants will dance in various sets and rate the experience after every set. This

will be a blind study; they will not be told which sets are interactive.

Under condition B, different participants will dance in various sets and rate the experience after every

set. They will be told which sets are interactive.

The unit of observation will depend on the availability of participants. If there is no time to implement

the questionnaires, plan B would be to do this experiment in my home institution after the STMS.

Figure 7 - Visual representation of experiment sets

Furthermore, systematic usability tests through the mission and community feedback will help steer

the project during the implementation as well as assuring sustainability after its completion.



10


Work Package 4: Final Dissemination

_Week 8-… _Home Institution _No Funding

Although dissemination will begin in the first week of the mission, along with the scientific report, this

final step will formalize the results of the mission.

Dates & Budget:

This STSM will start on the 1st

of August and end on the 30th

of September, 2010. The budget is of

2500€. The cheapest housing possible in Oslo is a youth hostel, and a private room costs 56 € per

night. As of right now, airfare from Porto to Oslo prices 300€ and daily predicted expenditures

amount to 30€ per day. The 2500€ financial support barely covers the housing expenses for the 6

weeks; therefore I will have to bear the costs of the travel and daily expenses.

Possible Candidates for Publications

• SMC 2011

• NIME 2011

• ICMC 2011

• Computer Music Journal

• Journal of New Music Research

Exhibitions

• Mostra da UP (March 2011)

• Maus habitos

Conferences and Workshops

• FEUP

FCT

• Further research funding (December)

• PhD funding (March 2011)

Web

• wordpress

• Twitter

• Facebook

• FriendFeed

• Scribd

• Slideshare

• Vimeo & Youtube

• Flickr



11


Figure 8 - Mission Timeline

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

describe use

gesture taxonomy

assemble mocap

run experiment uestionnaire

dissemination dissemination

music com osition

esture ma in

data anal sis

beta testin

WP1 – initial

dissemination WP1 – theoretical & methodological framework WP2 – motion capture WP4 – final

dissemination

usability testing

al ha testin

cost sid - interactive music 2.0 - short term scientific mission work plan

Documents