DAAD Edgar Varèse Guest-Professorship 2007TU Berlin – Audio Kommunikation

Norbert Schnell

Berlin, May 24 & 25, 2007

Basics of Real-Time Interactive Audio Processing

Real-time Interaction with Recorded Sounds

INTE

RACT

ION

chapter 0

interaction concepts

“PCI”

?

?

???

perfor

mer,

user

?

compos

er,

artist

instru

ment,

interfa

cemach

ine

performer-computer interaction

multi-modal interactioncapture

input devices (mouse, keyboard, faders, ...)

sensorsvideoaudio

feedbackaudio (auditive)visualhaptic

+ movement and gesture analysis/recognition+ computer vision+ audio and music analysis/recognition

discourses & levels

“elements” of interaction

interaction metaphors

interaction paradigms

models & representations

techniques

applications

chapter 1

elements of interaction

sculptu

re

box

thingm

achin

einstrument

elements of interactionobject

elements of interactionobject - space -

roomstagewallyardtown

elements of interactionobject - space - image

recallimpressionillusionappearance

mirror

imagination

space

object

map

shadow

material

material? (surface?, texture?)

active surfaces and textures

wearables

interactive fabrics

etc...

chapter 2

metaphors and paradigms

metaphor vs. paradigmmetaphor• dict.: a figure of speech in which a word or phrase is applied to

an object or action to which it is not literally applicable

• analogy

• ideal

paradigm• dict.: a typical example or pattern of something; a model

• operational or even established

• way of doing

• standard

music related metaphorsplaying an instrument

playing with accompaniment

conducting an orchestra

one-man-band

ensemble playing (duo, trio, many)

mixing and DJing

listening to music

extra-musical metaphorscommunication

conducting

navigation

games and puzzles

sports and competition

combat

writing and desktop work

life (biological processes, social behaviour)

...

paradigms & stereotypesperformance and expression

the music player

automatic accompaniment

the continuator

simulation and control

active (enhanced) listening

real-time interactive content-based audio proc.

visualisation & sonification

chapter 3

models and representations

model vs. representation(don’t mind :-)

... vs. description

IwCS* models + devices

sound model+ sound rendering system

gesture/movement model+ interface, capture device

→ mapping between sound and gesture model(correspondence, coincidence, adequacy, ???)

+ overall interaction design

* Interaction with Captured Sounds

sound model*model of a temporal object• already there, ready made, completely defined, deferred

modelling = conditioning• frozen, solid → liquid, fluid, supple, elastic, articulated

representation• encoding

• description (characteristics, morphology, semantics, etc.)

• decomposition

→ interaction

* recorded sound (vs. captured sound)

gesture/movement modelmodel of a process• action, stream, real-time

modelling = analysis & recognition• shadow, projection of an action → intention, interaction

representation• measurement (interface parameters)• projection

• reconstitution (of original action and intention)

• decomposition

• semantics, symbolics, intention

→ sound and music

sound representationswaveformspectrogram & co (linear/logarithmic, continuous/quantised)

sonogram, cepstrum, MFCC, X-Micks, chromagram, ...

harmonics and partialsaudio features (instantaneous and dynamic)

modulation spectrumsegmentationsimilarity matrixscorepiano roll...

“gesture” interfaceskeyboard and mouse (with or without GUI)

(graphic) tabletfaders and knobsMIDI keyboardphysical sensors

accelerometer, giroscope, flex, pressure, distance, etc.

computer vision (EyesWeb & Co)object and motion descriptors

3D motion captureaudio analysis...see NIME & Co

(= representations)

chapter 4

examples

performance interfacesNIME IwCS performers• Michel Waisvisz – The Hands

• Atau Tanaka – Biomuse

• Serge Delaubier – MetaInstrument

• eRikm – 3-k.pad.system• Tomomi Adachi - Voice and Infrared Sensor Shirt

• Sven König – sCrAmBlEd?HaCkZ! (see below: corpus based)

more interfaces• ReacTable – Sergi Jorda et al.

... other NIME 06 performances/interfaces

more artistslive FX• Joel Ryan (esp. with Evan Parker)

• Olvier Sens (Usine)

composition and IwCS• Philippe Manoury – En Echo, Partita 1

• Florence Baschet – Bogenlied

• Thierry DeMey – Light Music

• Pierre Jodlowski – Le Grand Dehors

installationsListen – Gerhard Eckel et al.16:9 – Daniel Teige & Martin RumoriAudio Shaker – Mark Hauenstein

(... more ??? !!!)

(examples)

gesture analysis/recognitionsoftware• EyesWeb – A.Camuri et al. (Univ. Genova GIST)

• cv.jit Max/MSP/Jitter computer vision tools – IAMAS

• MnM Max/MSP/FTM toolbox – IRCAM IMTR

• gesture follower – F. Bevilacqua et al. (IRCAM IMTR)

dance projects• one video example: Olivia Granville – comment taire

• many many projects !!! ???

appletsENIGMES: Feuillage – Eric Broitmann et al.IXI Software

generative systemsThe Continuator - F. PachetOMax – G. Assayag, G. Bloch et al.Khorwa – M. Malt

(... too many missing)

MIR / corpus basedconcatenative/corpus based synthesis systems• cataRT – Diemo Schwarz (IRCAM IMTR)

• SoundSpotter – Michael Casey

• bbcut – Nick Collins

• song segmentation/résumé sonore – G.Peeters et al. (IRCAM)

performances and installations• sCrAmBlEd?HaCkZ! – Sven König

• messa di voce – Golan Levin and Zach Lieberman

Hausaufgabe zum 25.Mai• 1-5 Klänge auswählen (mitbringen)

• Beschreibung ihrer Morphologie

• Vorstellung einer möglichen Interaktion

Finden/Auswählen bis zum 14.Juni• IwCS Paradigma

• Schnittstelle

• Klangverarbeitungsumgebung Wilms Studio Di 5. & Fr 8.Juni ab 14-16 Uhr

• Arbeitsgruppe

Varèse SS 2007