cross-modal mappings and audiovisual feedback
TRANSCRIPT
![Page 1: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/1.jpg)
Cross-modal mappings and audiovisual feedback
October 30, 2011 Cross-modal mappings
1
Nick Collins
![Page 2: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/2.jpg)
Summary
Experiments in audiovisual combination and data flow
Challenges of audiovisual mapping
October 30, 2011 Cross-modal mappings
2
![Page 3: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/3.jpg)
Theoretical backdrop
Film music vs music film
1-1 mappings, mickey-mousing, synchresis
Meanings in parallel, audiovisual objects
October 30, 2011 Cross-modal mappings
3
![Page 4: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/4.jpg)
Problems with mapping
Maps between domains are usually not homomorphisms; not structure-preserving
Sonification/visualisation from data, audio driving visuals or vice versa; all examples of potentially incompatible spaces
October 30, 2011 Cross-modal mappings
4
![Page 5: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/5.jpg)
Psychological backdrop
Synaesthesia is personal, not universal
Audiovisual integration, attentional resources
Coping with multiple objects in counterpoint
October 30, 2011 Cross-modal mappings
5
![Page 6: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/6.jpg)
Some successful close audiovisual integrations?
Coldcut/Hex Timber (1997)
Autechre/Alex Rutterford Gantz Graf (2002)
chdh’s audiovisual objects
October 30, 2011 Cross-modal mappings
6
![Page 7: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/7.jpg)
Now…
Some unsuccessful relations?
Case studies from experiments
October 30, 2011 Cross-modal mappings
7
![Page 8: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/8.jpg)
Case 1: PhotoNoise iPad app
Image -> sound via pixel data
Multi-touch -> RGB + intensity -> nonlinear oscillators
Quirky, noisy, more of an esoteric instrument than deep in itself
October 30, 2011 Cross-modal mappings
8
![Page 9: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/9.jpg)
Audiovisual feedback
Round a processing chain incorporating both audio and video material and its analysis
E.g., audio input -> audio analysis -> drive generative graphics -> video analysis -> drive audio synthesis -> audio input …
Processing delays
October 30, 2011 Cross-modal mappings
9
A V
![Page 10: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/10.jpg)
Case 2: klipp av (2002-2009)
October 30, 2011 Cross-modal mappings
10
![Page 11: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/11.jpg)
Audiovisual performance score
October 30, 2011 Cross-modal mappings
11
![Page 12: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/12.jpg)
Case 3: iterative cross-convolution
October 30, 2011 Cross-modal mappings
12
![Page 13: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/13.jpg)
With onset detection
October 30, 2011 Cross-modal mappings
13
![Page 14: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/14.jpg)
Case 4: audiovisual concatenative synthesis
Concatenative synthesis = feature-matched substitution of an input from a database
Usually audio or visual alone.
Has been explored where video is brought along for free (Grierson and Casey 2007, Scrambled Hackz et al.)
Can do both at once = audio and video feature data muddle
October 30, 2011 Cross-modal mappings
14
![Page 15: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/15.jpg)
October 30, 2011 Cross-modal mappings
15
![Page 16: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/16.jpg)
Case 5: auto aligned video
Take an existing music work
Extract audio features -> drive frame by frame rendering of generative graphics
End up with close correlation
Tool chain: SCMIR->Processing->ffmpeg
October 30, 2011 Cross-modal mappings
16
![Page 17: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/17.jpg)
What is coming out of this?
1-1 mapping might be simple, but it is perceptually strong!
Need for audiovisual objects as coherent entities
Details of many mappings lost (can be other reasons to keep on regardless, e.g. improvisation in complexity)
October 30, 2011 Cross-modal mappings
17
![Page 18: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/18.jpg)
Conclusions
Considered some of the challenges of correlating audio and visual
Discussed some projects of varying effectiveness
Details often lost when mapping between modalities
October 30, 2011 Cross-modal mappings
18
![Page 19: Cross-modal mappings and audiovisual feedback](https://reader031.vdocument.in/reader031/viewer/2022012102/6169fae611a7b741a34d7e66/html5/thumbnails/19.jpg)
Think you for lastening
October 30, 2011 Cross-modal mappings
19
http://www.sussex.ac.uk/Users/nc81/researchav.html
Alexander, A., and Collins, N. (2007) Live Audiovisuals. Book chapter in N.Collins and J. d'Escrivan (Eds.) (2007) The Cambridge Companion to Electronic Music
(2007) "Audiovisual Concatenative Synthesis". Proceedings of ICMC2007, International Computer Music Conference, Copenhagen.
(2006) Nick Collins and Fredrik Olofsson. "klipp av: Live Algorithmic Splicing and Audiovisual Event Capture", Computer Music Journal 30(2): pp 8-18
http://www.sussex.ac.uk/Users/nc81/iphone.html#PhotoNoise