s.laag(2016)-forgame-audiowith3Dbody-scannedperformer,composerandinstrument

RicardoCliment1,Manusamo&Bzika2

1NOVARSResearchCentre,TheUniversityofManchester,UKricardo.climent@manchester.ac.uk

2Manusamo&Bzika,interdisciplinarycollectivemanusamoandbzika@gmail.com

Abstract.Thispaperunfoldskeycompositionalaspectsof's.laag',aninteractivemusicalworkcomposedbyRicardoClimentforDutchBassClarinettistMarijVanGorkom,aspartofthedutch-uk.networkproject.Itincludes3DmodelsbybothClimentandManusamo&Bzikabasedon3DBodyscannersprovidedbySimeonGillattheSchoolofMaterialsDesign&Fashion,UniversityofManchester.Thepieceemployssonicpath-findingtechniquesusinggame-enginetoolstoexploretheconceptof"modularmetaphor".ItnavigatesacrosstheintersectionsbetweentheReal,theVirtualandthemodularaugmentationofamusicalinstrument.TheDutchword'laag'meanslowbutalsolayer,stratumandthickness,whichresonatemorewiththecompositionalthinkingbehindthispiece.Slaag(passinDutchlanguage)evokesthesonicfluxusbetweentheacousticinstrumentandtheelectronicmedium.Keywords:BassClarinet,game-audio,3Dbody-scan,Expo58,gamification,ExtendedReality,Modularmetaphor,VirtualReality.

CreativeBackgroundThedutch-uk.network58/58projectisanewnetworkstartedin2015betweentheInstituteofSonologyattheRoyalConservatoryofTheHagueandtheMusicCompositiondepartmentatTheUniversityofManchester,UK.ItaimedtobuildbridgesbetweenUKandmainlandEurope,viaaddressingcreativequestionsintheformofconcertevents,newmusicalcompositions,workshopsanddebate.Thenetworkproposedthe"Philipspavilion"asaconceptandmodel,whichmayinformtheproject,just58yearsafterthisWorld'sFairpavilionwasdesignedforExpo'58inBrussels.

Figure1.s.laag'sExpo58buildingsinUnrealEngine4,informedbymodelsfromthePhilipspavilionandtheAtomium

Previousworkongame-audiocomposition:s.laagbuildsuponseveralgame-enginecollaborativeworksbythecomposeronthispaper.Itfeaturesmusicalinstrumentsasvirtualcharactersinagame-enginedrivenstageperformance.RecentexamplesincludetheTimbila(amarimbaofMozabique)forthepiece"Xi"withMiquelBernat,aVCS3analoguesynthesizerfor"Putney"withMarkPilkingtonperformingtheVCS3andnowfeaturingaBassclarinetfor"s.laag".The3DmodellingfortheseworkswassketchedbyClimentandfinallyrenderedbyManusamo&Bzika(AlenaMesarošováandManuelFerrer).Suchgame-audiocompositionsoftenemploytheconceptof"modularmetaphor"(Climent,Pilkington,Mesarošová,2016),totranscendthephysicalityoftherealinstrumentduringgame-play.Suchmetaphoroccurswhenthewholebecomesmorethanthesumofitspartsthroughtheauralassemblagebutasin

s.laag,itcanalsobecomesomethingelse.Suchimmersiveexperiencesareconceivedasgame-audiojourneyshighlightingdistinctiveaspectsoftheinstrumentbeingexplored.Forexample,theyinvestigateuniquetimbralorgesturalproperties,whichgraduallyunfoldwhiletheinstrumentbecomesre-assembledasthepieceprogresses.

CompositionalMethodologyRecordingsession:auralscores

ThecompositionapproachtogatheringBassclarinetrawmaterialfors.laagwasbuiltupontheconceptof'AuralScore'(RickNance,2006),whichwasadaptedforrecordingpurposes.Theultimateaimofthissystemistoextract'DNA'frominstrumentalperformers'uniqueauralmemory,whichtheyarenotevenconsciousabout.Inthepast,thecomposerinthispaperinvestigatedthistechniquewithviolinistDarraghMorganandpercussionistMiquelBernat,providinghimwithawealthypooloftheirDNA'sauralmaterial.Fors.laag,a2-hour'crunchfile'withsilentgapswasassembled,wheretheperformerhadtoinsertrestrictedmusicalresponsestomaterialsonthefly;e.g.bymimickingorplayingagainstwithinarangeortechnique.Thecrunchfileisnevertobeusedinthefinalcomposition.Instead,theplayer'sresponsebecomesthebasisforcompositionalinventions.Ins.laag,sixtypesofmaterialswerecrunched,rangingfrommanipulatedexcerptsofJimiHendrix'sbestsolos,toPeterFrampton'stalkingboxguitar,a1957recordingfromEdgarVarèseinaworkshopofJazzmusicians,harmonicaphrasingfromSonnyBoyWilliamson,aperformancebyJapaneseKotoPlayerFuyukiEnokido,allintercalatedwithsomeextendedtechniquesfromthecomposer'sownaltosaxophone,transformedwithaBigMuffdistortionpedal.

Scanningtypologies:mosaic-sizemusicalcells

Afterlisteningtotherecordingsession,only20%ofmaterialswereselectedandedited.Theyservedasrawsourcesforfurthertransformations.Atthispoint,severalcompositionaltoolswereappliedwiththeaimtogeneratealargecollectionofmosaic-sizemusicalcellswithspecificsonictaxonomies.Forinstance,101mosaic-sizesoundfileswerecreatedusingauralscanningandmanualediting,69withacustom-madewaveformpointerto'chop'materials,190fileswereobtainedfromaclattersystem(Bachelor,2003),93usingadopplertool,asoundfreezer,60soundexcerptsusingaCsoundsndwarper,10mid-sizephrasesfromaMaxMSPgranulator,17producebya'gesturalfaker'(aconstructorofgesturesfrommicro-materials)and20viaaNebulae(Qu-BitElextronix,2015)voltage-controlledgranulatorsynth.Totalofhours:76.5h.

structuralartefacts:3dscenes

ForarrangingmaterialsatstructurallevelIalreadyhadthescenesfromthegame-audioandengineversioninmind.Theabstractvisualisationofascene-orientedjourneyina3Denvironmentprovidedmewithfourauralpoolstomakecompositionaldecisions.[0.00-2:27InteractiveCounterpoint//2:27-4:25SpectralMulti-phonics//4:25-6:01 ValveVariations//6:01-8:43Inner-pulses]

Scoringthemusicandbuildingtheelectroniccounterpoint:studiocompositionthinking

Thecompositionalmethodologywasarguablyclosertoworkingwithstudiopracticesthantocomposingapieceforacousticinstrumentalongsideelectronics.Asaresult,soundmaterialsweredealtinabstractandlayeredintwoparts;anupperstratumconsistingmostlyofnon-transformedmaterials,(althoughheavilyedited)andahighly-articulatedcounterpointofelectronicresponsestothem,movingoutsidethephysicallimitsandidiomoftheacousticinstrument.

Figure2.s.laag'sreferencescore:Splittingthecounterpointbetweeninstrument(top)anddynamicmedia(bottom)

ComposingWithGame-EnginesThemodularmetaphor:gamificationoftheperformanceenvironment

Imagineweweretodismantleeverypartofamusicalinstrumentwiththepurposeofreassemblingtheminalternativewayswhilekeepingsomeoftheirfunctionsandbehavioursintact.Envisagealsowewereabletoextractthesonicnatureandextenditandmakeeverypartsonicallycapableofachievingmusicaloutcomes.Thisiswhatconstituteds.laag's'modularmetaphor'.Thebassclarinetmodulesbecameawindchime-likemobile(asinEarleBrown'smusic),clarinetkeysmarchesasinPinkFloyd'sTheWall'shammerscene,orbecamemultiplecharactersinformedbySillySymphony'sexperimentalcartoonsinthe1920sand30s.Howcouldthenew'game-instrument'soundlike?Wouldthesumofitspartsbecomesomethingaurallyoutrageous?,woulditssoundresonatewiththeinstrument'sDNA?Canthisformofre-embodimentprovidenewmodesofexpressioninaperformativegame-engine'sdomain?Toexploreso,s.laagheavilyreliesontheuseofphysics-graphics-game-audioenginesoftenfoundinludiccontextsandvideoconsoles,itrecontextualisessuchtechnologiesinaperformativeenvironment.Thevirtualgame-audioscenesbecomeapseudo-scorewherepre-composedsonicmaterialsareintroducedandneedtofindmusicaldirectionsduringgame-play(whicharepreconceivedbythecomposerasalternativeroutesandenginematinees).Thecomputermusicianplaystheroleofthecomposerandperformer(seenin3D)andunfoldstheelectronicmaterialsbynavigatingthescenesandinteractingwiththedifferentpartsoftheinstrumentastheyappear(viacolliders,DephofFieldraysandvirtualsonicscanners).

Figure3.Performativestageonleft("PutneyPonoskty",2016),ands.laag'sbassclarinethammermarch-likescene(onright)

Immersiveperformers:3dbody-scannedcharacters

Performer,Instrumentandcomposerwere3Dbody-scannedattheSchoolofMaterialsDesign&Fashion,UniversityofManchesterandthenimportedintothegame-engine(recombinedviaMakehuman,BlenderandUE4).

Figure4.HighPoly3DscanningsessionattheSchoolofMaterialsinvolvingperformer(left)andcomposer(right)

Bassclarinet3dparts:mobilewind-chimeconstructionandriggedperformer

Instrumentpartsandhumanswere3DmodelledinBlenderandexportedtoUnrealEngine4asthefinalperformancetool.

Figure5.Clarinetpartsmountedasmusicalmobilesandaperformer'srigandclockdrivenbyBhvfilesandFFTanalysis

Frequencyanalysis:theenvironmentisthemaincharacter

ThepieceheavilyreliesonrealtimeFFTanalysisoffrequenciesanddynamicstodriveandexcitetheenvironmentandcharactersonstage(usingthevisualisationplugininUnrealEngine4).Forinstance,the3D-mobiles,whichareconstructedbycollectinginstrumentparts,areFFTdriven,whilethecharacter'srigisdrivenbymotioncaptureBHVfiles.

Conclusions.laagisamusicalcompositionaimingtopushtheboundariesofexistingworkusinggame-audioand3Denvironmentatthecoreofthecompositionalthinking.Theprojectisacollaborativeworkbetweenperformer,composer,3Dscannersandmodellers.Thepiecetargetsnon-specialisedaudiencesbyfacilitatingthecomputermusicallanguageandgame-audioaestheticsviatheuseofgamificationprocesses.Thisistosaythatgame-enginetoolsandtheconceptofgame-playbecomepartofanimmersivejourney,whereaudiencesperceivecomplexmusicalideasandstructuresviadifferentchannelsofperception.Insucheffort,thecomposeraimstopreservethemusicalidentityoftheoriginalwork,whichin

thiscasewasconceivedbeforeitsgame-audioimplementation.Finally,s.laagaimstoexplorecreativeoutcomesinagame-audioformat,usingthe"modularmetaphor"asthemeansforassemblingauralcomplexitiesinthespiritofthedutch-uk.networkproject.

AdditionalinformationLinktoMaterials:http://game-audio.org/

Dutch-UK.networkhttp://dutch-uk.network

MotioncaptureBVHfiles:www.cgspeed.com(forperformancemockbeforecollectingowndata)

Acknowledgements.Dutch-UK.networkissupportedby:ResearchNetworkFund.SchoolofArts,HistoriesandCultures,TheUniversityofManchester,UK.SpecialthankstoKeesTazelaar(InstituteofSonologydirector),forprovidinganinsightabouttherecordingarchiveofthePhilipsPavillionExpo58.http://dutch-uk.network3DBodyscannerswereprovidedbyDrSimeonGillattheSchoolofMaterialsDesign&Fashion,UniversityofManchester.BassClarinetistMarijVanGorkomforherrecordingsessionandworkshopswhichtookplaceinStudio1,NOVARSResearchCentre.http://www.novars.manchester.ac.uk

ReferencesArnheim,Rudolf.1974"ArtandVisualPerception:APsychologyoftheCreativeEye"Berkeley:UniversityofCaliforniaPress.

Chion,Michel.Audio-Vision:SoundonScreen.Trans.ClaudiaGorbman.NewYorkNY:ColumbiaUniversityPress,1994.

Climent,RicardoandPilkington,Mark.Putney(2015)Vimeovideo“PutneyLivePerformance@xCoAxConference2015.

Climent,Ricardo(LiveGame-AudioEngines)andPilkington,Mark(1969VCS3).2015.Postedby“WunderbarLabBerlin”.http://vimeo.com/139128350[Lastaccessed21May2016].

Climent,Ricardo.Pilkington,Mark.Mesárošová,Alena."ExploringReal,VirtualandAugmentedWorldsThrough'Putney',anExtendedReality".http://econtact.ca/17_4/climent-etal_putney.html-Lastvisited21May2016

Collins,Karen.GameSound:Anintroductiontothehistory,theory,andpracticeofvideogamemusicandsounddesign.CambridgeMA:MITPress,2008.

Mesarosova,AlenaandManuelFerrerHernández.“Artbehindthemind:Exploringnewartformsbyimplementationoftheelectroencephalography.”Proceedingsofthe2015InternationalConferenceonCyberworlds(Visby,Gotland,Sweden:UppsalaUniversity,7-9October2015).

Nance,R.,“Compositionalexplorationsinplasticsound”,PhDthesis,DeMontfortUniversity,Leicester.2006JohnWiley&Sons,2008."GameDesign:Principles,practice,andtechniques—theultimateguidefortheaspiringgamedesigner.NewYorkCity:.Caillois,Roger.Man,PlayandGames.UniversityofIllinoisPress,1961

Pecino,IgnacioandRicardoCliment.“SonicMaps:Connectingtheritualoftheconcerthallwithalocativeaudiourbanexperience.”ICMC2013:“Idea”.Proceedingsofthe39thICMC(Perth,Australia)http://icmc2013.com.au

Sparnaay,Harry.2011."TheBassClarinet:APersonalHistory":PeriferiaSheetMusic.ISBN:978-84-938845