audio source separation: de-mixing for production€¦ · project timeline •2009 started research...
TRANSCRIPT
![Page 1: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/1.jpg)
Audio Source Separation: "De-mixing" for Production
De-mixing ‘The Beatles at the Hollywood Bowl’using Sound Source Separation
James Clarke
Abbey Road Studios
![Page 2: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/2.jpg)
Overview
• Historical Background
• Sound Source Separation
• Considerations for De-Mixing
• Source Separation Techniques
• Audio Examples
![Page 3: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/3.jpg)
Historical Background
• The Beatles at the Hollywood Bowl
– Beatlemania had just landed in the US
– Previous attempt to record live performances blocked by musicians union in the US
– 1964 tour, Capitol attempted to record concert
– 15,000 screaming girls created a wall of sound
![Page 4: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/4.jpg)
Historical Background
![Page 5: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/5.jpg)
Historical Background
• Three Concerts 1964 and 1965
– Recorded on Ampex 3-track machines
– Vocals on 1 track
– Instruments on remaining 2 tracks
– Crowd mixed purposely into instrument tracks
• Problems– Engineered by the Hollywood Bowl and not Capitol
– No monitoring on the 3-track
– Capitol had no influence on actual recording
– Excess crowd levels not discovered until playback in studio
![Page 6: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/6.jpg)
Spectral Sources
![Page 7: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/7.jpg)
• Solution – Create new stereo re-mixes using Sound Source Separation technology
• Sound Source Separation - Given a recording of a mixture of sound sources, attempt to recover the original sound sources in isolation.
• Problem: There are usually many more sources than signals, it is a difficult task, but still possible.
• Why use sound source separation?– Filtering by EQ affects all recorded signals as discovered by Martin and
Emerick in 1977. We can now attempt to isolate crowd directly
![Page 8: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/8.jpg)
Project Timeline
• 2009 started research into source separation• 2011 Capitol discover original 3-track recordings and send to Abbey
Road• Approached by Giles Martin to see if I could do anything to improve
the audio• Source Separation succeeds by assuming crowd is signal and not
noise• Early 2012, re-mixes approved by ”the board”• 2016, ‘Live at the Hollywood Bowl’ Album released, coinciding with
the release of the film The Beatles: Eight Days a Week
![Page 9: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/9.jpg)
Sound Source Separation
• 3 Main types of Sources
– Vocals
– Pitched Instruments
– Percussion/Drums
– No one algorithm is optimal for all these sources - use different techniques
– Prior knowledge of the sources should be used where possible (some multitracks are available)
![Page 10: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/10.jpg)
Results
![Page 11: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/11.jpg)
Considerations for re-mixing
• Use filters where the separated sources sum back to give the original signal – No information lost
• Any artefacts due to separation will then be masked – our brains re-integrate the parts
• Separation does not have to be perfect, just good enough to impart directionality.
![Page 12: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/12.jpg)
Considerations for re-mixing
• However, if panned too far re-integration can break down.
• No phase problems between channels due to the type of filtersused
• Source drift due to incorrect separation can be a problem –ameliorate by choice of pan position – Sources placed inoriginal positions
![Page 13: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/13.jpg)
Source Separation Techniques
• Drums/Percussion– Highly localised in time, broadband spectra and repetitive– Many approaches – Decomposition into parts (NMF), dictionary based methods,
heuristics-based (drums –vertical lines, pitched instruments horizontal)
• Vocals/Lead Instruments– Decomposition into parts (NMF), melody tracking and filtering, heuristics-based
(backing track is more repetitive than the main melody)
• Pitched Instruments– Again NMF, Additive Synthesis based decompositions, Source-Filter models
• User Assisted Algorithms– Use prior information to aid the separation process (Knowledge of melody,
existence of backing tracks, chord progressions and so on)
![Page 14: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/14.jpg)
Source Separation Techniques
• Different methods work better on different types of material, it varies from song to song
• Pick the techniques based on the song
• Order of separation can make a big difference in some cases
• Whatever sounds best
![Page 15: Audio Source Separation: De-mixing for Production€¦ · Project Timeline •2009 started research into source separation •2011 Capitol discover original 3-track recordings and](https://reader033.vdocument.in/reader033/viewer/2022060609/606001a967637e6bb87beb1e/html5/thumbnails/15.jpg)
Audio Examples
• SLY – Original Stereo
• SLY – Original Track 1
• SLY – Extracted Drums
• SLY – Extracted Bass
• SLY – Extracted Crowd