![Page 1: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/1.jpg)
Sound Synthesis and Processing
Xavier Serra
Music Technology Group
Universitat Pompeu Fabra, Barcelona
![Page 2: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/2.jpg)
Xavier Serra - 2008
Index
Tradition of sound synthesis
Musique Concrète, Granular synthesis, Sampling
Spectral sound processing
Mosaicing, Concatenative synthesis
Gesture-based synthesis
![Page 3: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/3.jpg)
Xavier Serra - 2008
Tradition of sound synthesis
![Page 4: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/4.jpg)
Xavier Serra - 2008
4
![Page 5: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/5.jpg)
Xavier Serra - 2008
![Page 6: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/6.jpg)
Xavier Serra - 2008
graphical interface for Tbone (Perry Cook)
Synthesis with physical models
![Page 7: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/7.jpg)
Xavier Serra - 2008
Digital implementation of an acoustic system
![Page 8: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/8.jpg)
Xavier Serra - 2008
Musique concrète
Pierre SchaefferPierre Schaeffer, “Etude aux chemins de fer” (1948)
A class of electronic art music produced from editing together fragments of natural and industrial sounds.
![Page 9: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/9.jpg)
Xavier Serra - 2008
Granular synthesis
“All sound is an integration of grains, of elementary
sonic particles, of sonic
quanta” Xenakis (1971).
Helmuth’s example
![Page 10: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/10.jpg)
Xavier Serra - 2008
Sampling
Fairlight (1980)Pioneered two innovations that transformed music making, namely sampling and sequencing.
![Page 11: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/11.jpg)
Xavier Serra - 2008
Spectral processing
Transform. Spectral
Synthesis Fourier
Analysis Original Sound
TransformedSound
Original Spectrum
TransformedSpectrum
Feature Extraction
Feature Addition
Original Feature
Transformed Feature
Spectral Analysis
![Page 12: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/12.jpg)
12
( ) ( )∑=
++=N
kkkk tfAAtx
10 2cos φπ
∑=
−
++=N
k
tfjktfjk kk eX
eX
X1
2*
20 22
ππ
wherewhere kjkk eAX φ=
( ){ }
+=
+=
∑
∑
=
=
+
N
k
tfjjk
N
k
tfjk
kk
kk
eeAA
eAA
1
20
1
20
Re
Re
πφ
φπ
Spectral representation
![Page 13: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/13.jpg)
13
( ) ( ) ( ) ( ) ( ){ },,,,,,,,,0, 2*22
1222
11
*12
1112
10 fXfXfXfXX −−
A spectrum is a sequence of pairs of numbers:A spectrum is a sequence of pairs of numbers:
( ) ( ) ( )2500cos83200cos1410 ππππ ++−+= tttxThe spectrum of:
![Page 14: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/14.jpg)
14
Spectral analysis:Spectral analysis:
![Page 15: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/15.jpg)
15
• Short-Time Fourier Transform (STFT)Short-Time Fourier Transform (STFT)
• Sinusoidal ModelSinusoidal Model
• Sinusoidal plus Residual ModelSinusoidal plus Residual Model
• Higher Level AttributesHigher Level Attributes
Spectral models
![Page 16: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/16.jpg)
16
soundsound
additiveadditivesynthesissynthesis
sinusoidalsinusoidalcomponentcomponent
**windowwindowgenerationgeneration
smoothingsmoothingwindowwindow
FFTFFTphasephasespectrumspectrum
magnitudemagnitudespectrumspectrum
spectralspectralapproximationapproximation
residualresidualspectral dataspectral data
FFTFFT
magnitudemagnitudespectrumspectrum
phasephasespectrumspectrum
peakpeakdetectiondetection
peakpeakdatadata
pitchpitchdetectiondetection
peakpeakdatadata
F0F0frequencyfrequency
amplitudeamplitudecorrectioncorrection
peakpeakcontinuationcontinuation
sine frequenciessine frequenciessine magnitudessine magnitudessine phasessine phases
**
windowwindowgenerationgeneration
smoothingsmoothingwindowwindow
--residualresidualcomponentcomponent
Spectral analysis
![Page 17: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/17.jpg)
17
sine datasine data
spectral dataspectral data
amplitude amplitude andand
frequencyfrequencydetrendingdetrending
attack,attack,steady-state,steady-state,and decayand decaydetectiondetection
inflexioninflexionpointspoints
spectral shapespectral shapeextractionextraction
spectralspectralshapeshape
amplitude normalizationamplitude normalizationandand
amplitude envelopeamplitude envelopeextractionextraction
amplitude andamplitude andamplitude amplitude envelopeenvelope
frequency normalizationfrequency normalizationandand
pitch envelopepitch envelopeextractionextraction
pitch andpitch andpitch envelopepitch envelope
loop pointsloop pointsselectionselection
residualresidualsine datasine data
residualresidualspectral dataspectral data
loop pointsloop points
vibratovibratoextractionextraction
vibratovibratodatadata
Feature analysis
![Page 18: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/18.jpg)
18
IFFTIFFT++complexcomplexspectrumspectrum
sinesinefrequenciesfrequencies
sinesinemagnitudesmagnitudes
sinesinephasesphases
spectral dataspectral data
spectral spectral sinesine
generationgeneration
magnitudemagnitudespectrumspectrum
phasephasespectrumspectrum
polar to polar to rectangularrectangularconversionconversion
spectralspectralresidualresidual
generationgeneration
magnitudemagnitudespectrumspectrum
phasephasespectrumspectrum
**
windowwindowgenerationgeneration
soundsound
synthesissynthesiswindowwindow
**
spectral dataspectral data(ex: reverberation)(ex: reverberation)
polar to polar to rectangularrectangularconversionconversion
Spectral synthesis
![Page 19: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/19.jpg)
Xavier Serra - 2008
Spectral transformations
Original Sound
Transformation
Amp
f fi-1 fi fi+1 fi+2
desired timbre envelope
![Page 20: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/20.jpg)
Xavier Serra - 2008
Sound collage
John Oswald
“A phonograph in the hands of a hip hop/scratch artist produces sounds which are unique and not reproduced - the record player becomes a musical instrument. A sampler is simultaneously a documenting device and a creative device, in effect reducing a distinction manifested by copyright.”
no official distributor/not for sale 1989
![Page 21: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/21.jpg)
Xavier Serra - 2008
Photo mosaicing
Example of photomosaic made out of hundreds of portraits of Americans who have died at war in Iraq during the last few years.
![Page 22: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/22.jpg)
Xavier Serra - 2008
Audio mosaicing
(Schwarz, 2007)
Tristan Jehan, 2005
![Page 23: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/23.jpg)
Xavier Serra - 2008
Concatenative synthesis (1 of 2)
( ) ( )
111 2
2
2 1 1
n n
n n
i ii i n nn n
f transp f transpt
i t pitch transp
ϕ π
π
φ −−−
�+ �= − �¿ ∆
+ + ∆ ¿× −
2 ·c
syncn n
tpitch transp
ϕπ
−∆∆ =
PHASE TIMBRE
stretchedspectrum
1-SSIntp
SPP Spectrum n-i
differentialenvelope
Spectral Shape
Interpolation Factor
(SSIntp)
1Right EpRLeft EpR
Stretch Stretch
Spectral ShapeInterpolation Factor
Frame n-k Frame n-1
Interpolation zonek frames
Frame n
f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping
Timbre Envelope
stretchedspectrum
1-SSIntp
SPP Spectrum n-i
differentialenvelope
Spectral Shape
Interpolation Factor
(SSIntp)
1Right EpRLeft EpR
Stretch Stretch
Spectral ShapeInterpolation Factor
Frame n-k Frame n-1
Interpolation zonek frames
Frame n
f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping
Timbre Envelope
![Page 24: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/24.jpg)
Xavier Serra - 2008
Concatenative synthesis (2 of 2)
Apossible sounds produced by the instrumentBsounds produced by the performer playing the instrument
recorded audio samples
Instrument sonic space
performance trajectory generator
performance DB
sound rendering
Performance trajectory
performer model
Performance Score
Sound
Synthesizer diagram
Vocaloid example
![Page 25: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/25.jpg)
Xavier Serra - 2008
Synthesis based on gestures
gestures generatedby the system
![Page 26: Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based](https://reader031.vdocument.in/reader031/viewer/2022022600/5b3d7fca7f8b9a213f8dddc2/html5/thumbnails/26.jpg)
Violin gesture modeling