singing voice synthesis in the context of music technology...
TRANSCRIPT
Singing voice synthesis in the context
of music technology research
Xavier Serra Music Technology Group
Universitat Pompeu Fabra, Barcelona http://mtg.upf.edu
Musical Sound Synthesis
Theremin Lev Termen, 1917
RCA theremin, 1929 Clara Rockmore “Vocalise” by Rachmaninoff
Moog Synthesizer
Walter Carlos “Switched on Bach”, 1968
Moog synthesizer from the 60s
Robert Moog, 1964
FM synthesis John Chowning, 1972
DX7 by Yamaha, 1983
Waveguide synthesis Julius Smith, 1985
VL1 by Yamaha, 1994
Sampling
Fairlight, 1980
New musical interfaces
Reactable, 2005
Singing Voice Synthesis
Physical model from 50s
Max Mathews, 1965
Kelly, Lochbaum and Mathews, 1961 “Daisy”
Linear Predictive Coding
Charles Dodge “The days ahead”, 1975
FOF synthesis Xavier Rodet, 1977
Aria “Queen of the Night” from the Magic Flute by Mozart, 1979
SinSy Keiichi Tokuda et al., 2009
Vocaloid and Hatsune Miku
Sample editing
Harmonic + Stochastic analysis
sines
residual
Timbre (Resonances, Excitation)
Stationary Note attack Note release Note to note Articulation Vibrato
Excitation + Resonance
model estimation
Database Creation
residual
1/F
Flat residual excitation
Flat harmonic excitation
EpR spectral Amplitude
Filtering
Filtering
EpR spectral Phase
sines
residual
HpS synthesis
Synthesis
Demo from 2001
Yamaha, 2004
Yamaha, 2006
Some challenges in singing voice research
Carnatic music
T. M. Krishna
Beijing Opera
Zhao Xiu Jun
Intonation
Melodic Motives
Discovery Induction Extraction
Matching Retrieval
+ Characterization
Transform
N dimensions
Lyrics and music
Chinese is a tonal language
ma ma1 ma2 ma3 ma4 妈 麻 ⻢马 骂 吗
“mum” “hemp” “horse” “to scold” “?”
Hypothesis: performers try to reflect tone contours in the melody in order to make lyrics understandable
Conclusion
• Musical instruments <--> Voice • Singing voice synthesis <--> Speech synthesis • Timbre models --> Musical models
• Lessons learned from Hatsune Miku
Thanks