Julius O. Smith III Askenfelt60, KTH, Sweden – 1 / 23

Recent CCRMA Research on Virtual Musical Instruments

Julius O. Smith IIICCRMA, Stanford University

Askenfelt60, KTH, Sweden

October 21, 2009

Overview

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 2 / 23

Recent Research on Virtual Musical Instruments at CCRMA

Overview

• CCRMA

• Outline

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 3 / 23

CCRMA building: The Knoll, Stanford University

Outline

Overview

• CCRMA

• Outline

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 4 / 23

• Virtual Acoustic Guitar — Nelson Lee(Computer Science PhD student)

• Haptic Virtual Instruments — Ed Berdahl(Electrical Engineering PhD student)

• Virtual Harpsichord — Jack Perng(Physics PhD student)

• Acoustic Space Modeling — Consulting Professor JonathanAbel, Music PhD student Nick Bryan, EE graduate student TravisSkare, and others

• Special Issue on Virtual Analog Audio Effects & MusicalInstruments, for the IEEE Transactions on Audio, Speech, &Language Processing (ASLP) — edited by Valimaki, Fontana,Zolzer, & Smith

Virtual Acoustic Guitar Models

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 5 / 23

Coupled Strings Analysis and Synthesis

Overview

Acoustic Guitar Models

• Coupled Strings

• String Model

• Sound Examples

• Sound Examples

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 6 / 23

Submitted paper based on recent CCRMA/CS thesis by Nelson Lee :

“Analysis and Synthesis of Coupled Vibrating StringsUsing a Hybrid Modal-Waveguide Synthesis Model”

by Nelson Lee, Julius Smith, and Vesa Valimaki.Accepted for publication in the IEEE special issue onVirtual Analog Audio Effects and Musical Instruments,May 2010 (est.)

Overview of Nelson Lee String Model

Overview

Acoustic Guitar Models

• Coupled Strings

• String Model

• Sound Examples

• Sound Examples

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 7 / 23

• String excitation (for commuted waveguide synthesis) is highpassfiltered to avoid exciting first N partials

• Lowest N partials are replaced by fourth-order resonators (whichcan independently beat and give two-stage decay)

• Similar to Balazs Bank formulation which adds second-orderresonators to existing partials of the filtered-delay-loop

• New analysis methods (in thesis) for estimating partialparameters, as well as other results

Sound Examples of Individual Effects

Overview

Acoustic Guitar Models

• Coupled Strings

• String Model

• Sound Examples

• Sound Examples

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 8 / 23

From Nelson Lee’s thesis defense:

• Original waveform: (WAV) (MP3)• Simple lossless, reflectively terminated digital waveguide (DWG):

(WAV) (MP3)• Add loop filter: (WAV) (MP3)• Add interpolation filter: (WAV) (MP3)• Add excitation (ICMC07): (WAV) (MP3)• Add body response: (WAV) (MP3)• Add hybrid modal/waveguide model: (WAV) (MP3)• Exaggerate pitch glide due to tension modulation: (WAV) (MP3)

Virtual Acoustic Guitar Sound Examples

Overview

Acoustic Guitar Models

• Coupled Strings

• String Model

• Sound Examples

• Sound Examples

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 9 / 23

More Nelson Lee examples:

• Original 1: (WAV) (MP3)• Synthesized 1: (WAV) (MP3)

• Original 2: (WAV) (MP3)• Synthesized 2: (WAV) (MP3)

• Original 3: (WAV) (MP3)• Synthesized 3: (WAV) (MP3)

• Original 4: (WAV) (MP3)• Synthesized 4: (WAV) (MP3)

• Original 5: (WAV) (MP3)• Synthesized 5: (WAV) (MP3)

• Original 6: (WAV) (MP3)• Synthesized 6: (WAV) (MP3)

• Synthesized Chord Demo: (WAV) (MP3)

Haptic Feedback Control forVirtual Instruments

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 10 / 23

Haptic Virtual Musical Instruments

Overview

Acoustic Guitar Models

Haptic Instruments

• Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 11 / 23

CCRMA/EE PhD student Ed Berdahl is working on

Haptic Feedback Control for Virtual Instruments

Dissertation expected by the end of the year

Goals:

• Assist and/or augment gestures• Assist with accurate playing• Recent projects:

◦ Haptically plucked virtual string◦ Active drumhead (one-handed rolls, etc.):

http://ccrma.stanford.edu/~eberdahl/Projects/-HapticDrum/

Virtual Harpsichord

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 12 / 23

Harpsichord Modeling

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

• Harpsichord

• Harpsichord Jack

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 13 / 23

CCRMA/Physics PhD student Jack Perng is working on

1. Building a harpsichord jack and monochord (mostly done)2. Measuring position, velocity, etc. (getting started)3. Studying models of Giordano et al., Valimaki et al., and others

(well under way)4. Looking for ways to improve the model(s)

Prof. Tom Rossing collaborating

Harpsichord Jack and Monochord

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

• Harpsichord

• Harpsichord Jack

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 14 / 23

Acoustically Transparent andConfigurable Microphone Array

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 15 / 23

Microphone Array

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

• Mic Array

• Mic Array Paper

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 16 / 23

• Adustable geometry (software calibrated)• Sixteen microphones (Countryman B6 Omni Lavalier):

◦ 2 mm diameter capsules◦ 1 mm diameter flexible mounting wire◦ Acoustically transparent over most of the audio band

Upcoming Paper

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

• Mic Array

• Mic Array Paper

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 17 / 23

“A Configurable Microphone Array with Acoustically TransparentOmnidirectional Elements”

Jonathan Abel, Nicholas Bryan, Travis Skare, Patty Huang, DariusMostowfi, Miriam Kolar, and Julius Smith

AES-2009, New York

Current Application:

Recording and modeling acoustic properties of undergroundgalleries at pre-Inca archeological site Chavın de Huantar in Peru

Special Issue of the IEEE ASLP

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 18 / 23

IEEE ASLP Special Issue

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

• ASLP Special Issue

• Issue Overview

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 19 / 23

We have been working on a special issue devoted to

Virtual Analog Audio Effects and Musical Instruments

for the

IEEE Transactions on Audio, Speech, and LanguageProcessing (ASLP)

• Editors:

◦ Vesa Valimaki◦ Federico Fontana◦ Udo Zolzer◦ Julius Smith

• Eleven papers accepted so far, with more to come• Likely publication date: May 2010

Accepted Papers on Virtual Musical Instruments

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

• ASLP Special Issue

• Issue Overview

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 20 / 23

• “Tubular Bells — A Physical and Algorithmic Model” byRabenstein, Koch, and Popp

• “A block-based physical modeling approach to the soundsynthesis of drums” by Marogna and Avanzini

• “A Virtual Model of Spring Reverberation” by Bilbao and Parker

• “Analysis and Synthesis of Coupled Vibrating Strings Using aHybrid Modal-Waveguide Synthesis Model” by Lee, Smith, andValimaki

Likely additional papers on

• virtual plucking methods for waveguide string models,• piano modeling via physically directed modal synthesis, and• another approach to spring reverb modeling

Summary

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

Julius O. Smith III Askenfelt60, KTH, Sweden – 21 / 23

Summary

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

• Summary

• Conclusions

Julius O. Smith III Askenfelt60, KTH, Sweden – 22 / 23

A quick look at recent acoustic-modeling research at CCRMA:

• Coupled Strings Analysis and Synthesis — Nelson Lee (CS) —Fourth-order modes for low partials, waveguide model for upperpartials; new analysis techniques

• Haptic Virtual Instruments — Ed Berdahl (EE) — Real controllers(with force feedback) for virtual instruments

• Virtual Harpsichord — Jack Perng (Physics) — Monochord+jackmeasurements toward improved harpsichord synthesis models

• Microphone Array — Jonathan Abel et al. — Acousticallytransparent, configurable, software-calibrated microphone arrayfor sampling the 3D sound field

• Special Issue on Virtual Analog Audio Effects and MusicalInstruments — Vesa Valimaki et al., eds.

In Conclusion

Overview

Acoustic Guitar Models

Haptic Instruments

Harpsichord Models

Microphone Array

ASLP Special Issue

Summary

• Summary

• Conclusions

Julius O. Smith III Askenfelt60, KTH, Sweden – 23 / 23

Happy Birthday Anders!