![Page 1: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/1.jpg)
Nov 30, Fall 2006 IAT 410 1
Audio
SoundAudio synthesis
![Page 2: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/2.jpg)
Nov 30, Fall 2006 IAT 410 2
Audio Perception
Sound: Pressure waves in frequencies between 50Herz - 22,000Herz
Lower frequencies more felt by the whole body than heard
Sounds can be perceived as coming from a location– Not terribly accurate– Cone of confusion
![Page 3: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/3.jpg)
Nov 30, Fall 2006 IAT 410 3
Cone of confusion– Cone-shaped zones in front of and
behind head 3D Audio cues:
– Interaural Time Difference– Interaural Intensity Difference– Pinnae filtering– Body filtering
3D Audio Perception
![Page 4: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/4.jpg)
Nov 30, Fall 2006 IAT 410 4
3D Audio Perception
Goal for 3D sound is “Spatialization” The sense that the
– Sound originates outside your head– Sound has a direction
Interaural Time Difference– The more extremely left or right, the greater
the difference– Time difference < 5ms
![Page 5: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/5.jpg)
Nov 30, Fall 2006 IAT 410 5
3D Audio Perception
Interaural Intensity Difference– Head absorbs and reflects sound energy– The first ear to get sound gets loudest
sound– “Head Shadow”
Cone of confusion:– Time difference too small to detect– Intensity is similar in both ears
![Page 6: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/6.jpg)
Nov 30, Fall 2006 IAT 410 6
Pinnae Filtering
Outer ear (Pinna) shape filters sound based on its direction
Childhood learning trains brain to associate filtering effects with direction
Unique per person Record directional white noise
– Microphone in ear canal– Sounds from speakers located about head
![Page 7: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/7.jpg)
Nov 30, Fall 2006 IAT 410 7
Pinnae Filtering
A “Generic” Pinna can be simulated Record directional white noise
received by dummy head Body filtering
– Reflection and absorption– Included in Pinna model
![Page 8: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/8.jpg)
Nov 30, Fall 2006 IAT 410 8
Head-Related Transfer Function
HRTF is the general term Transformation of “real” sound to
spatialized sound Best delivered by earphones
![Page 9: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/9.jpg)
Nov 30, Fall 2006 IAT 410 9
Environmental Effects
Sound exists in an environment– Bounces off objects– Is absorbed by objects
Simple effects– Reverb: Simulate the environmental echo
• Echo is the attenuated signal• Gives a richer room-like feeling• Larger room has longer time delay
![Page 10: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/10.jpg)
Nov 30, Fall 2006 IAT 410 10
Audio signals
Nyquist limit:– Must sample signal at least twice as
frequently as highest reproducible frequency
– Audio: 44.1KHz (CD)– 22KHz– 11KHz (Analog AM Radio)– 8KHz (Telephone)
![Page 11: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/11.jpg)
Nov 30, Fall 2006 IAT 410 11
Audio - Digital Implications
44,100 Hz– 44,100 Samples/sec– 16-bit samples– Stereo– 172KBytes/sec
Specialized hardware - Sound card
![Page 12: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/12.jpg)
Nov 30, Fall 2006 IAT 410 12
Reproduction
Sampling– Record sounds by whatever means
Synthesis– Analog Synthesis– FM Synthesis– Wavetable Synthesis
![Page 13: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/13.jpg)
Nov 30, Fall 2006 IAT 410 13
Control
MIDI - Musical Instrument Digital Interface
Developed to control music synthesizers– Details of synthesis are controlled by
synthesizer MIDI data
– Sets synthesis parameters– Sets music sequence
![Page 14: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/14.jpg)
Nov 30, Fall 2006 IAT 410 14
Synthesis
Analog Synthesis– Simple sum of frequencies– Select from a palette of source
frequencies– Sum of frequencies is filtered
FM– One frequency is controlled by another
Wavetable– Digitize audio signals
![Page 15: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/15.jpg)
Nov 30, Fall 2006 IAT 410 15
Analog Synthesis
Fourier’s observation– Any signal can be created as the sum of
sine waves– Square wave: Infinite sum– f + 2f + 4f…
Synthesizer:– Collection of
oscillators
![Page 16: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/16.jpg)
Nov 30, Fall 2006 IAT 410 16
Frequency Domain
A perfectly periodic signal plotted in the frequency domain
(Time domain plot)
![Page 17: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/17.jpg)
Nov 30, Fall 2006 IAT 410 17
Spectrum
Spectrum represents the set of frequencies present in the signal
![Page 18: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/18.jpg)
Nov 30, Fall 2006 IAT 410 18
Filters
Eliminate part of the signal by removing certain frequencies
Analog filters don’t have these “square” response shapes
Band pass – Bandwidth
![Page 19: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/19.jpg)
Nov 30, Fall 2006 IAT 410 19
FM Synthesis Modulate the frequency of a wave
Carrier frequency is modulated by Modulator signal
![Page 20: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/20.jpg)
Nov 30, Fall 2006 IAT 410 20
FM Synthesis for Synthesizers
The greater the Modulator amplitude, the greater the Carrier frequency variation– Higher Carrier bandwidth
DX: Carrier and Modulator are “musically-tuned frequency”– Depends on the note you are playing – Controls the harmonic content of a note
![Page 21: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/21.jpg)
Nov 30, Fall 2006 IAT 410 21
Wavetable Synthesis
Collect a sample of the real sound Issues:
– Reduce memory load by looping sample– Shift pitch instead of sampling each
individual note– Apply interpolation techniques to make
pitch shifting work right
![Page 22: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/22.jpg)
Nov 30, Fall 2006 IAT 410 22
Audio Path
Raw Sound(Sample, FM, etc)
Tuned
MIDINote Sequence
Resampling
EnvelopeLoudness Control
Mixing/Combination
Reverb,Environment.Spatialization
![Page 23: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/23.jpg)
Nov 30, Fall 2006 IAT 410 23
Wavetable Synthesis Example
Leyanda (Guitar) Leyanda (CDShaw)
![Page 24: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/24.jpg)
Nov 30, Fall 2006 IAT 410 24
Interactive Sound
Goal– Want to enhance the interactive
experience– Give the user a sense of presence– Add to the emotional content of the
game– Make it more fun
![Page 25: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/25.jpg)
Nov 30, Fall 2006 IAT 410 25
Interactive Sound
Music Sound effects
– Noises– Commentary - Sports
Narrative Conversations
![Page 26: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/26.jpg)
Nov 30, Fall 2006 IAT 410 26
Interactive Problems
Regular music composition has – Beginning– Middle– End
Interactive user control makes this difficult– Some genres have this structure
![Page 27: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/27.jpg)
Nov 30, Fall 2006 IAT 410 27
Interactive Music
Game genres with order– Sports– Racing– Fighting
Semi-Ordered– Puzzle– Adventure
![Page 28: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/28.jpg)
Nov 30, Fall 2006 IAT 410 28
Music Genres
The Infinite Loop– Theme and variation plays forever– Pomp & Circumstance– Diablo
Problems:– 30 second piece repeated over 6 hours!– 720 repetitions!– Diablo example: 12 Repetitions/hour
![Page 29: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/29.jpg)
Nov 30, Fall 2006 IAT 410 29
Repetition Solutions
Make the Dominant theme hard to find– No catchy theme!– Create a variety of textures– Make only transitions stand out
Where repetition is small– Don’t repeat musical phrases
![Page 30: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/30.jpg)
Nov 30, Fall 2006 IAT 410 30
Music Strategies
Play Win or Lose music– Music must be long enough to be
meaningful– Music may be so long that the game
situation changes before completion– Very short music makes little sense
Interrupt current music– Sounds jarring
![Page 31: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/31.jpg)
Nov 30, Fall 2006 IAT 410 31
Modules
Modular chunks– Each segment of the game plays
independently of others– Some thematic relation– Disjointed
![Page 32: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/32.jpg)
Nov 30, Fall 2006 IAT 410 32
Music Strategies
Compose many themes in parallel Switch between themes Connect modular components
together
![Page 33: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/33.jpg)
Nov 30, Fall 2006 IAT 410 33
Analogy: Parallel Trains
N trains of music running in parallel Each train serves an emotional
purpose– Train A: Calm– Train B: Rising Excitement– Train C: Climactic moments– Train D: Falling Excitement
Generally, Train A would be most commonly played
![Page 34: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/34.jpg)
Nov 30, Fall 2006 IAT 410 34
Within Each Musical Train
Each “Car” contains a few bars of music Switch between trains when a “Car” is
complete– Don’t switch in the middle of a “Car”
Simple version:– Each musical phrase ends on last bar of
“Car” Complex:
– Notes at end are carried over to next
![Page 35: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/35.jpg)
Nov 30, Fall 2006 IAT 410 35
Parallel Trains
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
![Page 36: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/36.jpg)
Nov 30, Fall 2006 IAT 410 36
Parallel Trains: Shuffle Cars
Shuffle cars – Instead of playing cars in order
Problem: Random cars sound like random radio tuning
Must determine – Appropriate car pairings– Reasonable paths
![Page 37: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/37.jpg)
Nov 30, Fall 2006 IAT 410 37
Repetition In Trains
Use repeated phrases carefully Maybe use a statistical tool to
analyze paths– Bayesian nets
Endings:– Use transitions as an opportunity to
“End” – Use next Car to “Begin” new series
![Page 38: Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d455503460f94a21c3c/html5/thumbnails/38.jpg)
Nov 30, Fall 2006 IAT 410 38
Composing a Train
Create a piece with all layers– Piece can probably survive a layer or two
removed– Variation = piece with layer removed
Be careful with prominent instruments– Fallback: Use instruments with similar
acoustical properties• Piano, Organ, Woodwinds• No Trumpets, Drums or Screaming guitar!