music-driven motion editing marc cardle rainbow group computer laboratory university of cambridge...

Music-Driven Motion EditingMusic-Driven Motion EditingMusic-Driven Motion EditingMusic-Driven Motion Editing

Marc CardleMarc Cardle

Rainbow GroupRainbow Group

Computer LaboratoryComputer Laboratory

University of CambridgeUniversity of Cambridge

[email protected]@cl.cam.ac.uk

Marc CardleMarc Cardle

Rainbow GroupRainbow Group

Computer LaboratoryComputer Laboratory

University of CambridgeUniversity of Cambridge

[email protected]@cl.cam.ac.uk

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Overview of TalkOverview of Talk

• Background: Example AnimationBackground: Example Animation

• Research goals and framework overviewResearch goals and framework overview

• Musical FeaturesMusical Features

• Motion EditingMotion Editing

• ExamplesExamples

• Background: Example AnimationBackground: Example Animation

• Research goals and framework overviewResearch goals and framework overview

• Musical FeaturesMusical Features

• Motion EditingMotion Editing

• ExamplesExamples


What is driving what?What is driving what?

Film SoundtracksFilm Soundtracks

Music VideosMusic Videos


Animusic ApproachAnimusic Approach

(Lytle ’94)


• A framework for motion editing using A framework for motion editing using music analysismusic analysis

• Simplify and automate the generation of Simplify and automate the generation of synchronised musical animationssynchronised musical animations

• A framework for motion editing using A framework for motion editing using music analysismusic analysis

• Simplify and automate the generation of Simplify and automate the generation of synchronised musical animationssynchronised musical animations

Research GoalResearch GoalResearch GoalResearch Goal


MIDIMIDIMIDIMIDI

• Parametric representation Parametric representation

• Pitch, velocityPitch, velocity

• Start-time and End-timeStart-time and End-time

• Parametric representation Parametric representation

• Pitch, velocityPitch, velocity

• Start-time and End-timeStart-time and End-time

SynthesizerSampler

Audio Signal

(Lytle ’94)


Audio and MIDIAudio and MIDI

• Analysis is based on both MIDI and AudioAnalysis is based on both MIDI and Audio

• Complementary perspectives on the Complementary perspectives on the analysed musicanalysed music

• Analysis is based on both MIDI and AudioAnalysis is based on both MIDI and Audio

• Complementary perspectives on the Complementary perspectives on the analysed musicanalysed music

++AudioMIDI


System OverviewSystem Overview


Musical FeaturesMusical Features


MIDI-Based AnalysisMIDI-Based Analysis



• Low-level MIDI note information Low-level MIDI note information

• MIDI Channel and Instrument informationMIDI Channel and Instrument information

• Chord RecognizerChord Recognizer

• Virtual PitchVirtual Pitch

• Beat TrackingBeat Tracking

• MeterMeter

• Low-level MIDI note information Low-level MIDI note information

• MIDI Channel and Instrument informationMIDI Channel and Instrument information

• Chord RecognizerChord Recognizer

• Virtual PitchVirtual Pitch

• Beat TrackingBeat Tracking

• MeterMeter



• SegmentationSegmentation

• Pattern MatchingPattern Matching

• SegmentationSegmentation

• Pattern MatchingPattern Matching



• RegisterRegister

• LoudnessLoudness

• DensityDensity

• DurationDuration

• Attack SpeedAttack Speed

• Focus and DelayFocus and Delay

• RegisterRegister

• LoudnessLoudness

• DensityDensity

• DurationDuration

• Attack SpeedAttack Speed

• Focus and DelayFocus and Delay Time

Max Pitch

Min Pitch

Pit

ch


Audio-Based AnalysisAudio-Based Analysis


Audio-Based AnalysisAudio-Based Analysis

• Zero-Crossing RateZero-Crossing Rate

• Spectrum Amplitude VariationsSpectrum Amplitude Variations

• Spectral CentroidSpectral Centroid

• Zero-Crossing RateZero-Crossing Rate

• Spectrum Amplitude VariationsSpectrum Amplitude Variations

• Spectral CentroidSpectral Centroid


Features AnimationFeatures Animation

MIDISoundtrack

Register

Density - spread

Density - notes

Novelty

Frequency

Duration

Dynamics

Attack Speed

AudioSoundtrack

Mean Roll-off

Mean Zero-Cr.

Std Centroid

Std Flux

Std RMS

Mean Centroid

Mean Flux

Mean RMS

Std Roll-off

Std Zero-Cr.


Motion EditingMotion Editing



• Cyclification Cyclification (Silva ‘99)

• Time warping Time warping (Bruderlin ‘95)

• Various motion linear and non-linear Various motion linear and non-linear filtering techniquesfiltering techniques

• Cyclification Cyclification (Silva ‘99)

• Time warping Time warping (Bruderlin ‘95)

• Various motion linear and non-linear Various motion linear and non-linear filtering techniquesfiltering techniques

(Gleicher ’00)

(Litwinowicz ‘91) (Bruderlin ‘95)



• Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95)

• Standard and multi-target blendingStandard and multi-target blending

• Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95)

• Standard and multi-target blendingStandard and multi-target blending



• Emotional Transforms Emotional Transforms (Polich. ‘01)

• Noise addition Noise addition (Perlin ‘02)

• Motion warping Motion warping (Witkin ’95)

• Emotional Transforms Emotional Transforms (Polich. ‘01)

• Noise addition Noise addition (Perlin ‘02)

• Motion warping Motion warping (Witkin ’95)

(Gleicher ’00)


Example - 1Example - 1

Original Keyframe Motion

Boring…Boring…



Modified Keyframe Motion



Vel

oci

ty

End-effectors Time

Minima



Time

Mo

tio

n A

mp

litu

de

Density Event

New Motion


Animation - 2Animation - 2



Zero-Crossing Rate

RMSVolume


Animation - 3Animation - 3


Future WorkFuture Work

• Add more music analysis and motion Add more music analysis and motion editing methodsediting methods

• Maya Plugin with high-level descriptionsMaya Plugin with high-level descriptions

• User testing (interested ?)User testing (interested ?)

• Add more music analysis and motion Add more music analysis and motion editing methodsediting methods

• Maya Plugin with high-level descriptionsMaya Plugin with high-level descriptions

• User testing (interested ?)User testing (interested ?)


Thank youThank you

Any Questions?Any Questions?

Marc CardleMarc CardleRainbow GroupRainbow Group

Computer LaboratoryComputer LaboratoryUniversity of CambridgeUniversity of Cambridge

[email protected]@cl.cam.ac.ukhttp://www.cl.cam.ac.uk/users/mpc33/http://www.cl.cam.ac.uk/users/mpc33/

Thank youThank you

Any Questions?Any Questions?

Marc CardleMarc CardleRainbow GroupRainbow Group

Computer LaboratoryComputer LaboratoryUniversity of CambridgeUniversity of Cambridge

[email protected]@cl.cam.ac.ukhttp://www.cl.cam.ac.uk/users/mpc33/http://www.cl.cam.ac.uk/users/mpc33/


S2002S2002 Bookman Old Bookman Old Style, Bold, 37 pointsStyle, Bold, 37 pointsS2002S2002 Bookman Old Bookman Old Style, Bold, 37 pointsStyle, Bold, 37 points

This subtitle is 31 points This subtitle is 31 points • Bullets are orange; text is 26 pointsBullets are orange; text is 26 points

• They have 110% line spacing, 6 points They have 110% line spacing, 6 points before/afterbefore/after

• Longer bullets in the form of a paragraph Longer bullets in the form of a paragraph are harder to read if there is insufficient line are harder to read if there is insufficient line spacing. This is the maximum recommended spacing. This is the maximum recommended number of lines per slide (seven).number of lines per slide (seven).• Sub-bullets look like this.

This subtitle is 31 points This subtitle is 31 points • Bullets are orange; text is 26 pointsBullets are orange; text is 26 points

• They have 110% line spacing, 6 points They have 110% line spacing, 6 points before/afterbefore/after

• Longer bullets in the form of a paragraph Longer bullets in the form of a paragraph are harder to read if there is insufficient line are harder to read if there is insufficient line spacing. This is the maximum recommended spacing. This is the maximum recommended number of lines per slide (seven).number of lines per slide (seven).• Sub-bullets look like this.

music-driven motion editing marc cardle rainbow group computer laboratory university of cambridge...

Documents

motion warping witkin

various motion linear

music analysissimplify

time warping bruderlin

points bullets

information midi channel

insufficient line

descriptionsuser testing