music-driven motion editing marc cardle rainbow group computer laboratory university of cambridge...

Music-Driven Motion EditingMusic-Driven Motion EditingMusic-Driven Motion EditingMusic-Driven Motion Editing

Marc CardleMarc Cardle

Rainbow GroupRainbow Group

Computer LaboratoryComputer Laboratory

University of CambridgeUniversity of Cambridge

mpc33@cl.cam.ac.ukmpc33@cl.cam.ac.uk

Marc CardleMarc Cardle

Rainbow GroupRainbow Group

Computer LaboratoryComputer Laboratory

University of CambridgeUniversity of Cambridge

mpc33@cl.cam.ac.ukmpc33@cl.cam.ac.uk

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Overview of TalkOverview of Talk

• Background: Example AnimationBackground: Example Animation

• Research goals and framework overviewResearch goals and framework overview

• Musical FeaturesMusical Features

• Motion EditingMotion Editing

• ExamplesExamples

• Background: Example AnimationBackground: Example Animation

• Research goals and framework overviewResearch goals and framework overview

• Musical FeaturesMusical Features

• Motion EditingMotion Editing

• ExamplesExamples

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

What is driving what?What is driving what?

Film SoundtracksFilm Soundtracks

Music VideosMusic Videos

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Animusic ApproachAnimusic Approach

(Lytle ’94)

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

• A framework for motion editing using A framework for motion editing using music analysismusic analysis

• Simplify and automate the generation of Simplify and automate the generation of synchronised musical animationssynchronised musical animations

• A framework for motion editing using A framework for motion editing using music analysismusic analysis

• Simplify and automate the generation of Simplify and automate the generation of synchronised musical animationssynchronised musical animations

Research GoalResearch GoalResearch GoalResearch Goal

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

MIDIMIDIMIDIMIDI

• Parametric representation Parametric representation

• Pitch, velocityPitch, velocity

• Start-time and End-timeStart-time and End-time

• Parametric representation Parametric representation

• Pitch, velocityPitch, velocity

• Start-time and End-timeStart-time and End-time

SynthesizerSampler

Audio Signal

(Lytle ’94)

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Audio and MIDIAudio and MIDI

• Analysis is based on both MIDI and AudioAnalysis is based on both MIDI and Audio

• Complementary perspectives on the Complementary perspectives on the analysed musicanalysed music

• Analysis is based on both MIDI and AudioAnalysis is based on both MIDI and Audio

• Complementary perspectives on the Complementary perspectives on the analysed musicanalysed music

++AudioMIDI

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

System OverviewSystem Overview

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Musical FeaturesMusical Features

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

MIDI-Based AnalysisMIDI-Based Analysis

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

MIDI-Based AnalysisMIDI-Based Analysis

• Low-level MIDI note information Low-level MIDI note information

• MIDI Channel and Instrument informationMIDI Channel and Instrument information

• Chord RecognizerChord Recognizer

• Virtual PitchVirtual Pitch

• Beat TrackingBeat Tracking

• MeterMeter

• Low-level MIDI note information Low-level MIDI note information

• MIDI Channel and Instrument informationMIDI Channel and Instrument information

• Chord RecognizerChord Recognizer

• Virtual PitchVirtual Pitch

• Beat TrackingBeat Tracking

• MeterMeter

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

MIDI-Based AnalysisMIDI-Based Analysis

• SegmentationSegmentation

• Pattern MatchingPattern Matching

• SegmentationSegmentation

• Pattern MatchingPattern Matching

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

MIDI-Based AnalysisMIDI-Based Analysis

• RegisterRegister

• LoudnessLoudness

• DensityDensity

• DurationDuration

• Attack SpeedAttack Speed

• Focus and DelayFocus and Delay

• RegisterRegister

• LoudnessLoudness

• DensityDensity

• DurationDuration

• Attack SpeedAttack Speed

• Focus and DelayFocus and Delay Time

Max Pitch

Min Pitch

Pit

ch

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Audio-Based AnalysisAudio-Based Analysis

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Audio-Based AnalysisAudio-Based Analysis

• Zero-Crossing RateZero-Crossing Rate

• Spectrum Amplitude VariationsSpectrum Amplitude Variations

• Spectral CentroidSpectral Centroid

• Zero-Crossing RateZero-Crossing Rate

• Spectrum Amplitude VariationsSpectrum Amplitude Variations

• Spectral CentroidSpectral Centroid

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Features AnimationFeatures Animation

MIDISoundtrack

Register

Density - spread

Density - notes

Novelty

Frequency

Duration

Dynamics

Attack Speed

AudioSoundtrack

Mean Roll-off

Mean Zero-Cr.

Std Centroid

Std Flux

Std RMS

Mean Centroid

Mean Flux

Mean RMS

Std Roll-off

Std Zero-Cr.

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Motion EditingMotion Editing

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Motion EditingMotion Editing

• Cyclification Cyclification (Silva ‘99)

• Time warping Time warping (Bruderlin ‘95)

• Various motion linear and non-linear Various motion linear and non-linear filtering techniquesfiltering techniques

• Cyclification Cyclification (Silva ‘99)

• Time warping Time warping (Bruderlin ‘95)

• Various motion linear and non-linear Various motion linear and non-linear filtering techniquesfiltering techniques

(Gleicher ’00)

(Litwinowicz ‘91) (Bruderlin ‘95)

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Motion EditingMotion Editing

• Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95)

• Standard and multi-target blendingStandard and multi-target blending

• Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95)

• Standard and multi-target blendingStandard and multi-target blending

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Motion EditingMotion Editing

• Emotional Transforms Emotional Transforms (Polich. ‘01)

• Noise addition Noise addition (Perlin ‘02)

• Motion warping Motion warping (Witkin ’95)

• Emotional Transforms Emotional Transforms (Polich. ‘01)

• Noise addition Noise addition (Perlin ‘02)

• Motion warping Motion warping (Witkin ’95)

(Gleicher ’00)

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Example - 1Example - 1

Original Keyframe Motion

Boring…Boring…

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Example - 1Example - 1

Modified Keyframe Motion

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Example - 2Example - 2

Vel

oci

ty

End-effectors Time

Minima

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Example - 2Example - 2

Time

Mo

tio

n A

mp

litu

de

Density Event

New Motion

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Animation - 2Animation - 2

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Example - 3Example - 3

Zero-Crossing Rate

RMSVolume

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Animation - 3Animation - 3

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Future WorkFuture Work

• Add more music analysis and motion Add more music analysis and motion editing methodsediting methods

• Maya Plugin with high-level descriptionsMaya Plugin with high-level descriptions

• User testing (interested ?)User testing (interested ?)

• Add more music analysis and motion Add more music analysis and motion editing methodsediting methods

• Maya Plugin with high-level descriptionsMaya Plugin with high-level descriptions

• User testing (interested ?)User testing (interested ?)

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

Thank youThank you

Any Questions?Any Questions?

Marc CardleMarc CardleRainbow GroupRainbow Group

Computer LaboratoryComputer LaboratoryUniversity of CambridgeUniversity of Cambridge

mpc33@cl.cam.ac.ukmpc33@cl.cam.ac.ukhttp://www.cl.cam.ac.uk/users/mpc33/http://www.cl.cam.ac.uk/users/mpc33/

Thank youThank you

Any Questions?Any Questions?

Marc CardleMarc CardleRainbow GroupRainbow Group

Computer LaboratoryComputer LaboratoryUniversity of CambridgeUniversity of Cambridge

mpc33@cl.cam.ac.ukmpc33@cl.cam.ac.ukhttp://www.cl.cam.ac.uk/users/mpc33/http://www.cl.cam.ac.uk/users/mpc33/

Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle

S2002S2002 Bookman Old Bookman Old Style, Bold, 37 pointsStyle, Bold, 37 pointsS2002S2002 Bookman Old Bookman Old Style, Bold, 37 pointsStyle, Bold, 37 points

This subtitle is 31 points This subtitle is 31 points • Bullets are orange; text is 26 pointsBullets are orange; text is 26 points

• They have 110% line spacing, 6 points They have 110% line spacing, 6 points before/afterbefore/after

• Longer bullets in the form of a paragraph Longer bullets in the form of a paragraph are harder to read if there is insufficient line are harder to read if there is insufficient line spacing. This is the maximum recommended spacing. This is the maximum recommended number of lines per slide (seven).number of lines per slide (seven).• Sub-bullets look like this.

This subtitle is 31 points This subtitle is 31 points • Bullets are orange; text is 26 pointsBullets are orange; text is 26 points

• They have 110% line spacing, 6 points They have 110% line spacing, 6 points before/afterbefore/after

• Longer bullets in the form of a paragraph Longer bullets in the form of a paragraph are harder to read if there is insufficient line are harder to read if there is insufficient line spacing. This is the maximum recommended spacing. This is the maximum recommended number of lines per slide (seven).number of lines per slide (seven).• Sub-bullets look like this.