multiresolution motion analysis with applications jehee lee sung yong shin dept of ee&cs, kaist...
TRANSCRIPT
Multiresolution Motion Analysiswith Applications
Jehee Lee Sung Yong ShinJehee Lee Sung Yong Shin
Dept of EE&CS, KAISTDept of EE&CS, KAIST
Jehee Lee Sung Yong ShinJehee Lee Sung Yong Shin
Dept of EE&CS, KAISTDept of EE&CS, KAIST
Contents
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
Character Animation
Realistic motion dataRealistic motion data
• Motion capture technologyMotion capture technology
• Commercial librariesCommercial libraries
Producing animation from available motion clipsProducing animation from available motion clips
• requires specialized toolsrequires specialized tools
– interactive editing, smoothing, enhancement, blending, interactive editing, smoothing, enhancement, blending, stitching, and so onstitching, and so on
• stimulates the need for a unified approachstimulates the need for a unified approach
Realistic motion dataRealistic motion data
• Motion capture technologyMotion capture technology
• Commercial librariesCommercial libraries
Producing animation from available motion clipsProducing animation from available motion clips
• requires specialized toolsrequires specialized tools
– interactive editing, smoothing, enhancement, blending, interactive editing, smoothing, enhancement, blending, stitching, and so onstitching, and so on
• stimulates the need for a unified approachstimulates the need for a unified approach
Multiresolution Analysis
Representing a signal at multiple resolutionsRepresenting a signal at multiple resolutions
• gives hierarchy of successively smoother signalsgives hierarchy of successively smoother signals
• facilitates a variety of signal processing tasksfacilitates a variety of signal processing tasks
Representing a signal at multiple resolutionsRepresenting a signal at multiple resolutions
• gives hierarchy of successively smoother signalsgives hierarchy of successively smoother signals
• facilitates a variety of signal processing tasksfacilitates a variety of signal processing tasks
(3)m(3)m (2)m(2)m (1)m(1)m (0)m(0)m
Previous Work
Image and signal processingImage and signal processing
• Gauss-Laplacian pyramidGauss-Laplacian pyramid [Burt and Adelson [Burt and Adelson
83]83]
• Texture analysis and synthesis, image editing, curve and Texture analysis and synthesis, image editing, curve and surface manipulation, data compression, and so onsurface manipulation, data compression, and so on
Motion synthesis and editingMotion synthesis and editing
• Hierarchical spacetime controlHierarchical spacetime control [Liu, Gortler and Cohen [Liu, Gortler and Cohen
94]94]
• Motion signal processingMotion signal processing [Bruderlin and Williams [Bruderlin and Williams
95]95]
Image and signal processingImage and signal processing
• Gauss-Laplacian pyramidGauss-Laplacian pyramid [Burt and Adelson [Burt and Adelson
83]83]
• Texture analysis and synthesis, image editing, curve and Texture analysis and synthesis, image editing, curve and surface manipulation, data compression, and so onsurface manipulation, data compression, and so on
Motion synthesis and editingMotion synthesis and editing
• Hierarchical spacetime controlHierarchical spacetime control [Liu, Gortler and Cohen [Liu, Gortler and Cohen
94]94]
• Motion signal processingMotion signal processing [Bruderlin and Williams [Bruderlin and Williams
95]95]
Issues in Motion Analysis
Difficulties in handling motion dataDifficulties in handling motion data• Inherent non-linearity of orientation spaceInherent non-linearity of orientation space
Coordinate-invarianceCoordinate-invariance• Independent of the choice of coordinate framesIndependent of the choice of coordinate frames
Difficulties in handling motion dataDifficulties in handling motion data• Inherent non-linearity of orientation spaceInherent non-linearity of orientation space
Coordinate-invarianceCoordinate-invariance• Independent of the choice of coordinate framesIndependent of the choice of coordinate frames
Contents
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
Motion Representation
Configuration of articulated figuresConfiguration of articulated figures
• Bundle of motion signalsBundle of motion signals
• Each signal represents time-varying positions and Each signal represents time-varying positions and orientationsorientations
• Rigid transformationRigid transformation
Configuration of articulated figuresConfiguration of articulated figures
• Bundle of motion signalsBundle of motion signals
• Each signal represents time-varying positions and Each signal represents time-varying positions and orientationsorientations
• Rigid transformationRigid transformation
33))()(()( SRqpm ttt , 33))()(()( SRqpm ttt ,
pququqp 1)( )(T , pququqp 1)( )(T ,
33:T RR 33:T RR
Decomposition
Expansion : Expansion : up-sampling followed by smoothingup-sampling followed by smoothing
Reduction : Reduction : smoothing followed by down-samplingsmoothing followed by down-sampling
Expansion : Expansion : up-sampling followed by smoothingup-sampling followed by smoothing
Reduction : Reduction : smoothing followed by down-samplingsmoothing followed by down-sampling
)(nm )(nm
)1( nm )1( nm
)1( nd )1( nd
ReductionReduction ExpansionExpansion
Decomposition and Reconstruction
DecompositionDecomposition
ReconstructionReconstruction
DecompositionDecomposition
ReconstructionReconstruction
)(nm )(nm )1( nm )1( nm )2( nm )2( nm )0(m )0(m
)1( nd )1( nd )2( nd )2( nd )0(d )0(d
)0(m )0(m
)1( nd )1( nd )2( nd )2( nd )0(d )0(d
)(nm )(nm )1( nm )1( nm )2( nm )2( nm
Our Approach
Multiresolution Motion AnalysisMultiresolution Motion Analysis
• Hierarchical displacement mappingHierarchical displacement mapping
– How to representHow to represent
– Displacement mapping Displacement mapping [Bruderlin and Williams 95][Bruderlin and Williams 95]
– Motion warping Motion warping [Popovic and Witkin 95][Popovic and Witkin 95]
• Spatial filtering for motion dataSpatial filtering for motion data
– How to constructHow to construct
– Implement reduction and expansionImplement reduction and expansion
Multiresolution Motion AnalysisMultiresolution Motion Analysis
• Hierarchical displacement mappingHierarchical displacement mapping
– How to representHow to represent
– Displacement mapping Displacement mapping [Bruderlin and Williams 95][Bruderlin and Williams 95]
– Motion warping Motion warping [Popovic and Witkin 95][Popovic and Witkin 95]
• Spatial filtering for motion dataSpatial filtering for motion data
– How to constructHow to construct
– Implement reduction and expansionImplement reduction and expansion
Contents
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
Motion Displacement
global (fixed) reference frame
global (fixed) reference frame
TT
ation transformrigid a : T ation transformrigid a : T
)( qpm , )( qpm ,)( qpm , )( qpm ,
33
by zedparameterilocally
RRv)(u,d 33
by zedparameterilocally
RRv)(u,d
Motion Displacement
)exp(
)()(1 vqpquq
vuqp
dmm
,
,,
'
)exp(
)()(1 vqpquq
vuqp
dmm
,
,,
'
global (fixed) reference frame
global (fixed) reference frame
vv
uu)( qpm , )( qpm ,
)( qpm , )( qpm ,
Hierarchical Displacement Mapping
)0(m )0(m
)0(d )0(d
)1(d )1(d
)2(d )2(d
Hierarchical Displacement Mapping
)0(m )0(m
)0(d )0(d
)1(d )1(d
)2(d )2(d
)0(m )0(m
Hierarchical Displacement Mapping
)0(m )0(m
)0(d )0(d
)1(d )1(d
)2(d )2(d
)0(mE )0(mE
Hierarchical Displacement Mapping
)0(m )0(m
)0(d )0(d
)1(d )1(d
)2(d )2(d
)(E 0)0()1( dmm )(E 0)0()1( dmm
A series of successively refined motionsA series of successively refined motions
• Coordinate-independentCoordinate-independent
– measured in a body-fixed coordinate framemeasured in a body-fixed coordinate frame
• UniformityUniformity
– through a local parameterizationthrough a local parameterization
A series of successively refined motionsA series of successively refined motions
• Coordinate-independentCoordinate-independent
– measured in a body-fixed coordinate framemeasured in a body-fixed coordinate frame
• UniformityUniformity
– through a local parameterizationthrough a local parameterization
Hierarchical Displacement Mapping
)))(E(E(E )()()()()( nn dddmm 100 )))(E(E(E )()()()()( nn dddmm 100
Contents
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
Spatial Filtering for Orientation Data
Linear shift-invariant (LSI) filterLinear shift-invariant (LSI) filter
• filter mask : filter mask :
• vector-valued signal :vector-valued signal :
Not suitable for unit quaternion dataNot suitable for unit quaternion data
• unit-length constraintsunit-length constraints
Linear shift-invariant (LSI) filterLinear shift-invariant (LSI) filter
• filter mask : filter mask :
• vector-valued signal :vector-valued signal :
Not suitable for unit quaternion dataNot suitable for unit quaternion data
• unit-length constraintsunit-length constraints
kikikiki aaa pppp 0)(F kikikiki aaa pppp 0)(F
R ikk aaaa ),,,,,( 0 R ikk aaaa ),,,,,( 0 3Rp i3Rp i
Previous Work
Re-normalizationRe-normalization
• Azuma and Bishop (‘94)Azuma and Bishop (‘94)
Exploit a local parameterizationExploit a local parameterization
• Lee and Shin (‘96)Lee and Shin (‘96)
• Welch and Bishop (‘97)Welch and Bishop (‘97)
• Fang Fang et al.et al. (‘98) (‘98)
• Hsieh Hsieh et al.et al. (‘98) (‘98)
– lack of crucial filter propertieslack of crucial filter properties
Re-normalizationRe-normalization
• Azuma and Bishop (‘94)Azuma and Bishop (‘94)
Exploit a local parameterizationExploit a local parameterization
• Lee and Shin (‘96)Lee and Shin (‘96)
• Welch and Bishop (‘97)Welch and Bishop (‘97)
• Fang Fang et al.et al. (‘98) (‘98)
• Hsieh Hsieh et al.et al. (‘98) (‘98)
– lack of crucial filter propertieslack of crucial filter properties
Basic Idea
Exploit correspondence in differential spacesExploit correspondence in differential spaces
• Linear motion : Linear motion :
• Angular motion :Angular motion :
Exploit correspondence in differential spacesExploit correspondence in differential spaces
• Linear motion : Linear motion :
• Angular motion :Angular motion : 3)( Sq t 3)( Sq t
3)( Rp t 3)( Rp t
3S3S 3R3R
VelocityVelocity
AccelerationAcceleration '''
'
pω
pω
pq
Transformation
Transformation between linear and angular signalsTransformation between linear and angular signalsTransformation between linear and angular signalsTransformation between linear and angular signals
3S3S 3R3R
2p2p
0p0p
1p1p0q0q
2q2q
1q1q
iiii ppqq
111log iiii ppqq
11
1log
Filter Design
Given: spatial filter Given: spatial filter FF
Output: spatial filter Output: spatial filter HH for orientation data for orientation data
• ““Unitariness” is guaranteedUnitariness” is guaranteed
Given: spatial filter Given: spatial filter FF
Output: spatial filter Output: spatial filter HH for orientation data for orientation data
• ““Unitariness” is guaranteedUnitariness” is guaranteed
iiii ppqq )(exp)( FH iiii ppqq )(exp)( FH
Filter Design
Given: spatial filter Given: spatial filter FF
Output: spatial filter Output: spatial filter HH for orientation data for orientation data
• Local supportLocal support
– #support(#support(HH ) = #support() = #support(FF ))
Given: spatial filter Given: spatial filter FF
Output: spatial filter Output: spatial filter HH for orientation data for orientation data
• Local supportLocal support
– #support(#support(HH ) = #support() = #support(FF ))
1
11logexp
)(exp)(
k
kmmimimi
iiii
b qqq
ppqq FH
1
11logexp
)(exp)(
k
kmmimimi
iiii
b qqq
ppqq FH
Properties of Orientation Filters
Coordinate-invarianceCoordinate-invariance
Time-invarianceTime-invariance
SymmetrySymmetry
Coordinate-invarianceCoordinate-invariance
Time-invarianceTime-invariance
SymmetrySymmetry
ii qqRH,RRH where ii qqRH,RRH where
niinnn
qqSH,SSH where niinnn
qqSH,SSH where
3where Sbabqabaq ,,)H()H( ii3where Sbabqabaq ,,)H()H( ii
Examples (1)
Blurring by binomial masksBlurring by binomial masksBlurring by binomial masksBlurring by binomial masks 1464116
1,,,, 14641
16
1,,,,
OriginalOriginal Angular accelerationAngular accelerationFilteredFiltered
OriginalFilteredOriginalFiltered
Examples (2)
SmoothingSmoothingSmoothingSmoothing 14104116
1 ,,,, 141041
16
1 ,,,,
OriginalOriginal Angular accelerationAngular accelerationFilteredFiltered
OriginalFilteredOriginalFiltered
Examples (3)
High-frequency boostingHigh-frequency boostingHigh-frequency boostingHigh-frequency boosting 14264116
1 ,,,, 142641
16
1 ,,,,
OriginalOriginal Angular accelerationAngular accelerationFilteredFiltered
OriginalFilteredOriginalFiltered
Our scheme vs. Re-normalization
iqiq
1iq 1iq
2iq 2iq
1iq 1iq
2iq 2iq
Re-normalizationiqiq
1iq 1iq
2iq 2iq
1iq 1iq
2iq 2iq
Our scheme
Filtering with an average filterFiltering with an average filterFiltering with an average filterFiltering with an average filter 111115
1,,,, 11111
5
1,,,,
Coordinate Frame-Invariance
mm
)0(
)0(
)1(
)(
mdd
d
n
)0(
)0(
)1(
)(
mdd
d
n
)0(
)0(
)1(
)(
mdd
d
T
n
)0(
)0(
)1(
)(
mdd
d
T
n
mTmTDecomposition Reconstruction
TT
TT
Contents
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
1.1. IntroductionIntroduction
2.2. Multiresolution AnalysisMultiresolution Analysis
3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping
4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data
5.5. ApplicationsApplications
Enhancement / Attenuation
Level-wise scaling of coefficientsLevel-wise scaling of coefficientsLevel-wise scaling of coefficientsLevel-wise scaling of coefficients
Motion Blending
Combine multiple motions togetherCombine multiple motions together
• select a base signal and details from different examplesselect a base signal and details from different examples
Combine multiple motions togetherCombine multiple motions together
• select a base signal and details from different examplesselect a base signal and details from different examples
straightwalkingstraightwalking
turningwith a walk
turningwith a walk
straightlimpingstraightlimping
turningwith a limp
turningwith a limp
Motion Blending
Motion Stitching
A simple approachA simple approach
• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then
• Perform -interpolationPerform -interpolation
A simple approachA simple approach
• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then
• Perform -interpolationPerform -interpolation1C1C
Motion Stitching
A simple approachA simple approach
• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then
• Perform -interpolationPerform -interpolation
A simple approachA simple approach
• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then
• Perform -interpolationPerform -interpolation1C1C
Motion Stitching
Difficulties of the simple approachDifficulties of the simple approach
• Hard to estimate velocity robustlyHard to estimate velocity robustly
Difficulties of the simple approachDifficulties of the simple approach
• Hard to estimate velocity robustlyHard to estimate velocity robustly
Motion Stitching
Stitching motion clips seamlesslyStitching motion clips seamlessly
• Merging coefficients level-by-levelMerging coefficients level-by-level
Stitching motion clips seamlesslyStitching motion clips seamlessly
• Merging coefficients level-by-levelMerging coefficients level-by-level
WalkingRunning
Motion Stitching
Stitching motion clips seamlesslyStitching motion clips seamlessly
• Merging coefficients level-by-levelMerging coefficients level-by-level
Stitching motion clips seamlesslyStitching motion clips seamlessly
• Merging coefficients level-by-levelMerging coefficients level-by-level
WalkingRunning
Motion Stitching
Stitching motion clips seamlesslyStitching motion clips seamlessly
• Merging coefficients level-by-levelMerging coefficients level-by-level
Stitching motion clips seamlesslyStitching motion clips seamlessly
• Merging coefficients level-by-levelMerging coefficients level-by-level
stub a toe limp stitching
System DemonstrationSystem DemonstrationSystem DemonstrationSystem Demonstration
Conclusion
Multiresolution motion AnalysisMultiresolution motion Analysis
• Coherency in positions and orientationsCoherency in positions and orientations
• Coordinate-invarianceCoordinate-invariance
Multiresolution motion AnalysisMultiresolution motion Analysis
• Coherency in positions and orientationsCoherency in positions and orientations
• Coordinate-invarianceCoordinate-invariance