Multivariate Extensions of EMDApplications in Data Fusion and BCI
Danilo P. Mandic
Imperial College London, UKAckn: Naveed Ur Rehman, David Looney, Cheolsoo Park
[email protected], URL: www.commsp.ee.ic.ac.uk/∼mandic
c© D. P. Mandic The Third HHT Conference, Qingdao, China 1
About Imperial College
c© D. P. Mandic The Third HHT Conference, Qingdao, China 2
Outline
2 Concept of Data Fusion Via Fission
2 Empirical Mode Decomposition (EMD)
2 Hilbert-Huang reconstruction and examples
2 Complex EMD
2 Applications: Image Restoration and Image Fusion
2 Trivariate EMD: Methodology and Applications
2 Multivariate EMD (MEMD)
2 Filter bank property of MEMD
2 Noise-assisted MEMD
2 Conclusions
c© D. P. Mandic The Third HHT Conference, Qingdao, China 3
Problem Statement
Classic spectrum estimation: From a finite record of stationary datasequence, estimate how the total power is distributed over frequency.
Has found a tremendous number of applications:-
◦ Seismology – oil exploration, earthquake, Radar and sonar – location ofsources, Speech and audio – recognition, Astronomy – periodicities,Economy – seasonal and periodic components, Medicine – EEG, ECG
Modern view: From a finite or infinite record of non-stationary andnon-liner data sequence, estimate how the total power is distributed overtime-frequency.
What are the basis for this analysis
◦ Parametric: stochastic models (AR, MA, ARMA), ...
◦ Non-parametric: Fourier analysis, wavelets, Vigner-Ville, ...
◦ Data driven: adaptivity in the derivation of the bases
c© D. P. Mandic The Third HHT Conference, Qingdao, China 4
Bases for Signal Decomposition
“Linear methods” - signal analysed based on inner products with apredefined family of basis functions.
◦ Fourier: bases are sin and cos which are orthonormal and in generalrequire an infinite number of terms in the expansion
F{x[n]} =N−1∑
n=0
x[n]e−ωn
= < x, e > x = [x0, . . . , xN−1]T , e = [1, e2ω, . . . , e(N−1)ω]T
◦ Wavelet: assumes projecting on a pre-defined “mother” wavelet, whichshrinks and expands, thus bypassing some of the problems of Fourieranalysis, but not entirely
The bases (template functions) determine the properties of representationand influences the physical reading
e.g. “frequency bins” instead of “instantaneous frequency”
c© D. P. Mandic The Third HHT Conference, Qingdao, China 5
Real World Signals – ‘Nonlinearity’ and ‘Stochasticity’
a) Periodic oscillations b) Small nonlinearity c) Route to chaos
d) Route to chaos e) small noise f) HMM and others
Hence: we need to look into nonparametric representations ofnonlinear and nonstationary signals
c© D. P. Mandic The Third HHT Conference, Qingdao, China 6
Speech Example – saying “Matlab” – ‘specgramdemo‘
Frequency
time
M aaa t l aaa b
For every time instant “t”, the PSD is plotted along the vertical axis
Darker areas:- higher magnitude of PSD
c© D. P. Mandic The Third HHT Conference, Qingdao, China 7
STFFT of a speech signal
(wide band spectrogram) (narrow band spectrogram)
dB
Data=[4001x1], Fs=7.418 kHz
-50
-40
-30
-20
-10
0
10
20
30
20
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Fre
qu
en
cy,
kH
z
dB
50 100 150 200 250 300 350 400 450 500-5
0
5
Time, ms
Am
pl
515.5028 ms
0.0000 Hz
29.2416 dB
dB
Data=[4001x1], Fs=7.418 kHz
-40
-30
-20
-10
0
10
20
30
20
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Fre
qu
en
cy,
kH
z
dB
50 100 150 200 250 300 350 400 450-5
0
5
Time, ms
Am
pl
241.5745 ms
1.8545 kHz
3.2925 dB
(win-len=256, overlap=200, ftt-len=32) (win-len=512, overlap=200, ftt-len=256)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 8
What is the Right Basis for Real World Data?
Consider
• Amplitude modulated signal x(t) = m(t) cos(ω0t) → m(t) - envelope
• Phase modulated signal x(t) = a cos(Φ(t)) → Φ(t) - phase
Problem: there is an infinite number of pairs [a(t),Φ(t)] s.t.m(t) cos(ω0t) = a cos(Φ(t))
Solution: an analytic transform z(t) = x(t) + H(
x(t))
Remark#1: z(t) cannot be real, as F(
z(t))
= 0 for ω < 0
Remark#2: Hilbert transform (analytic signal) makes it possible toassociate a unique pair
[
a(t),Φ(t)]
to any real x(t) = ℜ{
a(t)eΦ(t)}
Remark#3: For x(t) = a(t) cosΦ(t) ⇒ H{
x(t)}
= a(t) sinΦ(t)
Remark#4: From instantaneous phase Φ(t) → instantaneousfrequency
f(t) = dΦ(t)/dt
so we have an excellent resolution and do not depend on stationarity
c© D. P. Mandic The Third HHT Conference, Qingdao, China 9
MEMD vs STFFT for a speech signal
(STFFT spectrogram) (Hilbert-Huang spectra)
dB
Data=[4001x1], Fs=7.418 kHz
-60
-50
-40
-30
-20
-10
0
10
20
30
-6020
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Fre
qu
en
cy,
kH
z
dB
50 100 150 200 250 300 350 400 450 500-5
0
5
Time, ms
Am
pl
515.5028 ms
0.0000 Hz
29.2416 dB
Fre
qu
en
cy
Time
MEMD
500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500
3000
3500
(win-len=256, overlap=200, ftt-len=256) (no. of directions=64)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 10
Empirical Mode Decomposition (EMD): Introduction
2 Empirical Mode Decomposition has been recently introduced for thetime-frequency analysis of nonstationary and nonlinear signals
2 Projection-based techniques assume stationarity and/or linearity in theinput signal (Fourier and Wavelet approach), and hence, are notsuitable for non-linear, non-stationary data
2 The Empirical mode decomposition (EMD) algorithm is a fullydata-driven method which extracts the basis functions adaptively fromthe input data, through the so called sifting process
2 The adaptive nature of EMD also facilitates accurate time-frequencyrepresentation of the signal at the level of instantaneous frequency(through the Hilbert-Huang spectrum)
2 Standard EMD is limited to the analysis of single data channels -modern applications require its multichannel extensions
2 For data fusion ⇔ same number of IMFs for all data channels
c© D. P. Mandic The Third HHT Conference, Qingdao, China 11
Hilbert-Huang Spectrum: Results
Spectrogram and Hilbert–Huang Spectrum for a sum of two frequency modulated signals
and a tone. The Hilbert-Huang spectrum shown in (right) clearly shows better resolution
than the spectrogram shown on (left).
1000 2000 3000 4000 5000 6000 70000
500
1000
1500
2000
2500
3000
3500
4000
Spectrogram
Time Index
Fre
qu
en
cy (
Hz)
Fre
qu
en
cy (
Hz)
Time Index
Hilbert−Huang Spectrum
1000 2000 3000 4000 5000 6000 70000
500
1000
1500
2000
2500
3000
3500
4000
c© D. P. Mandic The Third HHT Conference, Qingdao, China 12
Vector sensors - Data Fusion - Synchrony
Renewable Energy Body motion sensor Wearable techologies
2D and 3D anemometers 3D - position, gyroscope, speed Biomechanics
control of wind turbine gait, biometrics virtual reality
c© D. P. Mandic The Third HHT Conference, Qingdao, China 13
Vector sensors - 3D anemometer
c© D. P. Mandic The Third HHT Conference, Qingdao, China 14
Applications: How do we Decompose - Fuse RGB
Computer Games Medical Applications Avionics
Rotation of polygons 3D time-space. Trajectory tracking of
to form 3D graphics. Data are naturally recorded in angular properties (eg. velocity).
2D and 3D electromagnetic field.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 15
A General Data Fusion Problem
How do we make a mental ”image” of a meal
• taste bitter, rotten
• smell pleasant, spices
• vision colour, presentation, rare, medium, well done
• touch bread, toast
• hearing, temperature, toughness
c© D. P. Mandic The Third HHT Conference, Qingdao, China 16
Data Fusion via Fission
Input
FusionFission
IMFN
IMF2
IMF1
...
frequency
time
Output
2 Fission : Decomposion into “particles”
2 Fusion: Recombination of particles into the desired signal
c© D. P. Mandic The Third HHT Conference, Qingdao, China 17
Automated Data Fusion via Fission (MLEMD)
Standard EMD
1
maskbinary
IMFn
IMF3IMF2IMF1
EMDrecombinationsignal
input
...
0
11
00
EMD for Data Fusion via Fission
x(k)
M
c (k)1
Bivariate
Machine Learning
EMD
W(k)
y(k)
c (k)
Incorporating the scale and temporal information
Most Machine Learning or Adaptive Filtering algorithms can beincorporated (Looney and Mandic, ICASSP 2009).
c© D. P. Mandic The Third HHT Conference, Qingdao, China 18
Benefits of the Data Fusion Approach
The synergy of information fragments offers some advantages overstandard algorithms, such as:-
• Improved confidence due to complementary and redundant information;
• Robustness and reliability in adverse conditions (smoke, noise,occlusion);
• Increased coverage in space and time; dimensionality of the data space;
• Better discrimination between hypotheses due to more completeinformation;
• System being operational even if one or several sensors aremalfunctioning;
• Possible solution to the vast amount available information.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 19
Some Literature
c© D. P. Mandic The Third HHT Conference, Qingdao, China 20
Image Enhancement and Fusion Using EMD
2 Image features (object texture or unwanted noise) can be attributed tolocal variations in spatial frequencies
2 Therefore, the behaviour of the extracted image modes can reflect thesefeatures
2 Correct fusion of the “relevant” IMFs can be used to highlight (orremove) specific image attributes
We consider the fusion capabilities of EMD under the following headings:
• Image Denoising
• Image Restoration (Illumination Removal)
• Image Fusion (of Images From Multiple Image Modalities)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 21
Image Denoising
Noise contamination is a common problem when acquiring real worldimages and consequently image denoising is an important element ofimage processing. Many existing methods, however, are sub-optimal forthe following reasons:
2 They make unrealistic assumptions about the data (ICA - unrealisticindependence conditions, PCA - noise and original image can beseparated by linear projection)
2 They are not optimised for enhancing higher order (nonlinear) statistics,that are commonly associated with the perceptual quality of an image,and do not cater for other real world data characteristics such asnonstationarity (block based Weiner filtering)
2 They are computationally complex (Bayesian and particle models)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 22
Image Denoising
Consider an original image corrupted by white Gaussian noise.
Original Contaminated (SR 12.3 dB)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 23
Image Denoising
Decomposing the contaminated image by EMD, we obtain the following:
Note how each of ‘Image Modes’ represents the frequency scales within theimage. The higher index IMFs contain high frequency detail such as theimage edges while slowly oscillating effects such as illumination arecontained within the low index IMFs.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 24
Image Denoising
Noisy image, SNR = 13 dB EMD of the image
0 100 200 300 400 5000
200400
imag
e
50 100 150 200 250 300 350 400 450 500−20
020
c 1
50 100 150 200 250 300 350 400 450 500−20
020
c 2
50 100 150 200 250 300 350 400 450 500−20
020
c 3
50 100 150 200 250 300 350 400 450 500−20
020
c 4
50 100 150 200 250 300 350 400 450 500−20
020
c 5
50 100 150 200 250 300 350 400 450 500−50
050
c 6
Time Index
Denoising using PREMD, SNR=17.5 dB Denoising using MLEMD, SNR=21 dB
c© D. P. Mandic The Third HHT Conference, Qingdao, China 25
The Method Naturally Deals With Texture
Cracked varnish on wood: Left – original; Midle – cracks; Right – wood pattern
Carpet: Left – original; Midle – texture; Right – carpet patternThe texture is separated naturally as higher frequency T-F components
c© D. P. Mandic The Third HHT Conference, Qingdao, China 26
Image Restoration (Illumination Removal)
2 A key problem for a machine vision system is image changes that occurdue to scene illumination
2 Incident light on a surface produces complex artifacts, making itdifficult for the system to separate changes caused by local variations inillumination intensity and colour
• It can be assumed that shade in images creates low valued regions withlarge extrema that change slowly
• It is therefore likely that the effects of the shade will be isolated in thelower index IMFs and a shade free image can be achieved by combiningthe relevant IMFs
c© D. P. Mandic The Third HHT Conference, Qingdao, China 27
Illumination Removal – Real World Objects
Image with shade Shade only Original image
Shade removal: the shading is now uniform across the image surfaceImage with shade Shade only Original image
c© D. P. Mandic The Third HHT Conference, Qingdao, China 28
Human Visual System – Importance of Phase
Information
Surrogate images. Top: Original images I1 and I2; Bottom: Images I1 and I2 generated
by exchanging the amplitude and phase spectra of the original images.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 29
Image Fusion (From Multiple Image Modalities)
2 Image fusion is becoming an important area of research, particularly asdifferent methods of image acquisition become available
2 The fused image retains all “relevant information” from the differentsources while disregarding unwanted artifacts
• Given the unique “fission” properties of EMD, it has a strong potentialfor fusion
• We propose the use of complex EMD with the input images as real andimaginary components respectively
• The instantaneous amplitude of the extracted IMFs indicates, for eachfrequency level at each pixel, which of the components contains thesalient information. Fusion can be achieved by combining only IMFcomponents with the largest instanteous amplitudes.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 30
Obstacles to Automatic Heterogeneous EMD Fusion
2 The fully adaptive and empirical nature of the algorithm compromisesthe uniqueness of the decomposition.
2 Signals with similar statistics often yield different IMFs (in both numberand frequency) - difficult to compare sources in T-F
Consider sinusoid corrupted by different realisations of AWGN. Note thethe difference in the IMFs.
0 100 200 300 400 500 600−1
01
U
0 100 200 300 400 500 600−1
01
C1+
C2
0 100 200 300 400 500 600−1
01
C3
100 200 300 400 500 600−1
01
C4
0 100 200 300 400 500 600−1
01
C5:
r
Time Index
0 100 200 300 400 500 600−1
01
U
0 100 200 300 400 500 600−1
01
C1+
C2
0 100 200 300 400 500 600−1
01
C3
100 200 300 400 500 600−1
01
C4
0 100 200 300 400 500 600−1
01
Time Index
C5:
r
Seven IMFs Eight IMFs
c© D. P. Mandic The Third HHT Conference, Qingdao, China 31
Obstacles to Automatic Heterogeneous EMD Fusion
Automatic fusion algorithms are necessary for widespread use!
But this is not often possible using standard EMD because
3 Uniqueness of the scales cannot be guaranteed;
3 Comparison of IMFs from different sources is meaningless!
Thus, automatic fusion of heterogeneous sources using EMD is onlypossible if their IMFs are
2 equal in number;
2 matched in properties (frequency).
[Rotation Invariant Complex EMD, Altaf, Mandic et al., 2007, Complex
EMD, Tanaka and Mandic, 2007, Bivariate EMD, Rilling, Flandrin andGoncalves, 2007]
c© D. P. Mandic The Third HHT Conference, Qingdao, China 32
Empirical Mode Decomposition: Underlying Idea
• The basic idea behind EMD is to consider an input signal as fastoscillations superimposed on slow oscillations.
• The fast oscillations are repeatedly sifted from the input signal until amonotonic signal (residue) is obtained.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 33
Complex EMD - Local Mean Estimation
−200 −150 −100 −50 0 50 100 150 200 250−100
−50
0
50
100
150
200
X
Y
Original dataenvelope 1envelope 2Mean
(e) RIEMD
−200 −150 −100 −50 0 50 100 150 200 250−100
−50
0
50
100
150
200
X
Y
Original Dataenvelope 1envelope 2envelope 3envelope 4Mean
(f) BEMD
1
2
3
4
5
30
210
60
240
90
270
120
300
150
330
180 0
(g) A complex wind signal
1
2
3
4
5
30
210
60
240
90
270
120
300
150
330
180 0
(h) IMF6 + IMF7
c© D. P. Mandic The Third HHT Conference, Qingdao, China 34
Heterogeneous EMD Fusion
3 It was proposed [Looney and Mandic] to use the complex extensions ofthe algorithm to decompose heterogeneous sources simultaneously.
3 The approach may be used to find “common scales” within differentdata sets, thus addressing the problem of uniqueness.
Observe how common frequency scales are found in different signals (U1and U2) by applying complex extensions of EMD to (U1 + jU2).
20 40 60 80 100−4
04
U1
20 40 60 80 100−4
04
C2
0 20 40 60 80 100−4
04
C1
20 40 60 80 100−4
04
C3
20 40 60 80 100−4
04
Time Index
C4:
r
20 40 60 80 100−4
04
U2
20 40 60 80 100−4
04
C1
20 40 60 80 100−4
04
C2
20 40 60 80 100−40 4
C3
20 40 60 80 100−4
04
Time Index
C4:
r
Real Imaginary
c© D. P. Mandic The Third HHT Conference, Qingdao, China 35
Fusion Results [Looney and Mandic ICDSC’08)
Visual Thermal Pixel Average Fusion
PCA Fusion Wavelet Fusion Complex EMD Fusion
c© D. P. Mandic The Third HHT Conference, Qingdao, China 36
Out of focus image fusion using complex EMD
A (original) B (original) Out of focus fusion
A (original) B (original) Fusion
Looney and Mandic, IEEE Transactions on Signal Processing, April 2009.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 37
Complex EMD vs Wavelets
EMD fusion Wavelet fusion
The wavelets produce artifacts - around the text visible as shaded “boxes”
c© D. P. Mandic The Third HHT Conference, Qingdao, China 38
Fusion of Exposure Images (gray scale) using Complex
EMD: Methodology
2 Two input gray scale images are converted into vectors by concatenating their rows, to
form a complex signal.
2 The complex signal is processed using the bivariate EMD algorithm, resulting in
multiple complex IMFs.
2 Scale images corresponding to each source are then combined locally using the local
fusion algorithm to give a fused image.
BIVARIATE EMD
Image A
Image B
B1
B2
BM
Fusion Based on Local
Variance
Fused Image
A1
A2
AM
(Gray-scale image fusion methodology using Bivariate EMD.)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 39
Fusion of Exposure Images (gray scale) using Complex
EMD: Results
(Input image 1) (Input image 2)
(Wavelet based image fusion) (EMD based image fusion)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 40
Fusion of Exposure Images (colored RGB) using
Complex EMD: Methodology
2 The three channels (red, green, and blue) of a color image are processed by separate
applications of bivariate EMD algorithm.
2 Three sets of complex-valued IMFs are obtained which correspond to the red, green,
and blue channels of the input images.
2 Each set is processed separately by the fusion algorithm to yield the fused red, green
and blue channel, which are combined to yield a fused RGB image.
a R
b R
a G a B
b G b B
BIVARIATE EMD
BIVARIATE EMD
BIVARIATE EMD
Image A
Image B
R1
R2
RL
G1
G2
GM
B1
B2
BN
Fusion based on Local Variance
Fusion based on Local Variance
Fusion based on Local Variance
R
G
B
Fused RGB
Image
b B
(RGB color image fusion methodology using Bivariate EMD.)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 41
Complex EMD-based Colored (RGB) Image Fusion:
Results
(Input image 1) (Input image 2)
(‘Local’ Image Fusion) (EMD based image fusion)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 42
Other Possibilities – “Enviromental Dimension”
Visual Thermal Image Fusion
From “Image Fusion and Enhancement via Empirical Mode Decomposition”, H. Hariharan
et al., Journal of Pattern Recognition Research, 2006
Here, the fusion was performed manually, without using any machinelearning or extensions of EMD.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 43
Trivariate EMD (TEMD): Underlying Idea
2 Designed to extend EMD to process trivariate signals.
2 Basic Idea: A trivariate signal is considered as a combination of a signal with fast 3D
rotating component superimposed on a slowly rotating component.
2 Local mean signal is considered as a slowly rotating component.
2 Huangs sifting algorithm is used to extract 3D rotating modes in a trivariate signal.
D. Looney and D. P. Mandic, IEEE Transactions on Signal Processing, 2009
c© D. P. Mandic The Third HHT Conference, Qingdao, China 44
TEMD: Local Mean Estimation
2 To calculate the local mean, multiple projections of the input signal are taken, with
each corresponding to a particular direction in 3D space.
2 The extrema of the projected signal are interpolated to yield quaternion-valued
envelopes, which are then averaged to yield an estimate of the local mean:
m(t) =1
KN
K∑
k=1
N∑
n=1
pφnθk
(1)
where pφnθk
denotes the projections along the direction, represented by {θk, φn}, in
3-dimensional space; K and N represent the total number of directions taken along
directions θ and φ respectively.
−200 −150 −100 −50 0 50 100 150 200 250−100
−50
0
50
100
150
200
X
Y
Original Dataenvelope 1envelope 2envelope 3envelope 4Mean
(Multiple envelopes along a multivariate signal.)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 45
TEMD: Choice of direction vectors
2 The choice of the set of direction vectors for generating multiple envelopes is a crucial
step.
2 The direction vectors are chosen along multiple longitudinal lines on a sphere, to
encompass the whole 3D space; a direction vector is represented by a point on the
sphere.
−1
0
1
−1
0
1 −1
0
1
XY
Z
O
A
(a) A direction vector in 3D space
c© D. P. Mandic The Third HHT Conference, Qingdao, China 46
TEMD: Tai-Chi data analysis
−100 0 100−100−50050−100
−50
0
50
XY
Z
0 100 200 300 400−100
0
100
Inpu
t
−50 0 50−50050−50
0
50
XY
Z
0 100 200 300 400−50
0
50
IMF1
−5 0 5−505
−2
0
2
XY
Z
0 100 200 300 400−5
0
5
IMF2
−5 0 5−10010−5
0
5
XY
Z
0 100 200 300 400−10
0
10
IMF3
−100 0 100−100−50050−100
−50
0
50
XY
Z
0 100 200 300 400−100
0
100
Time (Samples)
Resid
ue
(a) 3D plots of decomposed components (b) Decomposition of individual components of the input trivariate signal
c© D. P. Mandic The Third HHT Conference, Qingdao, China 47
Fusion of light exposure images (colored RGB) using
TEMD: Methodology
2 The three channels (red, green, and blue) of a color image are processed by separate
applications of TEMD algorithm.
2 Three sets of quaternion-valued IMFs are obtained which correspond to the red, green,
and blue channels of the input images.
2 Each set is processed separately by the fusion algorithm to yield the fused red, green
and blue channel, which are combined to yield a fused RGB image.
aR
bR
cR
aG
aB
bG
bB
cG
cB
TRIVARIATE
EMD
TRIVARIATE
EMD
TRIVARIATE
EMDImage A
Image B
Image
C
R1
R2
RL
G1
G2
GM
B1
B2
BN
Fusion based
on local
Variance
Fusion based
on local
Variance
Fusion based
on local
Variance
R
G
B
Fused
RGB
Image
(RGB color image fusion methodology using TEMD.)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 48
Multiple Color Image fusion using TEMD: Example
(Input Image 1) (Input Image 2) (Input Image 3)
(Fused Image)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 49
Multivariate EMD: Direction vectors on a 3D sphere
2 To generate a more uniform pointset in multidimensional spaces, the set of direction
vectors generated via Hammersley sequence is used.
2 Direction vectors for taking projections of trivariate signals on a sphere are shown for
(a) spherical coordinate system; (b) a low-discrepancy Hammersley sequence are
shown.
−1
0
1
−1
0
1−1
0
1
−1
0
1
−1
0
1−1
0
1
(a) (b)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 50
Multivariate EMD: Direction vectors on a 4D
hypersphere
Direction vectors on a hypersphere WXYZ generated by using low discrepancy
Hammersley sequence (a, b and c), and equiangular coordinate system (d, e and f), are
shown.‘
−1
0
1
−1
0
1−1
0
1
WX
Y
(a)−1
0
1
−1
0
1−1
0
1
XY
Z
(b)−1
0
1
−1
0
1−1
0
1
WY
Z
(c)
−1
0
1
−1
0
1−1
0
1
WX
Y
(d)−1
0
1
−1
0
1−1
0
1
WY
Z
(e)−1
0
1
−1
0
1−1
0
1
XY
Z
(f)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 51
Multivariate EMD: Illustration on a Hexavariate Signal
Original
signal
IMF1
IMF2
IMF3
IMF4
0 500 1000−5
0
5
U0 500 1000
−5
0
5
V
0 500 1000−5
0
5
W
0 500 1000−5
0
5
X
0 500 1000−5
0
5
Y
0 500 1000−5
0
5
Z
0 500 1000−2
0
2U 1
0 500 1000−2
0
2
V 1
0 500 1000−2
0
2
W1
0 500 1000−2
0
2
X 1
0 500 1000−5
0
5
Y 1
0 500 1000−5
0
5
Z 1
0 500 1000−2
0
2
U 2
0 500 1000−2
0
2V 2
0 500 1000−2
0
2
W2
0 500 1000−2
0
2
X 2
0 500 1000−2
0
2
Y 2
0 500 1000−2
0
2
Z 2
0 500 1000−2
0
2
U 3
0 500 1000−2
0
2
V 3
0 500 1000−2
0
2W
3
0 500 1000−2
0
2
X 3
0 500 1000−2
0
2
Y 3
0 500 1000−2
0
2
Z 3
0 500 1000−2
0
2
Time Index
U 4
0 500 1000−2
0
2
Time Index
V 4
0 500 1000−2
0
2
Time Index
W4
0 500 1000−2
0
2
Time Index
X 4
0 500 1000−5
0
5
Time Index
Y 4
0 500 1000−2
0
2
Time Index
Z 4
c© D. P. Mandic The Third HHT Conference, Qingdao, China 52
MEMD: Decomposition of a Hexavariate Tai Chi signal
The proposed algorithm is applied to the body motion data recorded in a Tai Chi
sequence. The data was captured using two inertial 3D MTx sensors attached to the left
hand and the left ankle of an athlete; these were combined to form a single hexavariate
signal.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 53
Rotational modes for 3D+3D bodysensor Tai-Chi data
−1000
100
−1000
100−200
0
200
X
Original (left hand)
YZ
−2000
200
050
100−200
0
200
X
Original (left ankle)
Y
Z
−40 −20 0 20 40
−500
50−100
0
100
X
IMF3
Y
Z
−500
50−50
050
−100
0
100
X
IMF3
Y
Z
−1000
100
−500
50−100
0
100
X
IMF4
Y
Z
−50 0 50−20
020
−100
0
100
X
IMF4
Y
Z
−200
20
−20−10
010
−40−20
02040
X
IMF5
Y
Z
−200
20−20
020
−20
0
20
X
IMF5
Y
Z
−80−60
−40−20
0
−200
2040
−500
50100
X
Residue
Y
Z
−100 −80 −60 −40
6065
7075
−50
0
50
X
Residue
Y
Z
c© D. P. Mandic The Third HHT Conference, Qingdao, China 54
MEMD as a Filter Bank
Spectra of IMFs (IMF1-IMF9) obtained for a single realization of an 8-channel white
Gaussian noise via MEMD (top) and the standard EMD (bottom). Overlapping of the
frequency bands corresponding to the same-index IMFs is more prominent in the case of
MEMD based filters.
100
101
102
Sp
ectr
um
Spectra of a single realization of white noise from MEMD
100
101
102
Frequency (Log)
Sp
ectr
um
Spectra of single realization of white noise from standard EMD
100
101
102
Sp
ectr
um
Averaged spectra of white noise realizations from MEMD
100
101
102
Frequency (log)
Sp
ectr
um
Averaged spectra of white noise realizations from standard EMD
All realisations Averaged spectra
c© D. P. Mandic The Third HHT Conference, Qingdao, China 55
Multivariate EMD: Noise-assisted MEMD to reduce
mode-mixing (a single channel case)
2 Extra noise channels are added as extra dimensions to the original signal.
2 The resulting multidimensional IMFs are aligned according to the filter bank structure
of MEMD reducing mode mixing within the signal IMFs.
(Standard EMD) (NA-MEMD: Signal channel with two extra noise channels)
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000−2
0
2
IMF
1
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000−2
0
2
IMF
2
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000−2
0
2
IMF
3
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000−2
0
2
Time Index
IMF
4−
en
d
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000−2
0
2
Orig
ina
l
0 5000 10000−2
0
2
Orig
inal
0 5000 10000−2
0
2
Noi
se
0 5000 10000−2
0
2
Noi
se
0 5000 10000−2
0
2
IMF
1−5
0 5000 10000−2
0
2
0 5000 10000−2
0
2
0 5000 10000−0.5
0
0.5
IMF6
0 5000 10000−2
0
2
0 5000 10000−2
0
2
0 5000 10000−2
0
2
0 5000 10000−2
0
2
0 5000 10000−2
0
2
Time Index
IMF8
0 5000 10000−2
0
2
Time Index0 5000 10000
−2
0
2
Time Index
0 5000 10000−0.5
0
0.5
IMF7
⇒ An alternative to EEMD without mixing signal and noise!
NA-MEMD= process your data channel and several noise channels with MEMD.
c© D. P. Mandic The Third HHT Conference, Qingdao, China 56
A 2-channel Case: NA-MEMD to reduce mode-mixing
Decomposition without noise channels Decomposition with two extra noise channels
0 700 1400−2
0
2
Orig
ina
l
0 700 1400−2
0
2
0 700 1400−2
0
2
IMF
1
0 700 1400−2
0
2
0 700 1400−2
0
2
IMF
2
0 700 1400−2
0
2
0 700 1400−2
0
2
IMF
3
0 700 1400−2
0
2
0 700 1400−2
0
2
Time Index
IMF
4
0 700 1400−2
0
2
Time Index
700 1400−2
0
2
Orig
ina
l
700 1400−2
0
2
700 1400−2
0
2
IMF
3
700 1400−2
0
2
700 1400−2
0
2
IMF
4
700 1400−2
0
2
700 1400−2
0
2
Time Index
IMF
5
700 1400−2
0
2
Time Index
◦ The exta noise channels help
◦ So with NA-EMD we have a unified framework for the decomposition of both
single-channel data and M-channel data
c© D. P. Mandic The Third HHT Conference, Qingdao, China 57
Operation of NA-MEMD and EEMD
Ensemble EMD (EEMD)
EMD
respect.
IMFs
all
over
Average
N_i − WGN realisation IMFs of EEMD
S − useful signal
IMF_3
IMF_2
IMF_m
IMF_1
IMF_nr
IMF_n1
IMF_2q
IMF_21
IMF_1p
IMF_11
S+N_n
S+N_2
S+N_1
......
......
...
EMD
EMD
Noise-assisted MEMD (NA-MEMD)Signal
IMF space
m x (n+1) dimensionalEMD
multivariate
using
Process
Noise_n
Noise_1
In NA-MEMD signal is
NOT added to noise,
but instead the signal
channel and noise channels
are processed in a
multidimensional space
using MEMD. This way
we guarantee the same
number of IMFs and the
same frequency contents
at every level of such
multidimensional IMFs.
Usually 2-4 noise channels
are sufficient for good
decomposition (see the
simulations)
c© D. P. Mandic The Third HHT Conference, Qingdao, China 58
NA-MEMD vs Ensemble EMD
Ensemble EMD (EEMD)
◦ Performs EMD over an ensembleof signal plus white noise, andaverages the ensemble IMFs
◦ Uses the filter bank property ofEMD on white noise by populatingthe time-frequency space
◦ This way it reduces mode mixing
◦ EMD is applied separately overan ensemble of signal plus noise
◦ ⇒ each realization may havedifferent number of modes (IMFs)and output contains residual noise
◦ Only valid for univariatesignals
Noise–Assisted MEMD
◦ Makes use of the filter bankproperty of MEMD on white noiseto populate entire T–F space
◦ Unlike EEMD, it does notdirectly interfere with the originalsignal (as noise is added toseparate channels)
◦ ⇒ the output contains minimalresidual noise due to leakagewhich is negligible
◦ Since a single MEMD is appliedto the multivariate signal → samenumber of modes (IMFs)
◦ Is valid for both univariateand multivariate signals
c© D. P. Mandic The Third HHT Conference, Qingdao, China 59
NA–MEMD vs EEMD: Some Examples
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
XDecomposition of X via EEMD
0 200 400 600 800 1000 1200 1400 1600 1800 2000−0.5
0
0.5
IMF1
0 200 400 600 800 1000 1200 1400 1600 1800 2000−1
0
1
IMF2
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
IMF3
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
IMF4
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
Time Index
IMF5
:end
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
X
Decomposition of X via N−A MEMD
0 200 400 600 800 1000 1200 1400 1600 1800 2000−0.1
0
0.1
IMF1
0 200 400 600 800 1000 1200 1400 1600 1800 2000−1
0
1
IMF2
0 200 400 600 800 1000 1200 1400 1600 1800 2000−0.5
0
0.5
IMF3
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
IMF4
0 200 400 600 800 1000 1200 1400 1600 1800 2000−2
0
2
Time Index
IMF5
:end
0 500 1000 1500 20000
0.002
0.004
0.006
0.008
0.01
0.012
0.014
|e(t
)|
Time Index
Error Function (e(t)) for EEMD
0 500 1000 1500 20000
1
2
3
4
5
6x 10
−16
|e(t
)|
Time Index
Error function (e(t)) for N−A MEMD
EEMD for 500 realisations of WGN NA-MEMD with two noise channels
c© D. P. Mandic The Third HHT Conference, Qingdao, China 60
Sensitivity to Noise Power
Ensemble EMD Noise Assisted MEMD
0 0.2 0.4 0.6 0.8 10
0.5
1
1.5
2
2.5
3
3.5
4x 10
−3
PWGN
/Psignal
Pe
(t)
Average Power of Error vs Noise Power for EEMD
0 0.2 0.4 0.6 0.8 10.8
1
1.2
1.4
1.6
1.8
2x 10
−32
PWGN
/Psignal
Pe
(t)
Average Power of Error vs Noise Power for NA−MEMD
EEMD:
◦ The decomposition error increases
with an increase in noise power
◦ This is because noise is added
to the signal and interferes with the
decomposition
NA-MEMD:
◦ The error does not increase with an
increase in noise power
◦ This independence of noise power is
because the noise is in separate channels
and does not interfere with the signal
c© D. P. Mandic The Third HHT Conference, Qingdao, China 61
BCI Application: Estimating cleaned EEG using MEMD
Filter bank structure of MEMD helps to clean the EEG signal by separating the brain
electrical activity from unwanted artefacts, such as the electrooculogram (EOG) and
electromyogram (EMG).
500 1000 1500−100
0
100Fp
1Recorded EEG
500 1000 1500−100
0
100Estimated EOG artefact
500 1000 1500−50
0
50
Cleaned EEG
500 1000 1500−100
0
100
Fp2
500 1000 1500−100
0
100
500 1000 1500−50
0
50
500 1000 1500−50
0
50
C3
500 1000 1500−40
−20
0
20
500 1000 1500−50
0
50
500 1000 1500−100
0
100
samples
C4
500 1000 1500−100
0
100
samples500 1000 1500
−50
0
50
samples
c© D. P. Mandic The Third HHT Conference, Qingdao, China 62
Imaginary Motion BCI: Mu Rhythm (8-12Hz)
• Occupies in the alpha range (8-12Hz)
• Strongly suppressed during the performance of contralateral motor acts
• Reflects the electrical output of the synchronisation of large portions of pyramidal
neurons of the motor cortex which control hand and arm movement
• Active movements/observed actions/imagined actions
(a)
(b)
(c)
(a) power of mu band on scalp during motor imagery(b) average time-frequency representation
(c) a typical single trial showing imagery-related modulation
c© D. P. Mandic The Third HHT Conference, Qingdao, China 63
Spectrogram Comparison
2 Motor imagery : A dynamic state during which an individual mentally simulates a
given action
2 Experiment : A subject imagined herself/himself moving left arm, right arm or foot
2 Event-related synchronization (ERS) over ipsilateral area during the motor imagery
task can be expected around 10Hz (mu rhythm)
Right hemisphere (C4+C6) during the imagination of right arm movement
time (s)
fre
qu
en
cy (
Hz)
0.5 1 1.5 2 2.5 30
5
10
15
20
25
30
time (s)
fre
qu
en
cy (
Hz)
0 0.5 1 1.5 2 2.5 30
5
10
15
20
25
30
time (s)
fre
qu
en
cy (
Hz)
0 0.5 1 1.5 2 2.5 30
5
10
15
20
25
30
time (s)
fre
qu
en
cy (
Hz)
0 0.5 1 1.5 2 2.5 30
5
10
15
20
25
30
(a) STFT (b) EMD (c) EEMD (d) MEMD
c© D. P. Mandic The Third HHT Conference, Qingdao, China 64
Classification Performances
Subject Algorithm IMFs Classification rate [%]
DFT(8-30Hz) 81.057 ± 4.291A EMD 1-3 65.710 ± 5.426
EEMD 1-3 79.727 ± 4.492MEMD 2-3 84.747 ± 4.762
DFT(8-30Hz) 57.370 ± 5.564B EMD 1-4 60.607 ± 5.882
EEMD 1-2 67.387 ± 5.824MEMD 2-3 70.167 ± 5.363
DFT(8-30Hz) 64.860 ± 6.020C EMD 1-4 60.653 ± 6.020
EEMD 1-3 75.250 ± 5.117MEMD 2-3 78.483 ± 4.362
DFT(8-30Hz) 86.700 ± 4.081D EMD 1-3 79.470 ± 5.447
EEMD 2 87.433 ± 3.917MEMD 2-3 89.133 ± 3.235
2 200 trials of motor imagery for left
hand, right hand and foot movements
2 Channels : ‘FC3’, ‘FC4’, ‘Cz’, ‘C3’,
‘C4’, ‘C5’, ‘C6’, ‘T7’, ‘T8’, ‘CCP3’,
‘CCP4’
2 Common spatial patterns (CSP) were
used to extract features
2 Average classification rates of 100
repetitions while mixing sample order
(Support vector machine was used)
2 MEMD produced the best
classification results using 2nd
and 3rd IMFs for all subjects
c© D. P. Mandic The Third HHT Conference, Qingdao, China 65
Multivariate extensions of EMD
2 For robust modelling, it is crucial that similar oscillatory modes frommultiple channels are aligned
2 Extensions of EMD for multivariate time series:– Original EMD algorithms [Huang et al, Proc. Roy. Soc. A, 1998]– Complex EMD [Tanaka and Mandic, IEEE SPL, 2007]– Rotation Invariant EMD [Mandic et al., Proc ICASSP, 2007]– Bivariate EMD [Rilling, Flandrin, Goncalves, IEEE SPL, 2007]– Trivariate EMD [Rehman and Mandic, IEEE TSP, 2009]– Quadrivariate EMD [Rehman and Mandic, Proc. IJCNN, 2010]– Multivariability via ’dendrograms’ [Rutkowski et al, Jour. Circ., Sys.,
and Comp, 2010]– Multivariate EMD [Rehman and Mandic, Proc. Roy. Soc. A, 2010]– Filterbank property of Multivariate EMD and Noise-Assisted MEMD
[Rehman and Mandic, IEEE Tr. Sig. Proc., 2011]
◦ Matlab code at www.commsp.ee.ic.ac.uk/∼mandic
c© D. P. Mandic The Third HHT Conference, Qingdao, China 66
Conclusions
2 EMD is non-parametric and self adaptive which is advantageous whendecomposing real world data into its natural frequency modes
2 It is a powerful tool for the purposes of “data fusion via fission”
2 Multivariate extensions of EMD have been proposed which takemultiple projections of a signal by sampling hyperspheres usingequi-angular coordinate system and low discrepancy quasi-Monte Carlobased Hammersley sequences
2 The proposed method extracts common rotational modes across thesignal components, making it suitable for e.g. fusion of informationfrom multiple sources, and follows a filter bank structure
2 Making use of the quasi-dyadic filter bank property of multivariateextension of EMD, noise-assisted MEMD method (NA-MEMD) hasbeen presented which has been shown to reduce mode mixing
2 NA-EMD is a viable alternative to EEMD
c© D. P. Mandic The Third HHT Conference, Qingdao, China 67
Thank you
十分感谢!
c© D. P. Mandic The Third HHT Conference, Qingdao, China 68