on the trend, detrend and the variability of nonlinear and nonstationary time series a new...
TRANSCRIPT
On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series
A new application of HHT
Satellite Altimeter Data : Greenland
Two Sets of Data
The State-of-the-Arts
“One economist’s trend is another economist’s cycle” Engle, R. F. and Granger, C. W. J. 1991 Long-run Economic Relationships.
Cambridge University Press.
• Simple trend – straight line
• Stochastic trend – straight line for each quarter
Philosophical Problem
名不正則言不順
言不順則事不成
——孔夫子
On Definition
Without a proper definition, logic discourse would be impossible.
Without logic discourse, nothing can be accomplished.
Confucius
Definition of the Trend
Within the given data span, the trend is an intrinsically fitted monotonic function, or a function in which there can be at most one extremum.
The trend should be determined by the same mechanisms that generate the data; it should be an intrinsic and local property.
Being intrinsic, the method for defining the trend has to be adaptive. The results should be intrinsic (objective); all traditional trend determination methods give extrinsic (subjective) results.
Being local, it has to associate with a local length scale, and be valid only within that length span as a part of a full wave cycle.
Definition of Detrend and Variability
Within the given data span, detrend is an operation to remove the trend.
Within the given data span, the Variability is the residue of the data after the removal of the trend.
As the trend should be intrinsic and local properties of the data; Detrend and Variability are also local properties.
All traditional trend determination methods are extrinsic and/or subjective.
The Need for HHT
HHT is an adaptive (local, intrinsic, and objective) method to find the intrinsic local properties of the given data set, therefore, it is ideal for defining the trend and variability.
History of HHT1998: The Empirical Mode Decomposition Method and the Hilbert Spectrum for
Non-stationary Time Series Analysis, Proc. Roy. Soc. London, A454, 903-995. The invention of the basic method of EMD, and Hilbert transform for determining the Instantaneous Frequency and energy.
1999: A New View of Nonlinear Water Waves – The Hilbert Spectrum, Ann. Rev. Fluid Mech. 31, 417-457.
Introduction of the intermittence in EMD.
2003: A confidence Limit for the Empirical mode decomposition and the Hilbert spectral analysis, Proc. of Roy. Soc. London, A459, 2317-2345.
Establishment of a confidence limit without the ergodic assumption.
2004: A Study of the Characteristics of White Noise Using the Empirical Mode Decomposition Method, Proc. Roy. Soc. London, (in press)
Defined statistical significance and predictability of IMFs.
2004: On the Instantaneous Frequency, Proc. Roy. Soc. London, (Under review)
Removal of the limitations posted by Bedrosian and Nuttall theorems for instantaneous Frequency computations.
Two Sets of Data
Global Temperature Anomaly
Annual Data from 1856 to 2003
Global Temperature Anomaly 1856 to 2003
IMF Mean of 10 Sifts : CC(1000, I)
Mean IMF
STD IMF
Statistical Significance Test
Data and Trend C6
Data and Overall Trends : EMD and Linear
Rate of Change Overall Trends : EMD and Linear
Variability with Respect to Overall trend
Data and Trend C5:6
Data and Trends: C5:6
Rate of Change Trend C5:6
Trend Period C5
Variability with Respect to 65-Year trend
Data and Trend C4:6
Data and Trend C4:6
Rate of Change Trend C4:6
Trend Period C4
Variability with Respect to 20-Year trend
Data and Trend C3:6
Trend Period C3
Histogram of Trend Period C3
Variability with Respect to 10-Year trend
Hilbert Spectrum Global Temperature Anomaly
Marginal Hilbert Spectrum
Morlet Wavelet Spectrum
Hilbert and Morlet Wavelet Spectra
Financial Data : NasDaqSC
October 11, 1984 – December 29, 2000
October 12, 2004
NasDaq Data
NasDaq IMF
NasDaq IMF Reconstruction : A
NasDaq IMF Reconstruction : B
NasDaq Various Overall Trends
NasDaq various Overall Detrends
Mean : L = 0 Exp = 73.1187 EMD = 0.3588
STD : L = 559.09 Exp = 426.66 EMD = 238.10
NasDaq Trend IMF (C8-C9)
NasDaq Local Period for Trend IMF (C8-C9)mean = 796.6
NasDaq Trend IMF (C7-C9)
NasDaq Local Period for Trend IMF (C7-C9)Mean = 425.7
NasDaq Trend IMF (C6-C9)
NasDaq Local Period for Trend IMF (C6-C9)Mean = 196.5
NasDaq Traditional Moving Mean Trends: Details
NasDaq Trends: Moving Mean and EMD : Details
NasDaq Period of EMD Trend (C4)Mean = 35.56
NasDaq Distribution of Period for EMD Trend (C4)
NasDaq Detrended Data (C4-C9)
NasDaq Detrended Data (C4-C9) : Details
NasDaq Histogram Detrended Data (C1-C3)
Various Definitions of Variability
• Variability defined by percentage Gain is the absolute value of the Gain.
• Variability defined by daily high-low is the percentage of absolute value of High-Low.
• Variability defined by Empirical Mode Decomposition is the percentage of the absolute value of the sum from selected IMFs.
• Financial data do not look like ARIMA.
NasDaq Variability defined by EMD : C1
NasDaq Variability defined by Gain
NasDaq Variability defined by Daily High-Low
NasDaq Period of Variability defined by EMD : C1Mean = 8.38
NasDaq Histogram Period of EMD Variability : C1
NASDAQ Price gradient vs. Gain Variability
NASDAQ Price gradient vs. High-Low Variability
NASDAQ Price gradient vs. EMD Variability
Relationship between Variability: Gain vs. EMD
Relationship between Variability: Gain vs. High-Low
Relationship between Variability: EMD vs. High-Low
Statistical Significance
Test for IMF
Methodology
• Based on observations from Monte Carlo numerical experiments on 1 million white noise data points.
• All IMF generated by 10 siftings.• Fourier spectra based on 200 realizations of
4,000 data points sections.• Probability density based on 50,000 data
points data sections.
IMF Period Statistics
IMF1 2 3 4 5 6 7 8 9
number of peaks
347042 168176 83456 41632 20877 10471 5290 2658 1348
Mean period 2.881 5.946 11.98 24.02 47.90 95.50 189.0 376.2 741.8
period in year 0.240 0.496 0.998 2.000 3.992 7.958 15.75 31.35 61.75
Fourier Spectra of IMFs
0 1 2 3 4 5 6 7 8 90
0.5
1
1.5
spectr
um
(10**
-3)
Fourier Spectra of IMFs
1 1.5 2 2.5 3 3.50
0.2
0.4
0.6
0.8
1
ln T
spectr
um
(10**
-3)
Shifted Fourier Spectra of IMFs
Empirical Observations : INormalized spectral area is constant
constTdS nT ln,ln
Empirical Observations : IIComputation of mean period
n
nT
nTnTnn T
TdS
T
TdS
T
dTSdSNE
lnln ,ln
,ln2,,
T
TdS
TdST
nT
nT
n ln
ln
,ln
,ln
Empirical Observations : IIIThe product of the mean energy and period is
constant
constTE nn
constTE nn lnln
Monte Carlo Result : IMF Energy vs. Period
Empirical Observation: Histograms IMFs By Central Limit theory IMF should be normally distributed.
-1 0 10
5000
-1 -0.5 0 0.5 10
5000
-0.5 0 0.50
5000
-0.5 0 0.50
5000
-0.4 -0.2 0 0.2 0.40
5000
-0.2 0 0.20
5000
-0.2 -0.1 0 0.1 0.20
5000
-0.1 0 0.10
5000
mode 2 mode 3
mode 4 mode 5
mode 6 mode 7
mode 8 mode 9
Histograms : IMF Energy Density
0.15 0.2 0.250
100
200
0.05 0.1 0.150
100
200
0.02 0.04 0.06 0.080
100
200
0.01 0.02 0.03 0.04 0.050
100
200
0 0.01 0.02 0.030
100
200
0 0.01 0.020
100
200
0 0.005 0.010
100
200
0 0.005 0.010
100
200
300
mode 2 mode 3
mode 4 mode 5
mode 6 mode 7
mode 8 mode 9
By Central Limit theory, IMF should be normally distributed; therefore, its energy should be Chi-squared distributed.
Chi-Squared Energy Density Distributions
212)( nn NEENnn eNENE
By Central Limit theory, IMF should be normally distributed; therefore, its energy should be Chi-squared distributed.
Formula of Confidence Limit for IMF Distributions
Ey ln yeE
Introduce new variable y:
Then,
!3!21
2exp
32 yyyyy
ENCy
Confidence Limit for IMF Distributions
Data and IMFs SOI
1930 1940 1950 1960 1970 1980 1990 2000
-0.4-0.2
00.2
R
-0.5
0
0.5
C9
-0.5
0
0.5
C8
-10
1
C7
-10
1
C6
-10
1
C5
-2
0
2
C4
-2
0
2
C3
-20
2
C2
-20
2
C1
-50
5
Raw
SO
I
Statistical Significance for SOI IMFs
1 mon 1 yr 10 yr 100 yr
IMF 4, 5, 6 and 7 are 99% statistical significance signals.
Summary
• Not all IMF have the same statistical significance.
• Based on the white noise study, we have established a method to determine the statistical significant components.
• References:• Wu, Zhaohua and N. E. Huang, 2003: A Study of the
Characteristics of White Noise Using the Empirical Mode Decomposition Method, Proceedings of the Royal Society of London (in press)
• Flandrin, P., G. Rilling, and P. Gonçalvès, 2003: Empirical Mode Decomposition as a Filterbank, IEEE Signal Processing, (in press).
Statistical Significance Test
Only the statistical Significant IMF components are signal above noise; therefore, they might be predictable.
Statistical Significance Test : Gain
Statistical Significance Test : High-Low
Statistical Significance Test : EMD
Statistical Significance Test : All Variability Definitions
The Sum of all the Statistical Significance IMFs
Relationship among Trends: Gain vs. EMD
Relationship among Trends: Gain vs. High-Low
Relationship among Trends: EMD vs. High-Low
Summary
• A working definition for the trend is established; it is a function of the local time scale.
• Need adaptive method to analysis nonstationary and nonlinear data for trend and variability
• Various definitions for variability should be compared in details to determine their significance.