exploring the feature space of large collections of time series
TRANSCRIPT
Exploring the feature space of large time series collections 1
Exploring thefeature space oflarge collectionsof time series
Rob J Hyndman
Exploring the feature space of large time series collections 1
Exploring thefeature space oflarge collectionsof time series
Rob J Hyndman
with Earo Wang, Nikolay LaptevYanfei Kang, Kate Smith-Miles
Outline
1 M3 competition data
2 Yahoo web traffic
3 What next?
Exploring the feature space of large time series collections M3 competition data 2
M3 forecasting competition
Exploring the feature space of large time series collections M3 competition data 3
M3 forecasting competition
Exploring the feature space of large time series collections M3 competition data 3
M3 forecasting competition
“The M3-Competition is a final attempt by the authors tosettle the accuracy issue of various time series methods. . .The extension involves the inclusion of more methods/researchers (in particular in the areas of neural networksand expert systems) and more series.”
Makridakis & Hibon, IJF 2000
3003 series
All data from business, demography, finance andeconomics.
Series length between 14 and 126.
Either non-seasonal, monthly or quarterly.
All time series positive.
Exploring the feature space of large time series collections M3 competition data 4
M3 forecasting competition
Exploring the feature space of large time series collections M3 competition data 5
●
Key idea
Examples for time series
lag correlationsize and direction of trendstrength of seasonalitytiming of peak seasonalityspectral entropy
Called “features” or “characteristics” in themachine learning literature.
Exploring the feature space of large time series collections M3 competition data 6
John W Tukey
Cognostics
Computer-produced diagnostics(Tukey and Tukey, 1985).
Key idea
Examples for time series
lag correlationsize and direction of trend
strength of seasonalitytiming of peak seasonalityspectral entropy
Called “features” or “characteristics” in themachine learning literature.
Exploring the feature space of large time series collections M3 competition data 6
John W Tukey
Cognostics
Computer-produced diagnostics(Tukey and Tukey, 1985).
Key idea
Examples for time series
lag correlationsize and direction of trend
strength of seasonalitytiming of peak seasonalityspectral entropy
Called “features” or “characteristics” in themachine learning literature.
Exploring the feature space of large time series collections M3 competition data 6
John W Tukey
Cognostics
Computer-produced diagnostics(Tukey and Tukey, 1985).
An STL decompositionQuarterly visitor nights: Mornington Peninsula
Yt = St + Tt + Rt St is periodic with mean 0
5.0
6.0
7.0
data
−0.
50.
5
seas
onal
5.8
6.1
6.4
tren
d
−0.
40.
0
2000 2005 2010
rem
aind
er
timeExploring the feature space of large time series collections M3 competition data 7
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Candidate features
STL decompositionYt = St + Tt + Rt
Seasonal period
Strength of seasonality: 1− Var(Rt)Var(Yt−Tt)
Strength of trend: 1− Var(Rt)Var(Yt−St)
Spectral entropy: H = −∫ π−π fy(λ) log fy(λ)dλ,
where fy(λ) is spectral density of Yt.Low values of H suggest a time series that iseasier to forecast (more signal).
Autocorrelations: r1, r2, r3, . . .
Optimal Box-Cox transformation parameter λExploring the feature space of large time series collections M3 competition data 8
Dimension reduction for time series
Exploring the feature space of large time series collections M3 competition data 9●
Dimension reduction for time series
Exploring the feature space of large time series collections M3 competition data 9●
SpecEntr
0.0 0.4 0.8 2 6 10 0.0 0.4 0.8
0.5
0.9
0.0
0.6
Trend
Season
0.0
0.6
28 Freq
ACF
−0.
40.
6
0.5 0.7 0.9
0.0
0.6
0.0 0.4 0.8 −0.4 0.2 0.8
Lambda
Featurecalculation
Dimension reduction for time series
Exploring the feature space of large time series collections M3 competition data 9●
SpecEntr
0.0 0.4 0.8 2 6 10 0.0 0.4 0.8
0.5
0.9
0.0
0.6
Trend
Season
0.0
0.6
28 Freq
ACF
−0.
40.
6
0.5 0.7 0.9
0.0
0.6
0.0 0.4 0.8 −0.4 0.2 0.8
Lambda
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●● ●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
Featurecalculation
Principalcomponentdecomposition
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●● ●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
First two PCs explain 68% of variation.
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●●●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●●●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●●●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
●●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●●●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●●●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
●●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
3
6
9
12value
Freq
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●●●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●●●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●●●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●●●
●
●
●● ●●
●●●
●●
●●●●●●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
●●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●●●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●●●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
●●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
0.00
0.25
0.50
0.75
value
Season
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●●●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●●●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●●●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●●●
●
●
●● ●●
●●●
●●
●●●●●●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
●●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●●●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●●●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
●●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
0.25
0.50
0.75
value
Trend
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●●●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●●●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●●●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●●●
●
●
●● ●●
●●●
●●
●●●●●●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
●●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●●●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●●●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
●●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
0.0
0.5
value
ACF
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●●●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●●●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●●●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●●●
●
●
●● ●●
●●●
●●
●●●●●●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
●●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●●●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●●●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
●●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
0.50.60.70.80.9
value
SpecEntr
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 10
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●●●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●● ●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
● ●●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●●
●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●●●●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●●●●●● ●
● ●●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●●●
●
●
●● ●●
●●●
●●
●●●●●●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
●●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●●
●●
●
● ●●
●
●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●●●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●●●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
● ●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
● ●
● ●
●●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●●
●●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
●●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●●●
●●●
●●●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
●●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−3
−2
−1
0
1
2
3
−2 0 2 4PC1
PC
2
0.000.250.500.751.00
value
Lambda
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 11
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●●
●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
●●
●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●
●●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●
●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●
●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●
●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
●●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●
●
●
●●
●
●
●
●
●
● ●
● ●
●
●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●
●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●
●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−2 0 2 4
−3
−2
−1
01
23
PC1
PC
2
●
●
●
●
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 11
N16
27
1990 1991 1992 1993 1994
N06
36
1920 1925 1930 1935 1940 1945
N29
81
0 10 20 30 40 50 60
N26
99
1984 1986 1988 1990 1992
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 11
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●●
●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
●●
●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●
●●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●
●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●
●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●
●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
●●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●
●
●
●●
●
●
●
●
●
● ●
● ●
●
●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●
●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●
●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−2 0 2 4
−3
−2
−1
01
23
PC1
PC
2
Yahoo log returns series
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 11
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●●
●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
●●
●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●
●●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●
●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●
●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●
●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
●●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●
●
●
●●
●
●
●
●
●
● ●
● ●
●
●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●
●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●
●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−2 0 2 4
−3
−2
−1
01
23
PC1
PC
2
●Yahoo log returns series
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 12
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●●
●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
●●
●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●
●●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●
●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●
●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●
●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
●●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●
●
●
●●
●
●
●
●
●
● ●
● ●
●
●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●
●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●
●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−2 0 2 4
−3
−2
−1
01
23
PC1
PC
2
White noise series
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 13
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●●
●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
●●
●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●
●●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●
●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●
●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●
●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
●●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●
●
●
●●
●
●
●
●
●
● ●
● ●
●
●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●
●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●
●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−2 0 2 4
−3
−2
−1
01
23
PC1
PC
2
Random walk series
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 14
● ●●
●
●
●●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●●●
●
●●●
●
●●● ●
●●● ●
●● ●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●●
●
●
●●
●
●●●
●●●
● ●
●
●
●●
●
●
●
●●●
●●
●●
●
●●
●●
●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●●● ●●
●
●
●●●
●●●●●●●
●●●
●
●
●●
●●●
●●●●
●●●●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
● ●
●●●●●
●● ●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
●●
●
●●
●●
●
●●●●●●●●
●
●●●●●
●●●●●
●●●●●●●●●●
●
●
●● ●
●●●
●
●●● ●
●● ●
●●
●●●
●●●●
●●●●●
● ●●●●●●●
●
●●
●●●
●● ●●●●
●
●
●●●
●●
●
●
●●●● ●
●●
●
●●●
●●●
●
●●●●●●●●
●●●
●
●
●
●
●
● ●●
●
●●●
●●●●
●
●●●
●●●
●
●●
●
●
●●
●
●
●●●●●
●●
●●
●
●
●●●
● ●●
●
●●
●●
●
● ●
●●
●●
●
●
●●●●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●●●
●●
●
●●●
●
●●●
●●●●
●●
●
●●●
●
● ●●●●
●●
●
●●●●●●
●●●●
● ●●
●●
●●
●●
●●
●●
●●● ●
●●
●●●●
●●
●
●
●
●●●
●
●●●● ●●
●●
●
●●
●
●
●●
●
●●
●●●
● ●●
●
●
●●●
●
●●
●●● ●
●
●●● ●●●
●
●●
●
●●
●●●
●
●●
●
●
●●● ●●
●
●●
●●
●●
●●●●●●
●●●
●
●●
●
●●●●●●●●●
●
●
●
●●
●
●
●
●●
●
●
● ● ●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
● ●●
●
● ●
●
● ●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●●
●
●
●● ●●●● ●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●●
●
●
●●● ●
●● ●
●●
●
●
●● ●●
●●●
●●
●●●●● ●
●●●
●
●●
●
●●
●
●●●●
●●●
●
●
● ●●●●●●
●
●●
●●● ●●●●
●● ●●●
●
●●●
●
●
●● ●●●●● ●
●●
●
●●●●
●●
●
●● ●
●
● ●
●
●
●●
●●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●●●
●
●●●
●
●●●
●
●●● ●●
●●
●●●
●●●
●●●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●●
● ●
●
●
●
●●
●
●● ●●
●
●
●●●
●
●
●●●●
●
●
●
●
●
●
●
●●●
●●●●●
●
●● ●●
●●
●●●
●●
●
●
●● ●
●●
●●●
●
●
●
●
●●●
●
●● ●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●●
●●
●● ●
●●
●
●
●
●●
●
●●
●●
●● ●
●●●
●
●
●●
●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
● ●●
●
●
●●
●●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●
●●
●●
●●
●
● ●● ●
●●
●● ●●
●●
●
●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
● ●●●
●
●
●●
●
●
●
● ●● ●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●●
●
●
●●●
●
● ●
● ●●
● ●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
● ●●
●●
●
● ●●
●
●●
●
● ●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●●
●
●
● ●●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●
● ●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
● ●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
● ●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
●
●● ●●●●
●●
● ●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
●●
● ●
● ●
●
●●
●●
●●
●●
●●
●
●●
●
●
●
● ●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●● ●●
● ●●
●
●
●
●
●●●●
●
●
●●
●
●●
●
●●
●●●●●
●
●
●●●● ●
●
●●●● ●
●
●
●
●
●
● ●●●●●
●●
●
●
●●
●●
●●
●
●●
●
●●
●●
●
●
● ●
●
●● ●
●
●
●
●●●
●
●
●
●
●● ●●● ●●●● ●●●●
●
●●
●●●●●●
●
●●●●
●●
●●
● ●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●
●●●● ●
●
●●
●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●●●●
●
●●
● ●
●●
●
●●
●●●
●●
●
●●●
●●
● ●●
●●●
●
●
●
●
●●
● ●●
●●●
●
●
●●●
●●
●
●
●●
● ●●
●
●
●●
●
●●
●●●
●●
●
●
●
●●
●
●●●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●●● ●●
● ●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●●
●●●
● ●
●
●●
●
●●●●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●●
●
●
●●
●
●
●
●
●
● ●
● ●
●
●
●
●●
●●
●●●
●●
●● ● ●●●
●●
●●●●
●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●●●
●●
●
●●●
●●
●●
●●●●●
●
●
●
●
●
● ●
●●●●●●
●●
●
●
●●●● ●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
● ●● ●●
●
●●●●
●●
●
●
●
●●●
●
●
●
● ●●
●●
●
●●
● ●● ●●●
●●
●●●
● ●●●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●●
●
●●
●
●●●
●
●●●●
●
●
●
●
●●●
●●●
●
●
●
●●
● ●●
●
●● ●● ● ●
●
●● ●●
●●●
●
●
●
●
●●
●●
●●●
●●●
●
●
●
●●
● ●●●●●●●●●
●
●●●●●●●●●●●●●●●●●
● ●●
●●
●
●
●●●●●
●●●
●
●
●●●●●●
●
●
●●●●
●●
●
●●
●
●
●●
●
●●
●●●●●●
●●●●●
●
●●
●●●●●●●
●●●
●
●●
●
●●
●●●
●●
−2 0 2 4
−3
−2
−1
01
23
PC1
PC
2
●
●
Feature space of M3 data
Exploring the feature space of large time series collections M3 competition data 14
Time
5 10 15
Outline
1 M3 competition data
2 Yahoo web traffic
3 What next?
Exploring the feature space of large time series collections Yahoo web traffic 15
Yahoo web-trafficTens of thousands of time series collected atone-hour intervals over one month.Consisting of several server metrics (e.g. CPU usageand paging views) from many server farms globally.Aim: find unusual (anomalous) time series.
Exploring the feature space of large time series collections Yahoo web traffic 16
Yahoo web-traffic
3
6
9
10
20
30
40
1020304050
1
2
3
4
25
50
75
100
bu
sy2
33
bu
sy2
71
bu
sy5
0bu
sy2
00
bu
sy3
69
20
14
−1
1−
09
20
14
−1
1−
10
20
14
−1
1−
11
20
14
−1
1−
12
20
14
−1
1−
13
20
14
−1
1−
14
20
14
−1
1−
15
20
14
−1
1−
16
20
14
−1
1−
17
20
14
−1
1−
18
20
14
−1
1−
19
20
14
−1
1−
20
20
14
−1
1−
21
20
14
−1
1−
22
20
14
−1
1−
23
20
14
−1
1−
24
20
14
−1
1−
25
20
14
−1
1−
26
20
14
−1
1−
27
20
14
−1
1−
28
20
14
−1
1−
29
20
14
−1
1−
30
20
14
−1
2−
01
20
14
−1
2−
02
20
14
−1
2−
03
20
14
−1
2−
04
20
14
−1
2−
05
20
14
−1
2−
06
20
14
−1
2−
07
20
14
−1
2−
08
20
14
−1
2−
09
20
14
−1
2−
10
20
14
−1
2−
11
20
14
−1
2−
12
date
va
lue
25
30
35
40
45
20
25
30
35
40
50
60
70
10
15
20
25
50
60
me
mo
ry4
60
me
mo
ry4
29
me
mo
ry1
47
me
mo
ry4
13
me
mo
ry4
84
20
14
−1
1−
09
20
14
−1
1−
10
20
14
−1
1−
11
20
14
−1
1−
12
20
14
−1
1−
13
20
14
−1
1−
14
20
14
−1
1−
15
20
14
−1
1−
16
20
14
−1
1−
17
20
14
−1
1−
18
20
14
−1
1−
19
20
14
−1
1−
20
20
14
−1
1−
21
20
14
−1
1−
22
20
14
−1
1−
23
20
14
−1
1−
24
20
14
−1
1−
25
20
14
−1
1−
26
20
14
−1
1−
27
20
14
−1
1−
28
20
14
−1
1−
29
20
14
−1
1−
30
20
14
−1
2−
01
20
14
−1
2−
02
20
14
−1
2−
03
20
14
−1
2−
04
20
14
−1
2−
05
20
14
−1
2−
06
20
14
−1
2−
07
20
14
−1
2−
08
20
14
−1
2−
09
20
14
−1
2−
10
20
14
−1
2−
11
20
14
−1
2−
12
date
va
lue
0
5000
10000
15000
20000
200
400
600
0
5000
10000
15000
20000
500
1000
0
5000
10000
15000
20000
25000
pa
gin
g5
3p
ag
ing
46
7p
ag
ing
37
1p
ag
ing
33
7p
ag
ing
36
7
20
14
−1
1−
09
20
14
−1
1−
10
20
14
−1
1−
11
20
14
−1
1−
12
20
14
−1
1−
13
20
14
−1
1−
14
20
14
−1
1−
15
20
14
−1
1−
16
20
14
−1
1−
17
20
14
−1
1−
18
20
14
−1
1−
19
20
14
−1
1−
20
20
14
−1
1−
21
20
14
−1
1−
22
20
14
−1
1−
23
20
14
−1
1−
24
20
14
−1
1−
25
20
14
−1
1−
26
20
14
−1
1−
27
20
14
−1
1−
28
20
14
−1
1−
29
20
14
−1
1−
30
20
14
−1
2−
01
20
14
−1
2−
02
20
14
−1
2−
03
20
14
−1
2−
04
20
14
−1
2−
05
20
14
−1
2−
06
20
14
−1
2−
07
20
14
−1
2−
08
20
14
−1
2−
09
20
14
−1
2−
10
20
14
−1
2−
11
20
14
−1
2−
12
date
va
lue
Exploring the feature space of large time series collections Yahoo web traffic 17
Feature spaceACF1: first order autocorrelation = Corr(Yt, Yt−1)
Strength of trend and seasonality based on STLTrend linearity and curvatureSize of seasonal peak and troughSpectral entropyLumpiness: variance of block variances (block size 24).Spikiness: variances of leave-one-out variances of STL remainders.Level shift: Maximum difference in trimmed means of consecutivemoving windows of size 24.Variance change: Max difference in variances of consecutivemoving windows of size 24.Flat spots: Discretize sample space into 10 equal-sized intervals.Find max run length in any interval.Number of crossing points of mean line.Kullback-Leibler score: Maximum ofDKL(P‖Q) =
∫P(x) ln P(x)/Q(x)dx where P and Q are estimated by
kernel density estimators applied to consecutive windows of size 48.Change index: Time of maximum KL score
Exploring the feature space of large time series collections Yahoo web traffic 18
Principal component analysis
●●
●● ●●
● ●
●
●
●●
●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●●
●
●●●●
● ●●●
● ●
●
●●
●●
●
●
●●
●
●
●
● ●●
●●●●
●●
●●
●
● ●●●
●●
●● ●
●●
●
● ●
●
●●●
●
●
● ●
●● ●● ●●
●●
●●●
●
●
●●●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●
●
● ●●
● ●● ● ●●●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●●●
●●
●
●●●●●
●● ●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●● ●●
●
●●
●●
●●●●
●
●
●
●●●
●●●
●●●●●●●●
●
●●
● ●●
●
● ●●
●
●●●
●●●● ●●●●
●●
●●
●
●
●●
●
●
●
●●●● ●
●
●●
●●
●●
●●●
● ●●●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
● ●●●
●●
●●●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
● ●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●●
● ●●●●
● ●●
●
● ●●● ●●
●●
●●
● ●
●●
●
●●
●●
●
● ●●
●
●
●
● ●●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●● ●
● ●
●
●●●
●
●
●
●
●●
●●
●●
●●
●
●
●● ●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●
●●
●
●●
●
●●
●
●●
●●●●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●●●
●●
●
●
● ●
●●
●
●
●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●
●
●●
● ●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●●
●
●
●●
● ●
●●
●
●
●●
●
●
● ●●
●
●
●
●
●●●
●●●
●
● ●●●●
●● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ● ●
●
●
●●
●
●
●●
● ●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●● ●
●
●●
●
●
●
● ●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●●
●●
●
● ●
●●●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
● ●
●
●●
●
●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
● ● ●●
●●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●●●
●●
●●
●
●
●●
●●●
● ●●● ●
● ●●●
●●
●●●
●●●
●●
●
● ●● ●
●●●●●
●
●●●●
●●●
●● ●●●●
●●●●
●●●●
●●● ●
● ●●
● ●
●
●●
●
●
●
● ●●
●● ● ●●
●
● ●● ●● ●●
●
●●
●
●● ●●●●
● ●● ●
●
●●●
●
●
●●
●
●
●●
●●
●●●●
●
●
●
● ●
●●
●●
●● ●●●●
●
●
●● ●●
●●● ●●
●● ●
●
●
●
●
●
●●
●
●
●
●●
● ●
●● ●●● ●
●●●●
●
●●
●●●
●●
●
●
●●●
●●
●
●●
● ●● ● ●
●●●
● ●
●
●
●
●●
●
●
●● ●●
●●
● ●
●●
●
●● ●●●● ●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●● ●●●
●● ●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●
●
●●
● ●●
●
●
● ●
●
● ● ●
●●
●
●●
●
● ●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●
●
●●●
●●
●● ●●●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●● ●
●●
●
●●●
●
●
●
● ●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●●●
●
●●
●
●
●
●
● ●
●●●●
●
●
●
● ●
●
●●
● ●●
●
●●
●
●●
● ●
●
●
●
●
●●
●
●
●
●●
●● ●●●●
●●
●
● ●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
● ●
●
●●
●
●●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●●
●●
● ●
●●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●●●
●● ●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●
●
●
●
●●
●●
● ●●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
● ●●
●●●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●
●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●●
●
● ●
●●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●● ●●●
●●●
● ●●●● ●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●●●
●●
●●●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
● ●●●
●●
●
●● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●●●
● ●
●
●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●
●●●●● ●
●
●
●
●●
●
●●
●●
●
●●
●
●● ●
●
●●
●●
●
● ●●
●●
●
●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●● ●
●
●
●
●●● ●
●●
●
● ●● ●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●● ●●
●
●●
●
●● ●
●●
●
●● ●●
●
●
●
● ●● ●
●●
●
● ●●
●
●●
●
●
●●●●
●●●
●
● ●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
●●●
●●●
●●
●
● ●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●● ●●
●
●●
●●
●●
●●●●
●●
●
●
●
● ●
●
●●
●●
●● ●
●
●
● ●
●
●
●
●
●
● ●●●
●●
●●
●
●
●
●●●
●
●● ●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●● ●
●
●
●
●
●
●●●
●●
●● ●
●
● ●
●● ●
●
●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●●●●●
●● ●●●
●
●
● ●● ●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●
●
●
●● ●●
●●
●●●●●●●
●●
●●●●
●
●●
●
●●
●●●
●
●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●●
● ●●●
●
●
●
●
●
●●
● ●
●
●
●●
●●●
● ●●●
●●
●
●●●●
●●
●
●
●
● ●●
●
●●
●●
●●
●●
●
●
●
●●
●
●● ●
● ●●●
●●●●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
● ●●
●
●
●● ●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●●
●●
●●
●●
●
●●
●●
●
●● ●
●●●
●●
● ●●
●●
●●
●●
●
●
●
●●●
●
●●
●
●●●
●
●● ●
●●
●●
●●
●●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●●●
●
●●●
● ●
●
●
● ●
●
●●
● ●
●
●●
● ●●●●
●●● ●
●●
●
●●●
●● ●
●●●
●
●●●
●●●
● ●
●
●
●
●●
●
●●●
●
●●
●
●
●● ●
●
●
●●
●●
●
●
●●●
●
●●●
●●
●
●●
●
●
●
● ●
●
● ●●●
●
●●●
●●
●
●●
●●●
●
●● ●●●● ●
●●
●●
●●
●
●
●
●●● ●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
● ●●
●●
●●●●
●
● ●●●●●
●
●●
●
●●●
●●●
●●
●●
●●
●
●●●● ●
●●
●
●
●●●
●●●
●
●●
●
●
●●●
●● ● ●
●●
●
●
●●
●
●●●
● ●●● ●●●
●
●
●
●●
●
● ●●
●
●●
●
●●
●
●
●●
●● ●●●●●●● ●
●●●
●●
●
●
●●
●●●
●●●●
●●●●
● ●
●
●
●●
●●
●
●●
● ●●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●
●
● ●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
● ●●●
● ●●●
●
●
●
●●
●● ●●
●
●
●
●●●
●
●●
●●●
●
●● ●
● ●●
●●
●●●
●●
● ●
●● ●
●● ●●●
●●●
●
●
●●
●●
●● ●●
●●
● ●●
●●●
●●
●●● ●
●
●●●
●●
●
● ●
●●●
●●
●
●●
●●
●
●
●●●
●●
●
●●●●●
●●●
●●●
●
●
●●●
●
●
●●●
●● ●●
● ●●●●
● ●
●●
●●
●●
●
●
●●●
●●
●
●
●
●
●
●
●●
●●● ●
●
●●
●
●●
●
●●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●●
●●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●● ●
●
●●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●●
● ●
●
●●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●● ●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●
●●
●
●
●
●●●●
●●
●●
●
●●
●
●
●
●● ●● ●●
●● ●
●
●
●
●●●
●
●● ●●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
●
●● ●
●●● ●
●●
●●
●
●●
●
●
●
●● ●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
● ●
● ●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●●
●●
●
●●●
●
●
●
●●●●●●●●
●●●●● ●●●
● ●●●
●● ●●
●●●
●
●●●
●● ●●
●
●● ●
● ●●
●
●●●
●● ●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ●
●
●●
● ●●
●●
●
●
●
●
●
●
●
●
●
● ●●●
●●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
● ●
● ●
●
●
●●
●●
●
●
●● ●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
● ●●●
●
●●
●
●
●
●
●
●●
● ●
●
●● ●●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●
●
●
●●●
●
●
●●
●
●●●●
●●
●●
●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●●
●●
●●●
●
●●
●●● ●
●●●
●●
●
● ●
●●
●
●●
●●
●● ●●
●●
●●
●●
●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●●
●●●
● ●●●
●●
●
●
●●
●● ●
●●● ●
●
●
●●●
●●
●
●
●●● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●● ●●● ●●
●
●●
●●
●● ●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●
●●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
● ●●●●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●●
●
●
●
●●●●
●●
●●
●●
●
●
●
●●
●
●
●●
●●
●●
●
●
●●
●
●
●
●●●●
●●
● ●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●
●
● ●●
● ●●
●
●
●● ●● ●
●
●●●
●
●●
●
● ●●
●●
● ●●
●
●
●
● ●●●●
●
●●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●
●
●
● ●
●●●
●
●●
●●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
● ●
●● ●
●● ●● ●●●
●●● ●
●
●
●●
●
●
●
● ●
●
●●●
●
●●
●●● ●●
●●
●
●●
●
●
●
●●
●
●●
●●●
●●
●
●
●
●
●
●●
●● ●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
● ●
●●●●
●●
●
●
●●
● ●
● ●●
●
●
● ●
●
●●
●
●
● ●
●
●
●
●
●●●●
●
●
●
●●●●
●●
●
● ●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●●● ●●●●
●●
●
●
● ●
●
●
●●
●
●●●●
●●●●
●
●●
● ●
●
●
●●●
● ●●
●●
●
●●●
●
●●
●
● ● ●
● ●●
●●
●●
●
●
●●
●
●●●
●●
●●
●●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●
●● ●
●●
●
●●
●
●
●●● ●
●●●
●
●●
●●
●●
●●●●
●
●● ●
●
●
●●
●
●●●
●
●●●
●●●
●●● ●● ●●
●● ●●
● ●●
●●
● ●
●●
● ●●
● ●●●
●
● ●●●●
●●
●● ●
●
●
● ●
● ●
●●●
● ●●
●
●●●●
●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●●
●
● ●●●●
●●
●
●●
●
●
●●
●
●
●
●●●●●●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●●
●● ●●
●
●
● ●
●
●●
●
●●
●
●
●● ●
●●
●●
●
●
●● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●● ●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●●●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●● ●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
● ●
●
●●●
●
●● ●
●
●
●
●
●
●●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●●
●●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●● ●●
●
●●
●
●●
●
●
●
●●
●
●● ● ●
●
●
●
●
●●
●●
●
●●
●●
●
●
● ●●
● ●
●●
●
●
●●
●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
● ●●●
●
●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●●
●
●●
●●
●●
●●● ●● ●●
●●●●
●
●●
●
●● ●●
●
●●●
●
●●
●
●●●
●
● ● ●● ●●●
●●● ●●●●
●
●●●
●
●●
●●
● ●● ●●●● ●●●
● ●●
●
● ●●
●
●●●
●●●●
●
●●●
●
● ●
●
●
●● ●●●●●●●●●●●
●●●
●
●●
●
●
●●●
●●●
●●●●
●●●●●●
●●●
● ●●
●
●●
●●
●●●●●●●●
●●
●
●● ●●
●● ●
●●●●
●●●●
●
●●●●●
●●●●● ●●
●●●
●
●●
●
●●
●
●● ●●●
●●●
● ●●
● ●●
●
●●●
●
●●
●● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●●
●●
●●
●●
●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
● ●●
● ●
●
● ●
●
●●
●
●●
●●●
●●
● ●●●●
●●●●
●●
●
● ●●● ●
●
●●
●
●●● ●
●●
●
●●
●
●●●●
●●
●●●
●● ● ●●
●●●
●●
●●
● ●●●
●●●●
●●
●●
●●
●●●
●
●
●● ●
●
●
● ●●
●● ● ●
●
●
●
●●
●●
●●●●
●
●
● ●● ●
●●
● ●● ●●●
●●●
●●● ●●●●
●
●●●●●
●
●
●
●●
●● ●
● ● ●●
●●●
●●●
●●●●●●● ● ●
●●● ●● ●● ●●
●●
●● ●●
●
●
●
●
●
● ●
●●
●●●
●
●●
●
●●
●●● ●●●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●●
● ●●
●
● ●● ●●●
●
●
●●
●●●●
●
●●●
●●
●
●
● ●●
●
●
●
●
●
●●
● ●●●
●● ●
●●
●
●●
●
●
●●
●
●
●●●
●●●●
●●●●
●
● ●● ●●
●●●●●●● ●
●●●
●
●●● ●●
● ●●
●
●● ●
● ●
●
●● ●
●
●●●
●●
●●●●
●●
● ●●●
●
●
●
●
●●
●
●
●
●●
●
● ●●
● ●●
●
●● ●
●
●
●
●● ●
●
● ●●
●
●
●
●
● ●●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●●
●●●
●●●
●
●
●● ●●●
●
●●
●
●
●
●
●●
●●
●●
●
● ●●
●
● ●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●● ●
●●●
●●
●
●● ●●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●● ●●
●
●
●
●
●
●●
●●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●
● ●
●
●●
●●
●●
●
● ●●
●●●
●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●●
● ●●
●●●
●●●
●●●●
●●●●●●●●●
●●
●
●●
● ●●●●
●
●
●●● ●●
●●
●● ●●
●●
●●●● ●
●●● ●
●
●●
●●●
●
●●●●●
●●●
●
●●
●●
●
●●
●
●
●●●
●
●●
●●
●●●● ●
●
●●
●
●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●
● ●●
●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●● ●
●●
●● ●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●● ●●
●
●●
●●
●
●●
●●
●
● ●● ●
●●
●●
●●
●
●●●
●●●●
●●
●● ●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
● ● ●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●● ●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
● ●●●●
●
●●●
●
●
●
●●
●●
●
●
●
●●●●●●●
● ●● ●●●
●●
●
●
●●
●● ●●●
●●●
●●●
●●
●●
●●●●●●●
●●●
●● ●● ●
●
●●●
●●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
● ●● ●●●●●
●
●
●●●
●●
●
●
●●● ●●
●●●●
●● ●● ●
●
●●
● ●●●●● ●●●
●●●●●● ●●●●●● ●●
●
●●
●
●●
●
●●
●●●●
●●●●●
● ●●●●
●
●●●● ●
●●●●●●●
●●
●
●
● ●●
●●●
●●
●●● ●●
●●●●● ●●●●●
●●●● ●●
●●● ●●● ●
●
●● ●●●●
●● ●●●●●●●
●● ●● ●● ●
●●●
●
●
●● ●
●
●●●
● ●●●●
●●
●● ●●● ●
●●●●
●●●
●●●●● ●●●●
●
●●
●
●
●●●●
●●●● ●●● ●
●
●●● ●●● ●●
●●●●●●
●
●●● ●●●●●● ●●●
● ●●●●●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●●
●
●●
●
●●
●●
●● ●●
●●
●●
● ●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●●
●●
●●● ●●
●
●●
●
●
●
●
●
●●
● ●●●●● ●
●
●●●●● ●●
● ●
●
●●
●●
●●●
●
●
●
●
●●
●●
●
●
● ●
●
●●●● ●
● ●●
●
●●●
●
●
●
●●
●
●●●●
●
●
●●
● ●●
●
●
●
●●
●
●
●
●
●
● ●● ●●
●
●
● ●
●
●●
●
●
● ●●
●● ●
●
●●
●●
●●
●● ●●●
●
●●
●●●●
●
●
●●●
●
●●
●●
●●
●
●●
●●●●
●●
●
● ●● ●●●●
●
● ●● ●●
●
●●
●
●●
●
●●●
●●
●●●●
●●●●●
●●●
●
●●
●
●●●
●
●● ●
●●
●
● ●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●
●●
●
●
●
●●● ● ●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●
●●●
●
●
●●
●
● ●
●●●
●
●
●
●●
●
● ●
●
●
●
● ●●
●
●
●● ●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●●● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
● ●
●●
●●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●
●●
●
● ●
●
●
● ●
●
●●
● ●
●
●
●
●●●●●
●
●● ●●
●●●
●
●●
●● ●
●
●●●
●●●
● ●● ●
●●●
●●●●● ●●
●●
●
●● ●●
●●
●
●● ●
●●
●●●
●
●●●
●●●
● ●●●
● ●●
●● ●●
●
●●●
●
●
●●
●●
●●●
●
●●
●●
●●
●
●●●● ●●●
●
●●●●●●
●●●
●●● ●●
●
●
●
●
● ●
●
●●●
●
●
●
●
●●●●●
●● ●●●●
●
●●
●● ●●
●
●● ●● ●
●●●●
●
●●●
●●
●
●●●
●●●●●●
●●● ●●
●●
●●
●●●
●● ●
●
●●●● ●
●●●
●
●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●●●
● ●
●● ●●
●●
●●
●●
●
●●● ●●●
●
●●
●●
●●●●●
●● ● ●
●●
●
●
● ●●●
●
●●
●
●
●●
●●●
●
●●●
●●
●●
●●●
●
●
●●
● ●● ●
●
●
● ●
●
● ●●
●●●
●●●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●●●●
●
●● ●●●
●
●●
●
●● ●●
●
●
●●
●
●●
●●●
●●●
● ●●
●
●
●●
●
●
●
●
●●
●
● ●
●●
●
●● ● ●●
●●
●●●●
●●
●●
●●
●●● ●●●●
●
●● ●
●
●
●●
●
●●
●●●●
●
●
● ●●●
●
●●●
●●
●
● ●●
●
●
●●
●●●●●●●● ●●●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
● ●●
●●●
●
●●
● ●
● ●●
●●●● ●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●●
●●● ●
●
●●●●
●●●●
●
●
●
●
●●
●●
●●●●●●●
●●●
●
●
●●●●●
●●
●
●●
●●●
●
●
●
●●●
●
●
●
● ●
●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●●●
●●
●
●●
●●
● ●●
●●●
● ●●
●
●
●
●
●
●●
●●●●
● ●●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●●
●●●
●
● ●●●
●
●● ●
●
●●
●●●
●●●
●
●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●●●●
●
●
●●●
●●
●●
●
●
●●
●●●● ●●
●
●●●●
●● ●●
●●●
●●
●●●●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
ACF1
lumpiness
entropy
lshiftvchange
cpoints
fspo
ts
trend
linearity
curvature
spikiness
seas
onpeak
trou
gh
klscore
chan
ge.id
x
−4
−2
0
2
−2.5 0.0 2.5standardized PC1 (28.7% explained var.)
stan
dard
ized
PC
2 (1
7.3%
exp
lain
ed v
ar.)
Exploring the feature space of large time series collections Yahoo web traffic 19
What is “anomalous”
●●
●● ●●
● ●
●
●
●●
●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●●
●
●●●●
● ●●●
● ●
●
●●
●●
●
●
●●
●
●
●
● ●●
●●●●
●●
●●
●
● ●●●
●●
●● ●
●●
●
● ●
●
●●●
●
●
● ●
●● ●● ●●
●●
●●●
●
●
●●●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●
●
● ●●
● ●● ● ●●●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●●●
●●
●
●●●●●
●● ●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●● ●●
●
●●
●●
●●●●
●
●
●
●●●
●●●
●●●●●●●●
●
●●
● ●●
●
● ●●
●
●●●
●●●● ●●●●
●●
●●
●
●
●●
●
●
●
●●●● ●
●
●●
●●
●●
●●●
● ●●●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
● ●●●
●●
●●●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
● ●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●●
● ●●●●
● ●●
●
● ●●● ●●
●●
●●
● ●
●●
●
●●
●●
●
● ●●
●
●
●
● ●●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●● ●
● ●
●
●●●
●
●
●
●
●●
●●
●●
●●
●
●
●● ●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●
●●
●
●●
●
●●
●
●●
●●●●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●●●
●●
●
●
● ●
●●
●
●
●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●
●
●●
● ●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●●
●
●
●●
● ●
●●
●
●
●●
●
●
● ●●
●
●
●
●
●●●
●●●
●
● ●●●●
●● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ● ●
●
●
●●
●
●
●●
● ●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●● ●
●
●●
●
●
●
● ●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●●
●●
●
● ●
●●●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
● ●
●
●●
●
●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
● ● ●●
●●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●●●
●●
●●
●
●
●●
●●●
● ●●● ●
● ●●●
●●
●●●
●●●
●●
●
● ●● ●
●●●●●
●
●●●●
●●●
●● ●●●●
●●●●
●●●●
●●● ●
● ●●
● ●
●
●●
●
●
●
● ●●
●● ● ●●
●
● ●● ●● ●●
●
●●
●
●● ●●●●
● ●● ●
●
●●●
●
●
●●
●
●
●●
●●
●●●●
●
●
●
● ●
●●
●●
●● ●●●●
●
●
●● ●●
●●● ●●
●● ●
●
●
●
●
●
●●
●
●
●
●●
● ●
●● ●●● ●
●●●●
●
●●
●●●
●●
●
●
●●●
●●
●
●●
● ●● ● ●
●●●
● ●
●
●
●
●●
●
●
●● ●●
●●
● ●
●●
●
●● ●●●● ●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●● ●●●
●● ●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●
●
●●
● ●●
●
●
● ●
●
● ● ●
●●
●
●●
●
● ●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●
●
●●●
●●
●● ●●●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●● ●
●●
●
●●●
●
●
●
● ●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●●●
●
●●
●
●
●
●
● ●
●●●●
●
●
●
● ●
●
●●
● ●●
●
●●
●
●●
● ●
●
●
●
●
●●
●
●
●
●●
●● ●●●●
●●
●
● ●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
● ●
●
●●
●
●●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●●
●●
● ●
●●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●●●
●● ●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●
●
●
●
●●
●●
● ●●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
● ●●
●●●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●
●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●●
●
● ●
●●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●● ●●●
●●●
● ●●●● ●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●●●
●●
●●●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
● ●●●
●●
●
●● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●●●
● ●
●
●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●
●●●●● ●
●
●
●
●●
●
●●
●●
●
●●
●
●● ●
●
●●
●●
●
● ●●
●●
●
●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●● ●
●
●
●
●●● ●
●●
●
● ●● ●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●● ●●
●
●●
●
●● ●
●●
●
●● ●●
●
●
●
● ●● ●
●●
●
● ●●
●
●●
●
●
●●●●
●●●
●
● ●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
●●●
●●●
●●
●
● ●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●● ●●
●
●●
●●
●●
●●●●
●●
●
●
●
● ●
●
●●
●●
●● ●
●
●
● ●
●
●
●
●
●
● ●●●
●●
●●
●
●
●
●●●
●
●● ●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●● ●
●
●
●
●
●
●●●
●●
●● ●
●
● ●
●● ●
●
●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●●●●●
●● ●●●
●
●
● ●● ●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●
●
●
●● ●●
●●
●●●●●●●
●●
●●●●
●
●●
●
●●
●●●
●
●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●●
● ●●●
●
●
●
●
●
●●
● ●
●
●
●●
●●●
● ●●●
●●
●
●●●●
●●
●
●
●
● ●●
●
●●
●●
●●
●●
●
●
●
●●
●
●● ●
● ●●●
●●●●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
● ●●
●
●
●● ●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●●
●●
●●
●●
●
●●
●●
●
●● ●
●●●
●●
● ●●
●●
●●
●●
●
●
●
●●●
●
●●
●
●●●
●
●● ●
●●
●●
●●
●●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●●●
●
●●●
● ●
●
●
● ●
●
●●
● ●
●
●●
● ●●●●
●●● ●
●●
●
●●●
●● ●
●●●
●
●●●
●●●
● ●
●
●
●
●●
●
●●●
●
●●
●
●
●● ●
●
●
●●
●●
●
●
●●●
●
●●●
●●
●
●●
●
●
●
● ●
●
● ●●●
●
●●●
●●
●
●●
●●●
●
●● ●●●● ●
●●
●●
●●
●
●
●
●●● ●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
● ●●
●●
●●●●
●
● ●●●●●
●
●●
●
●●●
●●●
●●
●●
●●
●
●●●● ●
●●
●
●
●●●
●●●
●
●●
●
●
●●●
●● ● ●
●●
●
●
●●
●
●●●
● ●●● ●●●
●
●
●
●●
●
● ●●
●
●●
●
●●
●
●
●●
●● ●●●●●●● ●
●●●
●●
●
●
●●
●●●
●●●●
●●●●
● ●
●
●
●●
●●
●
●●
● ●●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●
●
● ●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
● ●●●
● ●●●
●
●
●
●●
●● ●●
●
●
●
●●●
●
●●
●●●
●
●● ●
● ●●
●●
●●●
●●
● ●
●● ●
●● ●●●
●●●
●
●
●●
●●
●● ●●
●●
● ●●
●●●
●●
●●● ●
●
●●●
●●
●
● ●
●●●
●●
●
●●
●●
●
●
●●●
●●
●
●●●●●
●●●
●●●
●
●
●●●
●
●
●●●
●● ●●
● ●●●●
● ●
●●
●●
●●
●
●
●●●
●●
●
●
●
●
●
●
●●
●●● ●
●
●●
●
●●
●
●●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●●
●●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●● ●
●
●●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●●
● ●
●
●●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●● ●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●
●●
●
●
●
●●●●
●●
●●
●
●●
●
●
●
●● ●● ●●
●● ●
●
●
●
●●●
●
●● ●●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
●
●● ●
●●● ●
●●
●●
●
●●
●
●
●
●● ●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
● ●
● ●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●●
●●
●
●●●
●
●
●
●●●●●●●●
●●●●● ●●●
● ●●●
●● ●●
●●●
●
●●●
●● ●●
●
●● ●
● ●●
●
●●●
●● ●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ●
●
●●
● ●●
●●
●
●
●
●
●
●
●
●
●
● ●●●
●●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
● ●
● ●
●
●
●●
●●
●
●
●● ●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
● ●●●
●
●●
●
●
●
●
●
●●
● ●
●
●● ●●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●
●
●
●●●
●
●
●●
●
●●●●
●●
●●
●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●●
●●
●●●
●
●●
●●● ●
●●●
●●
●
● ●
●●
●
●●
●●
●● ●●
●●
●●
●●
●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●●
●●●
● ●●●
●●
●
●
●●
●● ●
●●● ●
●
●
●●●
●●
●
●
●●● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●● ●●● ●●
●
●●
●●
●● ●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●
●●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
● ●●●●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●●
●
●
●
●●●●
●●
●●
●●
●
●
●
●●
●
●
●●
●●
●●
●
●
●●
●
●
●
●●●●
●●
● ●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●
●
● ●●
● ●●
●
●
●● ●● ●
●
●●●
●
●●
●
● ●●
●●
● ●●
●
●
●
● ●●●●
●
●●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●
●
●
● ●
●●●
●
●●
●●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
● ●
●● ●
●● ●● ●●●
●●● ●
●
●
●●
●
●
●
● ●
●
●●●
●
●●
●●● ●●
●●
●
●●
●
●
●
●●
●
●●
●●●
●●
●
●
●
●
●
●●
●● ●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
● ●
●●●●
●●
●
●
●●
● ●
● ●●
●
●
● ●
●
●●
●
●
● ●
●
●
●
●
●●●●
●
●
●
●●●●
●●
●
● ●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●●● ●●●●
●●
●
●
● ●
●
●
●●
●
●●●●
●●●●
●
●●
● ●
●
●
●●●
● ●●
●●
●
●●●
●
●●
●
● ● ●
● ●●
●●
●●
●
●
●●
●
●●●
●●
●●
●●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●
●● ●
●●
●
●●
●
●
●●● ●
●●●
●
●●
●●
●●
●●●●
●
●● ●
●
●
●●
●
●●●
●
●●●
●●●
●●● ●● ●●
●● ●●
● ●●
●●
● ●
●●
● ●●
● ●●●
●
● ●●●●
●●
●● ●
●
●
● ●
● ●
●●●
● ●●
●
●●●●
●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●●
●
● ●●●●
●●
●
●●
●
●
●●
●
●
●
●●●●●●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●●
●● ●●
●
●
● ●
●
●●
●
●●
●
●
●● ●
●●
●●
●
●
●● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●● ●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●●●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●● ●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
● ●
●
●●●
●
●● ●
●
●
●
●
●
●●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●●
●●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●● ●●
●
●●
●
●●
●
●
●
●●
●
●● ● ●
●
●
●
●
●●
●●
●
●●
●●
●
●
● ●●
● ●
●●
●
●
●●
●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
● ●●●
●
●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●●
●
●●
●●
●●
●●● ●● ●●
●●●●
●
●●
●
●● ●●
●
●●●
●
●●
●
●●●
●
● ● ●● ●●●
●●● ●●●●
●
●●●
●
●●
●●
● ●● ●●●● ●●●
● ●●
●
● ●●
●
●●●
●●●●
●
●●●
●
● ●
●
●
●● ●●●●●●●●●●●
●●●
●
●●
●
●
●●●
●●●
●●●●
●●●●●●
●●●
● ●●
●
●●
●●
●●●●●●●●
●●
●
●● ●●
●● ●
●●●●
●●●●
●
●●●●●
●●●●● ●●
●●●
●
●●
●
●●
●
●● ●●●
●●●
● ●●
● ●●
●
●●●
●
●●
●● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●●
●●
●●
●●
●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
● ●●
● ●
●
● ●
●
●●
●
●●
●●●
●●
● ●●●●
●●●●
●●
●
● ●●● ●
●
●●
●
●●● ●
●●
●
●●
●
●●●●
●●
●●●
●● ● ●●
●●●
●●
●●
● ●●●
●●●●
●●
●●
●●
●●●
●
●
●● ●
●
●
● ●●
●● ● ●
●
●
●
●●
●●
●●●●
●
●
● ●● ●
●●
● ●● ●●●
●●●
●●● ●●●●
●
●●●●●
●
●
●
●●
●● ●
● ● ●●
●●●
●●●
●●●●●●● ● ●
●●● ●● ●● ●●
●●
●● ●●
●
●
●
●
●
● ●
●●
●●●
●
●●
●
●●
●●● ●●●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●●
● ●●
●
● ●● ●●●
●
●
●●
●●●●
●
●●●
●●
●
●
● ●●
●
●
●
●
●
●●
● ●●●
●● ●
●●
●
●●
●
●
●●
●
●
●●●
●●●●
●●●●
●
● ●● ●●
●●●●●●● ●
●●●
●
●●● ●●
● ●●
●
●● ●
● ●
●
●● ●
●
●●●
●●
●●●●
●●
● ●●●
●
●
●
●
●●
●
●
●
●●
●
● ●●
● ●●
●
●● ●
●
●
●
●● ●
●
● ●●
●
●
●
●
● ●●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●●
●●●
●●●
●
●
●● ●●●
●
●●
●
●
●
●
●●
●●
●●
●
● ●●
●
● ●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●● ●
●●●
●●
●
●● ●●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●● ●●
●
●
●
●
●
●●
●●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●
● ●
●
●●
●●
●●
●
● ●●
●●●
●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●●
● ●●
●●●
●●●
●●●●
●●●●●●●●●
●●
●
●●
● ●●●●
●
●
●●● ●●
●●
●● ●●
●●
●●●● ●
●●● ●
●
●●
●●●
●
●●●●●
●●●
●
●●
●●
●
●●
●
●
●●●
●
●●
●●
●●●● ●
●
●●
●
●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●
● ●●
●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●● ●
●●
●● ●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●● ●●
●
●●
●●
●
●●
●●
●
● ●● ●
●●
●●
●●
●
●●●
●●●●
●●
●● ●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
● ● ●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●● ●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
● ●●●●
●
●●●
●
●
●
●●
●●
●
●
●
●●●●●●●
● ●● ●●●
●●
●
●
●●
●● ●●●
●●●
●●●
●●
●●
●●●●●●●
●●●
●● ●● ●
●
●●●
●●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
● ●● ●●●●●
●
●
●●●
●●
●
●
●●● ●●
●●●●
●● ●● ●
●
●●
● ●●●●● ●●●
●●●●●● ●●●●●● ●●
●
●●
●
●●
●
●●
●●●●
●●●●●
● ●●●●
●
●●●● ●
●●●●●●●
●●
●
●
● ●●
●●●
●●
●●● ●●
●●●●● ●●●●●
●●●● ●●
●●● ●●● ●
●
●● ●●●●
●● ●●●●●●●
●● ●● ●● ●
●●●
●
●
●● ●
●
●●●
● ●●●●
●●
●● ●●● ●
●●●●
●●●
●●●●● ●●●●
●
●●
●
●
●●●●
●●●● ●●● ●
●
●●● ●●● ●●
●●●●●●
●
●●● ●●●●●● ●●●
● ●●●●●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●●
●
●●
●
●●
●●
●● ●●
●●
●●
● ●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●●
●●
●●● ●●
●
●●
●
●
●
●
●
●●
● ●●●●● ●
●
●●●●● ●●
● ●
●
●●
●●
●●●
●
●
●
●
●●
●●
●
●
● ●
●
●●●● ●
● ●●
●
●●●
●
●
●
●●
●
●●●●
●
●
●●
● ●●
●
●
●
●●
●
●
●
●
●
● ●● ●●
●
●
● ●
●
●●
●
●
● ●●
●● ●
●
●●
●●
●●
●● ●●●
●
●●
●●●●
●
●
●●●
●
●●
●●
●●
●
●●
●●●●
●●
●
● ●● ●●●●
●
● ●● ●●
●
●●
●
●●
●
●●●
●●
●●●●
●●●●●
●●●
●
●●
●
●●●
●
●● ●
●●
●
● ●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●
●●
●
●
●
●●● ● ●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●
●●●
●
●
●●
●
● ●
●●●
●
●
●
●●
●
● ●
●
●
●
● ●●
●
●
●● ●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●●● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
● ●
●●
●●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●
●●
●
● ●
●
●
● ●
●
●●
● ●
●
●
●
●●●●●
●
●● ●●
●●●
●
●●
●● ●
●
●●●
●●●
● ●● ●
●●●
●●●●● ●●
●●
●
●● ●●
●●
●
●● ●
●●
●●●
●
●●●
●●●
● ●●●
● ●●
●● ●●
●
●●●
●
●
●●
●●
●●●
●
●●
●●
●●
●
●●●● ●●●
●
●●●●●●
●●●
●●● ●●
●
●
●
●
● ●
●
●●●
●
●
●
●
●●●●●
●● ●●●●
●
●●
●● ●●
●
●● ●● ●
●●●●
●
●●●
●●
●
●●●
●●●●●●
●●● ●●
●●
●●
●●●
●● ●
●
●●●● ●
●●●
●
●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●●●
● ●
●● ●●
●●
●●
●●
●
●●● ●●●
●
●●
●●
●●●●●
●● ● ●
●●
●
●
● ●●●
●
●●
●
●
●●
●●●
●
●●●
●●
●●
●●●
●
●
●●
● ●● ●
●
●
● ●
●
● ●●
●●●
●●●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●●●●
●
●● ●●●
●
●●
●
●● ●●
●
●
●●
●
●●
●●●
●●●
● ●●
●
●
●●
●
●
●
●
●●
●
● ●
●●
●
●● ● ●●
●●
●●●●
●●
●●
●●
●●● ●●●●
●
●● ●
●
●
●●
●
●●
●●●●
●
●
● ●●●
●
●●●
●●
●
● ●●
●
●
●●
●●●●●●●● ●●●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
● ●●
●●●
●
●●
● ●
● ●●
●●●● ●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●●
●●● ●
●
●●●●
●●●●
●
●
●
●
●●
●●
●●●●●●●
●●●
●
●
●●●●●
●●
●
●●
●●●
●
●
●
●●●
●
●
●
● ●
●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●●●
●●
●
●●
●●
● ●●
●●●
● ●●
●
●
●
●
●
●●
●●●●
● ●●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●●
●●●
●
● ●●●
●
●● ●
●
●●
●●●
●●●
●
●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●●●●
●
●
●●●
●●
●●
●
●
●●
●●●● ●●
●
●●●●
●● ●●
●●●
●●
●●●●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
ACF1
lumpiness
entropy
lshiftvchange
cpoints
fspo
ts
trend
linearity
curvature
spikiness
seas
onpeak
trou
gh
klscore
chan
ge.id
x
−4
−2
0
2
−2.5 0.0 2.5standardized PC1 (28.7% explained var.)
stan
dard
ized
PC
2 (1
7.3%
exp
lain
ed v
ar.)
We need a measure of the “anomalousness” of a timeseries.
1 Rank points based on their local density.2 Rank points based on whether they are within
α-convex hulls of different radius.Exploring the feature space of large time series collections Yahoo web traffic 20
What is “anomalous”
●●
●● ●●
● ●
●
●
●●
●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●●
●
●●●●
● ●●●
● ●
●
●●
●●
●
●
●●
●
●
●
● ●●
●●●●
●●
●●
●
● ●●●
●●
●● ●
●●
●
● ●
●
●●●
●
●
● ●
●● ●● ●●
●●
●●●
●
●
●●●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●
●
● ●●
● ●● ● ●●●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●●●
●●
●
●●●●●
●● ●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●● ●●
●
●●
●●
●●●●
●
●
●
●●●
●●●
●●●●●●●●
●
●●
● ●●
●
● ●●
●
●●●
●●●● ●●●●
●●
●●
●
●
●●
●
●
●
●●●● ●
●
●●
●●
●●
●●●
● ●●●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
● ●●●
●●
●●●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
● ●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●●
● ●●●●
● ●●
●
● ●●● ●●
●●
●●
● ●
●●
●
●●
●●
●
● ●●
●
●
●
● ●●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●● ●
● ●
●
●●●
●
●
●
●
●●
●●
●●
●●
●
●
●● ●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●
●●
●
●●
●
●●
●
●●
●●●●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●●●
●●
●
●
● ●
●●
●
●
●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●
●
●●
● ●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●●
●
●
●●
● ●
●●
●
●
●●
●
●
● ●●
●
●
●
●
●●●
●●●
●
● ●●●●
●● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ● ●
●
●
●●
●
●
●●
● ●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●● ●
●
●●
●
●
●
● ●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●●
●●
●
● ●
●●●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
● ●
●
●●
●
●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
● ● ●●
●●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●●●
●●
●●
●
●
●●
●●●
● ●●● ●
● ●●●
●●
●●●
●●●
●●
●
● ●● ●
●●●●●
●
●●●●
●●●
●● ●●●●
●●●●
●●●●
●●● ●
● ●●
● ●
●
●●
●
●
●
● ●●
●● ● ●●
●
● ●● ●● ●●
●
●●
●
●● ●●●●
● ●● ●
●
●●●
●
●
●●
●
●
●●
●●
●●●●
●
●
●
● ●
●●
●●
●● ●●●●
●
●
●● ●●
●●● ●●
●● ●
●
●
●
●
●
●●
●
●
●
●●
● ●
●● ●●● ●
●●●●
●
●●
●●●
●●
●
●
●●●
●●
●
●●
● ●● ● ●
●●●
● ●
●
●
●
●●
●
●
●● ●●
●●
● ●
●●
●
●● ●●●● ●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●● ●●●
●● ●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●
●
●●
● ●●
●
●
● ●
●
● ● ●
●●
●
●●
●
● ●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●
●
●●●
●●
●● ●●●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●● ●
●●
●
●●●
●
●
●
● ●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●●●
●
●●
●
●
●
●
● ●
●●●●
●
●
●
● ●
●
●●
● ●●
●
●●
●
●●
● ●
●
●
●
●
●●
●
●
●
●●
●● ●●●●
●●
●
● ●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
● ●
●
●●
●
●●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●●
●●
● ●
●●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●●●
●● ●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●
●
●
●
●●
●●
● ●●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
● ●●
●●●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●
●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●●
●
● ●
●●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●● ●●●
●●●
● ●●●● ●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●●●
●●
●●●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
● ●●●
●●
●
●● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●●●
● ●
●
●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●
●●●●● ●
●
●
●
●●
●
●●
●●
●
●●
●
●● ●
●
●●
●●
●
● ●●
●●
●
●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●● ●
●
●
●
●●● ●
●●
●
● ●● ●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●● ●●
●
●●
●
●● ●
●●
●
●● ●●
●
●
●
● ●● ●
●●
●
● ●●
●
●●
●
●
●●●●
●●●
●
● ●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
●●●
●●●
●●
●
● ●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●● ●●
●
●●
●●
●●
●●●●
●●
●
●
●
● ●
●
●●
●●
●● ●
●
●
● ●
●
●
●
●
●
● ●●●
●●
●●
●
●
●
●●●
●
●● ●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●● ●
●
●
●
●
●
●●●
●●
●● ●
●
● ●
●● ●
●
●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●●●●●
●● ●●●
●
●
● ●● ●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●
●
●
●● ●●
●●
●●●●●●●
●●
●●●●
●
●●
●
●●
●●●
●
●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●●
● ●●●
●
●
●
●
●
●●
● ●
●
●
●●
●●●
● ●●●
●●
●
●●●●
●●
●
●
●
● ●●
●
●●
●●
●●
●●
●
●
●
●●
●
●● ●
● ●●●
●●●●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
● ●●
●
●
●● ●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●●
●●
●●
●●
●
●●
●●
●
●● ●
●●●
●●
● ●●
●●
●●
●●
●
●
●
●●●
●
●●
●
●●●
●
●● ●
●●
●●
●●
●●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●●●
●
●●●
● ●
●
●
● ●
●
●●
● ●
●
●●
● ●●●●
●●● ●
●●
●
●●●
●● ●
●●●
●
●●●
●●●
● ●
●
●
●
●●
●
●●●
●
●●
●
●
●● ●
●
●
●●
●●
●
●
●●●
●
●●●
●●
●
●●
●
●
●
● ●
●
● ●●●
●
●●●
●●
●
●●
●●●
●
●● ●●●● ●
●●
●●
●●
●
●
●
●●● ●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
● ●●
●●
●●●●
●
● ●●●●●
●
●●
●
●●●
●●●
●●
●●
●●
●
●●●● ●
●●
●
●
●●●
●●●
●
●●
●
●
●●●
●● ● ●
●●
●
●
●●
●
●●●
● ●●● ●●●
●
●
●
●●
●
● ●●
●
●●
●
●●
●
●
●●
●● ●●●●●●● ●
●●●
●●
●
●
●●
●●●
●●●●
●●●●
● ●
●
●
●●
●●
●
●●
● ●●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●
●
● ●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
● ●●●
● ●●●
●
●
●
●●
●● ●●
●
●
●
●●●
●
●●
●●●
●
●● ●
● ●●
●●
●●●
●●
● ●
●● ●
●● ●●●
●●●
●
●
●●
●●
●● ●●
●●
● ●●
●●●
●●
●●● ●
●
●●●
●●
●
● ●
●●●
●●
●
●●
●●
●
●
●●●
●●
●
●●●●●
●●●
●●●
●
●
●●●
●
●
●●●
●● ●●
● ●●●●
● ●
●●
●●
●●
●
●
●●●
●●
●
●
●
●
●
●
●●
●●● ●
●
●●
●
●●
●
●●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●●
●●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●● ●
●
●●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●●
● ●
●
●●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●● ●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●
●●
●
●
●
●●●●
●●
●●
●
●●
●
●
●
●● ●● ●●
●● ●
●
●
●
●●●
●
●● ●●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
●
●● ●
●●● ●
●●
●●
●
●●
●
●
●
●● ●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
● ●
● ●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●●
●●
●
●●●
●
●
●
●●●●●●●●
●●●●● ●●●
● ●●●
●● ●●
●●●
●
●●●
●● ●●
●
●● ●
● ●●
●
●●●
●● ●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ●
●
●●
● ●●
●●
●
●
●
●
●
●
●
●
●
● ●●●
●●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
● ●
● ●
●
●
●●
●●
●
●
●● ●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
● ●●●
●
●●
●
●
●
●
●
●●
● ●
●
●● ●●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●
●
●
●●●
●
●
●●
●
●●●●
●●
●●
●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●●
●●
●●●
●
●●
●●● ●
●●●
●●
●
● ●
●●
●
●●
●●
●● ●●
●●
●●
●●
●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●●
●●●
● ●●●
●●
●
●
●●
●● ●
●●● ●
●
●
●●●
●●
●
●
●●● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●● ●●● ●●
●
●●
●●
●● ●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●
●●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
● ●●●●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●●
●
●
●
●●●●
●●
●●
●●
●
●
●
●●
●
●
●●
●●
●●
●
●
●●
●
●
●
●●●●
●●
● ●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●
●
● ●●
● ●●
●
●
●● ●● ●
●
●●●
●
●●
●
● ●●
●●
● ●●
●
●
●
● ●●●●
●
●●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●
●
●
● ●
●●●
●
●●
●●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
● ●
●● ●
●● ●● ●●●
●●● ●
●
●
●●
●
●
●
● ●
●
●●●
●
●●
●●● ●●
●●
●
●●
●
●
●
●●
●
●●
●●●
●●
●
●
●
●
●
●●
●● ●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
● ●
●●●●
●●
●
●
●●
● ●
● ●●
●
●
● ●
●
●●
●
●
● ●
●
●
●
●
●●●●
●
●
●
●●●●
●●
●
● ●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●●● ●●●●
●●
●
●
● ●
●
●
●●
●
●●●●
●●●●
●
●●
● ●
●
●
●●●
● ●●
●●
●
●●●
●
●●
●
● ● ●
● ●●
●●
●●
●
●
●●
●
●●●
●●
●●
●●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●
●● ●
●●
●
●●
●
●
●●● ●
●●●
●
●●
●●
●●
●●●●
●
●● ●
●
●
●●
●
●●●
●
●●●
●●●
●●● ●● ●●
●● ●●
● ●●
●●
● ●
●●
● ●●
● ●●●
●
● ●●●●
●●
●● ●
●
●
● ●
● ●
●●●
● ●●
●
●●●●
●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●●
●
● ●●●●
●●
●
●●
●
●
●●
●
●
●
●●●●●●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●●
●● ●●
●
●
● ●
●
●●
●
●●
●
●
●● ●
●●
●●
●
●
●● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●● ●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●●●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●● ●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
● ●
●
●●●
●
●● ●
●
●
●
●
●
●●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●●
●●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●● ●●
●
●●
●
●●
●
●
●
●●
●
●● ● ●
●
●
●
●
●●
●●
●
●●
●●
●
●
● ●●
● ●
●●
●
●
●●
●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
● ●●●
●
●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●●
●
●●
●●
●●
●●● ●● ●●
●●●●
●
●●
●
●● ●●
●
●●●
●
●●
●
●●●
●
● ● ●● ●●●
●●● ●●●●
●
●●●
●
●●
●●
● ●● ●●●● ●●●
● ●●
●
● ●●
●
●●●
●●●●
●
●●●
●
● ●
●
●
●● ●●●●●●●●●●●
●●●
●
●●
●
●
●●●
●●●
●●●●
●●●●●●
●●●
● ●●
●
●●
●●
●●●●●●●●
●●
●
●● ●●
●● ●
●●●●
●●●●
●
●●●●●
●●●●● ●●
●●●
●
●●
●
●●
●
●● ●●●
●●●
● ●●
● ●●
●
●●●
●
●●
●● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●●
●●
●●
●●
●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
● ●●
● ●
●
● ●
●
●●
●
●●
●●●
●●
● ●●●●
●●●●
●●
●
● ●●● ●
●
●●
●
●●● ●
●●
●
●●
●
●●●●
●●
●●●
●● ● ●●
●●●
●●
●●
● ●●●
●●●●
●●
●●
●●
●●●
●
●
●● ●
●
●
● ●●
●● ● ●
●
●
●
●●
●●
●●●●
●
●
● ●● ●
●●
● ●● ●●●
●●●
●●● ●●●●
●
●●●●●
●
●
●
●●
●● ●
● ● ●●
●●●
●●●
●●●●●●● ● ●
●●● ●● ●● ●●
●●
●● ●●
●
●
●
●
●
● ●
●●
●●●
●
●●
●
●●
●●● ●●●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●●
● ●●
●
● ●● ●●●
●
●
●●
●●●●
●
●●●
●●
●
●
● ●●
●
●
●
●
●
●●
● ●●●
●● ●
●●
●
●●
●
●
●●
●
●
●●●
●●●●
●●●●
●
● ●● ●●
●●●●●●● ●
●●●
●
●●● ●●
● ●●
●
●● ●
● ●
●
●● ●
●
●●●
●●
●●●●
●●
● ●●●
●
●
●
●
●●
●
●
●
●●
●
● ●●
● ●●
●
●● ●
●
●
●
●● ●
●
● ●●
●
●
●
●
● ●●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●●
●●●
●●●
●
●
●● ●●●
●
●●
●
●
●
●
●●
●●
●●
●
● ●●
●
● ●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●● ●
●●●
●●
●
●● ●●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●● ●●
●
●
●
●
●
●●
●●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●
● ●
●
●●
●●
●●
●
● ●●
●●●
●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●●
● ●●
●●●
●●●
●●●●
●●●●●●●●●
●●
●
●●
● ●●●●
●
●
●●● ●●
●●
●● ●●
●●
●●●● ●
●●● ●
●
●●
●●●
●
●●●●●
●●●
●
●●
●●
●
●●
●
●
●●●
●
●●
●●
●●●● ●
●
●●
●
●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●
● ●●
●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●● ●
●●
●● ●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●● ●●
●
●●
●●
●
●●
●●
●
● ●● ●
●●
●●
●●
●
●●●
●●●●
●●
●● ●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
● ● ●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●● ●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
● ●●●●
●
●●●
●
●
●
●●
●●
●
●
●
●●●●●●●
● ●● ●●●
●●
●
●
●●
●● ●●●
●●●
●●●
●●
●●
●●●●●●●
●●●
●● ●● ●
●
●●●
●●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
● ●● ●●●●●
●
●
●●●
●●
●
●
●●● ●●
●●●●
●● ●● ●
●
●●
● ●●●●● ●●●
●●●●●● ●●●●●● ●●
●
●●
●
●●
●
●●
●●●●
●●●●●
● ●●●●
●
●●●● ●
●●●●●●●
●●
●
●
● ●●
●●●
●●
●●● ●●
●●●●● ●●●●●
●●●● ●●
●●● ●●● ●
●
●● ●●●●
●● ●●●●●●●
●● ●● ●● ●
●●●
●
●
●● ●
●
●●●
● ●●●●
●●
●● ●●● ●
●●●●
●●●
●●●●● ●●●●
●
●●
●
●
●●●●
●●●● ●●● ●
●
●●● ●●● ●●
●●●●●●
●
●●● ●●●●●● ●●●
● ●●●●●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●●
●
●●
●
●●
●●
●● ●●
●●
●●
● ●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●●
●●
●●● ●●
●
●●
●
●
●
●
●
●●
● ●●●●● ●
●
●●●●● ●●
● ●
●
●●
●●
●●●
●
●
●
●
●●
●●
●
●
● ●
●
●●●● ●
● ●●
●
●●●
●
●
●
●●
●
●●●●
●
●
●●
● ●●
●
●
●
●●
●
●
●
●
●
● ●● ●●
●
●
● ●
●
●●
●
●
● ●●
●● ●
●
●●
●●
●●
●● ●●●
●
●●
●●●●
●
●
●●●
●
●●
●●
●●
●
●●
●●●●
●●
●
● ●● ●●●●
●
● ●● ●●
●
●●
●
●●
●
●●●
●●
●●●●
●●●●●
●●●
●
●●
●
●●●
●
●● ●
●●
●
● ●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●
●●
●
●
●
●●● ● ●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●
●●●
●
●
●●
●
● ●
●●●
●
●
●
●●
●
● ●
●
●
●
● ●●
●
●
●● ●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●●● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
● ●
●●
●●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●
●●
●
● ●
●
●
● ●
●
●●
● ●
●
●
●
●●●●●
●
●● ●●
●●●
●
●●
●● ●
●
●●●
●●●
● ●● ●
●●●
●●●●● ●●
●●
●
●● ●●
●●
●
●● ●
●●
●●●
●
●●●
●●●
● ●●●
● ●●
●● ●●
●
●●●
●
●
●●
●●
●●●
●
●●
●●
●●
●
●●●● ●●●
●
●●●●●●
●●●
●●● ●●
●
●
●
●
● ●
●
●●●
●
●
●
●
●●●●●
●● ●●●●
●
●●
●● ●●
●
●● ●● ●
●●●●
●
●●●
●●
●
●●●
●●●●●●
●●● ●●
●●
●●
●●●
●● ●
●
●●●● ●
●●●
●
●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●●●
● ●
●● ●●
●●
●●
●●
●
●●● ●●●
●
●●
●●
●●●●●
●● ● ●
●●
●
●
● ●●●
●
●●
●
●
●●
●●●
●
●●●
●●
●●
●●●
●
●
●●
● ●● ●
●
●
● ●
●
● ●●
●●●
●●●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●●●●
●
●● ●●●
●
●●
●
●● ●●
●
●
●●
●
●●
●●●
●●●
● ●●
●
●
●●
●
●
●
●
●●
●
● ●
●●
●
●● ● ●●
●●
●●●●
●●
●●
●●
●●● ●●●●
●
●● ●
●
●
●●
●
●●
●●●●
●
●
● ●●●
●
●●●
●●
●
● ●●
●
●
●●
●●●●●●●● ●●●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
● ●●
●●●
●
●●
● ●
● ●●
●●●● ●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●●
●●● ●
●
●●●●
●●●●
●
●
●
●
●●
●●
●●●●●●●
●●●
●
●
●●●●●
●●
●
●●
●●●
●
●
●
●●●
●
●
●
● ●
●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●●●
●●
●
●●
●●
● ●●
●●●
● ●●
●
●
●
●
●
●●
●●●●
● ●●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●●
●●●
●
● ●●●
●
●● ●
●
●●
●●●
●●●
●
●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●●●●
●
●
●●●
●●
●●
●
●
●●
●●●● ●●
●
●●●●
●● ●●
●●●
●●
●●●●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
ACF1
lumpiness
entropy
lshiftvchange
cpoints
fspo
ts
trend
linearity
curvature
spikiness
seas
onpeak
trou
gh
klscore
chan
ge.id
x
−4
−2
0
2
−2.5 0.0 2.5standardized PC1 (28.7% explained var.)
stan
dard
ized
PC
2 (1
7.3%
exp
lain
ed v
ar.)
We need a measure of the “anomalousness” of a timeseries.
1 Rank points based on their local density.2 Rank points based on whether they are within
α-convex hulls of different radius.Exploring the feature space of large time series collections Yahoo web traffic 20
What is “anomalous”
●●
●● ●●
● ●
●
●
●●
●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●●
●
●●●●
● ●●●
● ●
●
●●
●●
●
●
●●
●
●
●
● ●●
●●●●
●●
●●
●
● ●●●
●●
●● ●
●●
●
● ●
●
●●●
●
●
● ●
●● ●● ●●
●●
●●●
●
●
●●●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●
●
● ●●
● ●● ● ●●●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●●●
●●
●
●●●●●
●● ●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●● ●●
●
●●
●●
●●●●
●
●
●
●●●
●●●
●●●●●●●●
●
●●
● ●●
●
● ●●
●
●●●
●●●● ●●●●
●●
●●
●
●
●●
●
●
●
●●●● ●
●
●●
●●
●●
●●●
● ●●●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
● ●●●
●●
●●●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
● ●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●●
● ●●●●
● ●●
●
● ●●● ●●
●●
●●
● ●
●●
●
●●
●●
●
● ●●
●
●
●
● ●●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●● ●
● ●
●
●●●
●
●
●
●
●●
●●
●●
●●
●
●
●● ●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●
●●
●
●●
●
●●
●
●●
●●●●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●●●
●●
●
●
● ●
●●
●
●
●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●
●
●●
● ●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●●
●
●
●●
● ●
●●
●
●
●●
●
●
● ●●
●
●
●
●
●●●
●●●
●
● ●●●●
●● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ● ●
●
●
●●
●
●
●●
● ●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●● ●
●
●●
●
●
●
● ●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●●
●●
●
● ●
●●●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
● ●
●
●●
●
●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
● ● ●●
●●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●●●
●●
●●
●
●
●●
●●●
● ●●● ●
● ●●●
●●
●●●
●●●
●●
●
● ●● ●
●●●●●
●
●●●●
●●●
●● ●●●●
●●●●
●●●●
●●● ●
● ●●
● ●
●
●●
●
●
●
● ●●
●● ● ●●
●
● ●● ●● ●●
●
●●
●
●● ●●●●
● ●● ●
●
●●●
●
●
●●
●
●
●●
●●
●●●●
●
●
●
● ●
●●
●●
●● ●●●●
●
●
●● ●●
●●● ●●
●● ●
●
●
●
●
●
●●
●
●
●
●●
● ●
●● ●●● ●
●●●●
●
●●
●●●
●●
●
●
●●●
●●
●
●●
● ●● ● ●
●●●
● ●
●
●
●
●●
●
●
●● ●●
●●
● ●
●●
●
●● ●●●● ●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●● ●●●
●● ●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●
●
●●
● ●●
●
●
● ●
●
● ● ●
●●
●
●●
●
● ●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●
●
●●●
●●
●● ●●●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●● ●
●●
●
●●●
●
●
●
● ●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●●●
●
●●
●
●
●
●
● ●
●●●●
●
●
●
● ●
●
●●
● ●●
●
●●
●
●●
● ●
●
●
●
●
●●
●
●
●
●●
●● ●●●●
●●
●
● ●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
● ●
●
●●
●
●●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●●
●●
● ●
●●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●●●
●● ●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●
●
●
●
●●
●●
● ●●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
● ●●
●●●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●
●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●●
●
● ●
●●
●
●
●●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●● ●●●
●●●
● ●●●● ●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●●●
●●
●●●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
● ●●●
●●
●
●● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●●●
● ●
●
●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●●
●●
●●●
●●
●
●●
●
●
●
●
●
●●●●● ●
●
●
●
●●
●
●●
●●
●
●●
●
●● ●
●
●●
●●
●
● ●●
●●
●
●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●● ●
●
●
●
●●● ●
●●
●
● ●● ●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●● ●●
●
●●
●
●● ●
●●
●
●● ●●
●
●
●
● ●● ●
●●
●
● ●●
●
●●
●
●
●●●●
●●●
●
● ●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
●●●
●●●
●●
●
● ●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●● ●●
●
●●
●●
●●
●●●●
●●
●
●
●
● ●
●
●●
●●
●● ●
●
●
● ●
●
●
●
●
●
● ●●●
●●
●●
●
●
●
●●●
●
●● ●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●● ●
●
●
●
●
●
●●●
●●
●● ●
●
● ●
●● ●
●
●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●●●●●
●● ●●●
●
●
● ●● ●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●
●
●
●● ●●
●●
●●●●●●●
●●
●●●●
●
●●
●
●●
●●●
●
●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●●
● ●●●
●
●
●
●
●
●●
● ●
●
●
●●
●●●
● ●●●
●●
●
●●●●
●●
●
●
●
● ●●
●
●●
●●
●●
●●
●
●
●
●●
●
●● ●
● ●●●
●●●●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
● ●●
●
●
●● ●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●●
●●
●●
●●
●
●●
●●
●
●● ●
●●●
●●
● ●●
●●
●●
●●
●
●
●
●●●
●
●●
●
●●●
●
●● ●
●●
●●
●●
●●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●●●
●
●●●
● ●
●
●
● ●
●
●●
● ●
●
●●
● ●●●●
●●● ●
●●
●
●●●
●● ●
●●●
●
●●●
●●●
● ●
●
●
●
●●
●
●●●
●
●●
●
●
●● ●
●
●
●●
●●
●
●
●●●
●
●●●
●●
●
●●
●
●
●
● ●
●
● ●●●
●
●●●
●●
●
●●
●●●
●
●● ●●●● ●
●●
●●
●●
●
●
●
●●● ●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
● ●●
●●
●●●●
●
● ●●●●●
●
●●
●
●●●
●●●
●●
●●
●●
●
●●●● ●
●●
●
●
●●●
●●●
●
●●
●
●
●●●
●● ● ●
●●
●
●
●●
●
●●●
● ●●● ●●●
●
●
●
●●
●
● ●●
●
●●
●
●●
●
●
●●
●● ●●●●●●● ●
●●●
●●
●
●
●●
●●●
●●●●
●●●●
● ●
●
●
●●
●●
●
●●
● ●●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●
●
● ●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
● ●●●
● ●●●
●
●
●
●●
●● ●●
●
●
●
●●●
●
●●
●●●
●
●● ●
● ●●
●●
●●●
●●
● ●
●● ●
●● ●●●
●●●
●
●
●●
●●
●● ●●
●●
● ●●
●●●
●●
●●● ●
●
●●●
●●
●
● ●
●●●
●●
●
●●
●●
●
●
●●●
●●
●
●●●●●
●●●
●●●
●
●
●●●
●
●
●●●
●● ●●
● ●●●●
● ●
●●
●●
●●
●
●
●●●
●●
●
●
●
●
●
●
●●
●●● ●
●
●●
●
●●
●
●●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●●
●●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●● ●
●
●●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●●
● ●
●
●●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●● ●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●
●●
●
●
●
●●●●
●●
●●
●
●●
●
●
●
●● ●● ●●
●● ●
●
●
●
●●●
●
●● ●●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
●
●● ●
●●● ●
●●
●●
●
●●
●
●
●
●● ●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
● ●
● ●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●●
●●
●
●●●
●
●
●
●●●●●●●●
●●●●● ●●●
● ●●●
●● ●●
●●●
●
●●●
●● ●●
●
●● ●
● ●●
●
●●●
●● ●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ●
●
●●
● ●●
●●
●
●
●
●
●
●
●
●
●
● ●●●
●●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
● ●
● ●
●
●
●●
●●
●
●
●● ●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
● ●●●
●
●●
●
●
●
●
●
●●
● ●
●
●● ●●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●
●
●
●●●
●
●
●●
●
●●●●
●●
●●
●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●●
●●
●●●
●
●●
●●● ●
●●●
●●
●
● ●
●●
●
●●
●●
●● ●●
●●
●●
●●
●
●
●
●
●
●●
●
●●●
●
●●●
●
●
●●
●●●
● ●●●
●●
●
●
●●
●● ●
●●● ●
●
●
●●●
●●
●
●
●●● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●● ●●● ●●
●
●●
●●
●● ●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●
●●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
● ●●●●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●●
●
●
●
●●●●
●●
●●
●●
●
●
●
●●
●
●
●●
●●
●●
●
●
●●
●
●
●
●●●●
●●
● ●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●
●
● ●●
● ●●
●
●
●● ●● ●
●
●●●
●
●●
●
● ●●
●●
● ●●
●
●
●
● ●●●●
●
●●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●
●
●
● ●
●●●
●
●●
●●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
● ●
●● ●
●● ●● ●●●
●●● ●
●
●
●●
●
●
●
● ●
●
●●●
●
●●
●●● ●●
●●
●
●●
●
●
●
●●
●
●●
●●●
●●
●
●
●
●
●
●●
●● ●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
● ●
●●●●
●●
●
●
●●
● ●
● ●●
●
●
● ●
●
●●
●
●
● ●
●
●
●
●
●●●●
●
●
●
●●●●
●●
●
● ●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●●● ●●●●
●●
●
●
● ●
●
●
●●
●
●●●●
●●●●
●
●●
● ●
●
●
●●●
● ●●
●●
●
●●●
●
●●
●
● ● ●
● ●●
●●
●●
●
●
●●
●
●●●
●●
●●
●●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●
●● ●
●●
●
●●
●
●
●●● ●
●●●
●
●●
●●
●●
●●●●
●
●● ●
●
●
●●
●
●●●
●
●●●
●●●
●●● ●● ●●
●● ●●
● ●●
●●
● ●
●●
● ●●
● ●●●
●
● ●●●●
●●
●● ●
●
●
● ●
● ●
●●●
● ●●
●
●●●●
●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●●
●
● ●●●●
●●
●
●●
●
●
●●
●
●
●
●●●●●●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●●
●● ●●
●
●
● ●
●
●●
●
●●
●
●
●● ●
●●
●●
●
●
●● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●● ●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●●●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●● ●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
● ●
●
●●●
●
●● ●
●
●
●
●
●
●●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●●
●●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●● ●●
●
●●
●
●●
●
●
●
●●
●
●● ● ●
●
●
●
●
●●
●●
●
●●
●●
●
●
● ●●
● ●
●●
●
●
●●
●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
● ●●●
●
●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●●
●
●●
●●
●●
●●● ●● ●●
●●●●
●
●●
●
●● ●●
●
●●●
●
●●
●
●●●
●
● ● ●● ●●●
●●● ●●●●
●
●●●
●
●●
●●
● ●● ●●●● ●●●
● ●●
●
● ●●
●
●●●
●●●●
●
●●●
●
● ●
●
●
●● ●●●●●●●●●●●
●●●
●
●●
●
●
●●●
●●●
●●●●
●●●●●●
●●●
● ●●
●
●●
●●
●●●●●●●●
●●
●
●● ●●
●● ●
●●●●
●●●●
●
●●●●●
●●●●● ●●
●●●
●
●●
●
●●
●
●● ●●●
●●●
● ●●
● ●●
●
●●●
●
●●
●● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●●
●●
●●
●●
●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
● ●●
● ●
●
● ●
●
●●
●
●●
●●●
●●
● ●●●●
●●●●
●●
●
● ●●● ●
●
●●
●
●●● ●
●●
●
●●
●
●●●●
●●
●●●
●● ● ●●
●●●
●●
●●
● ●●●
●●●●
●●
●●
●●
●●●
●
●
●● ●
●
●
● ●●
●● ● ●
●
●
●
●●
●●
●●●●
●
●
● ●● ●
●●
● ●● ●●●
●●●
●●● ●●●●
●
●●●●●
●
●
●
●●
●● ●
● ● ●●
●●●
●●●
●●●●●●● ● ●
●●● ●● ●● ●●
●●
●● ●●
●
●
●
●
●
● ●
●●
●●●
●
●●
●
●●
●●● ●●●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●●
● ●●
●
● ●● ●●●
●
●
●●
●●●●
●
●●●
●●
●
●
● ●●
●
●
●
●
●
●●
● ●●●
●● ●
●●
●
●●
●
●
●●
●
●
●●●
●●●●
●●●●
●
● ●● ●●
●●●●●●● ●
●●●
●
●●● ●●
● ●●
●
●● ●
● ●
●
●● ●
●
●●●
●●
●●●●
●●
● ●●●
●
●
●
●
●●
●
●
●
●●
●
● ●●
● ●●
●
●● ●
●
●
●
●● ●
●
● ●●
●
●
●
●
● ●●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●●
●●●
●●●
●
●
●● ●●●
●
●●
●
●
●
●
●●
●●
●●
●
● ●●
●
● ●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●● ●
●●●
●●
●
●● ●●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●● ●●
●
●
●
●
●
●●
●●
●
●
● ●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●
● ●
●
●●
●●
●●
●
● ●●
●●●
●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●●
● ●●
●●●
●●●
●●●●
●●●●●●●●●
●●
●
●●
● ●●●●
●
●
●●● ●●
●●
●● ●●
●●
●●●● ●
●●● ●
●
●●
●●●
●
●●●●●
●●●
●
●●
●●
●
●●
●
●
●●●
●
●●
●●
●●●● ●
●
●●
●
●●●
●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●
● ●●
●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●● ●
●●
●● ●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●● ●●
●
●●
●●
●
●●
●●
●
● ●● ●
●●
●●
●●
●
●●●
●●●●
●●
●● ●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
● ● ●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●● ●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
● ●●●●
●
●●●
●
●
●
●●
●●
●
●
●
●●●●●●●
● ●● ●●●
●●
●
●
●●
●● ●●●
●●●
●●●
●●
●●
●●●●●●●
●●●
●● ●● ●
●
●●●
●●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
● ●● ●●●●●
●
●
●●●
●●
●
●
●●● ●●
●●●●
●● ●● ●
●
●●
● ●●●●● ●●●
●●●●●● ●●●●●● ●●
●
●●
●
●●
●
●●
●●●●
●●●●●
● ●●●●
●
●●●● ●
●●●●●●●
●●
●
●
● ●●
●●●
●●
●●● ●●
●●●●● ●●●●●
●●●● ●●
●●● ●●● ●
●
●● ●●●●
●● ●●●●●●●
●● ●● ●● ●
●●●
●
●
●● ●
●
●●●
● ●●●●
●●
●● ●●● ●
●●●●
●●●
●●●●● ●●●●
●
●●
●
●
●●●●
●●●● ●●● ●
●
●●● ●●● ●●
●●●●●●
●
●●● ●●●●●● ●●●
● ●●●●●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●●
●
●●
●
●●
●●
●● ●●
●●
●●
● ●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●●
●●
●●● ●●
●
●●
●
●
●
●
●
●●
● ●●●●● ●
●
●●●●● ●●
● ●
●
●●
●●
●●●
●
●
●
●
●●
●●
●
●
● ●
●
●●●● ●
● ●●
●
●●●
●
●
●
●●
●
●●●●
●
●
●●
● ●●
●
●
●
●●
●
●
●
●
●
● ●● ●●
●
●
● ●
●
●●
●
●
● ●●
●● ●
●
●●
●●
●●
●● ●●●
●
●●
●●●●
●
●
●●●
●
●●
●●
●●
●
●●
●●●●
●●
●
● ●● ●●●●
●
● ●● ●●
●
●●
●
●●
●
●●●
●●
●●●●
●●●●●
●●●
●
●●
●
●●●
●
●● ●
●●
●
● ●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●
●●
●
●
●
●●● ● ●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●
●●●
●
●
●●
●
● ●
●●●
●
●
●
●●
●
● ●
●
●
●
● ●●
●
●
●● ●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●
●
●
●●
●
●●
●
●●●●
●
●
●
●●● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
● ●
●●
●●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●
●●
●
● ●
●
●
● ●
●
●●
● ●
●
●
●
●●●●●
●
●● ●●
●●●
●
●●
●● ●
●
●●●
●●●
● ●● ●
●●●
●●●●● ●●
●●
●
●● ●●
●●
●
●● ●
●●
●●●
●
●●●
●●●
● ●●●
● ●●
●● ●●
●
●●●
●
●
●●
●●
●●●
●
●●
●●
●●
●
●●●● ●●●
●
●●●●●●
●●●
●●● ●●
●
●
●
●
● ●
●
●●●
●
●
●
●
●●●●●
●● ●●●●
●
●●
●● ●●
●
●● ●● ●
●●●●
●
●●●
●●
●
●●●
●●●●●●
●●● ●●
●●
●●
●●●
●● ●
●
●●●● ●
●●●
●
●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●●●
● ●
●● ●●
●●
●●
●●
●
●●● ●●●
●
●●
●●
●●●●●
●● ● ●
●●
●
●
● ●●●
●
●●
●
●
●●
●●●
●
●●●
●●
●●
●●●
●
●
●●
● ●● ●
●
●
● ●
●
● ●●
●●●
●●●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●●●●
●
●● ●●●
●
●●
●
●● ●●
●
●
●●
●
●●
●●●
●●●
● ●●
●
●
●●
●
●
●
●
●●
●
● ●
●●
●
●● ● ●●
●●
●●●●
●●
●●
●●
●●● ●●●●
●
●● ●
●
●
●●
●
●●
●●●●
●
●
● ●●●
●
●●●
●●
●
● ●●
●
●
●●
●●●●●●●● ●●●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
● ●●
●●●
●
●●
● ●
● ●●
●●●● ●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●●
●●● ●
●
●●●●
●●●●
●
●
●
●
●●
●●
●●●●●●●
●●●
●
●
●●●●●
●●
●
●●
●●●
●
●
●
●●●
●
●
●
● ●
●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●●●
●●
●
●●
●●
● ●●
●●●
● ●●
●
●
●
●
●
●●
●●●●
● ●●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●●
●●●
●
● ●●●
●
●● ●
●
●●
●●●
●●●
●
●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●●●●
●
●
●●●
●●
●●
●
●
●●
●●●● ●●
●
●●●●
●● ●●
●●●
●●
●●●●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
ACF1
lumpiness
entropy
lshiftvchange
cpoints
fspo
ts
trend
linearity
curvature
spikiness
seas
onpeak
trou
gh
klscore
chan
ge.id
x
−4
−2
0
2
−2.5 0.0 2.5standardized PC1 (28.7% explained var.)
stan
dard
ized
PC
2 (1
7.3%
exp
lain
ed v
ar.)
We need a measure of the “anomalousness” of a timeseries.
1 Rank points based on their local density.2 Rank points based on whether they are within
α-convex hulls of different radius.Exploring the feature space of large time series collections Yahoo web traffic 20
Bivariate kernel density
f̂(x;H) =1
n
n∑i=1
KH(x− Xi)
Xi ∈ a bivariate random sample {X1,X2, . . . ,Xn}KH(x) is the standard normal kernel function
H estimated by minimizing the sum of AMISE
Rank points based on f̂ values in 2d PCA space.
Exploring the feature space of large time series collections Yahoo web traffic 21
Bivariate kernel density
f̂(x;H) =1
n
n∑i=1
KH(x− Xi)
Xi ∈ a bivariate random sample {X1,X2, . . . ,Xn}KH(x) is the standard normal kernel function
H estimated by minimizing the sum of AMISE
Rank points based on f̂ values in 2d PCA space.
Exploring the feature space of large time series collections Yahoo web traffic 21
Bivariate density ranking
Exploring the feature space of large time series collections Yahoo web traffic 22
−5 0 5
−8
−6
−4
−2
02
46
pc1
pc2
1
2
3
45
Bivariate density ranking
Exploring the feature space of large time series collections Yahoo web traffic 22
010000200003000040000
0200040006000
01000020000300004000050000
010000200003000040000
010002000300040005000
S7793
S8494
S10464
S7833
S1715
2015
−02
−28
2015
−03
−01
2015
−03
−02
2015
−03
−03
2015
−03
−04
2015
−03
−05
2015
−03
−06
2015
−03
−07
2015
−03
−08
2015
−03
−09
2015
−03
−10
2015
−03
−11
2015
−03
−12
2015
−03
−13
2015
−03
−14
2015
−03
−15
2015
−03
−16
2015
−03
−17
2015
−03
−18
2015
−03
−19
2015
−03
−20
2015
−03
−21
2015
−03
−22
2015
−03
−23
2015
−03
−24
2015
−03
−25
2015
−03
−26
2015
−03
−27
2015
−03
−28
2015
−03
−29
2015
−03
−30
2015
−03
−31
2015
−04
−01
date
valu
e
α-convex hullsThe space generated by point pairs that can betouched by an empty disc of radius α.
α→∞ gives a convex hull.
Points can become isolated when α is small.
We rank points based on the value of α whenthey become isolated.
Exploring the feature space of large time series collections Yahoo web traffic 23
α-convex hullsThe space generated by point pairs that can betouched by an empty disc of radius α.
α→∞ gives a convex hull.
Points can become isolated when α is small.
We rank points based on the value of α whenthey become isolated.
Exploring the feature space of large time series collections Yahoo web traffic 23
α-convex hullsThe space generated by point pairs that can betouched by an empty disc of radius α.
α→∞ gives a convex hull.
Points can become isolated when α is small.
We rank points based on the value of α whenthey become isolated.
Exploring the feature space of large time series collections Yahoo web traffic 23
α-convex hullsThe space generated by point pairs that can betouched by an empty disc of radius α.
α→∞ gives a convex hull.
Points can become isolated when α is small.
We rank points based on the value of α whenthey become isolated.
Exploring the feature space of large time series collections Yahoo web traffic 23
α-convex hull
Exploring the feature space of large time series collections Yahoo web traffic 24
α-convex hull ranking
Exploring the feature space of large time series collections Yahoo web traffic 25
−5 0 5
−8
−6
−4
−2
02
46
●●
●● ●●
● ●●
●
●●●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●●
●
●●●●● ●●
●● ●
●
●●
●●
●
●
●●
●
●
●
● ●●
●●●●
●●
●●
●● ●
●●
●●
●● ●
●●
●
● ●
●●●
●●
●
● ●
●● ●● ●●
●●
●●●
●
●
●●●
●
●
●●●
●
●●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●
●
● ●●
● ●● ● ●●●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●●●●
●●●
●●●●●●● ●
●●●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●● ●●
●
●●
●●
●●●●
●
●
●
●●●
●●●
●●●●●●●●
●
●●
● ●●
●
● ●●
●
●●●
●●●● ●●●●
●●
●●
●
●
●●
●
●
●
●●●●●
●
●●
●●
●●
●●●
● ●●●
●
● ●●
●
●●
● ●
●
●
●
●
●
●●
●
●
● ●●●●
●●
●●
●
●
●
●●
●●●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
● ●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●●
● ●●● ●
● ●●
●
● ●●● ●●
●●
●●
● ●
●●
●
●●
●●
●
● ●●
●
●
●
● ●●●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●● ●
● ●
●
●●●
●
●
●
●●●
●●
●●
●●
●
●
●● ●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●●
●
●
●●
●
●●
●
●●
●●●●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●●●
●●
●
●
● ●
●●
●
●
●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●●
●●
● ●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●●
●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●●
● ●
●●
●
●
●●
●
●
● ●●
●
●
●
●
●●●
●●●
●
● ●●●●
●● ●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ● ●
●
●
●●
●
●
●●
● ●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●● ●●
●●
●
●
●
● ●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●●
●●●
● ●
●●●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
● ●
●
●●
●
●●●
● ●●
●●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
● ●●●
●●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●●●
●●
●●
●
●
●●
●●●
● ●●● ●
● ●●●
●●
●●●
●●●
●●
●
● ●● ●
●●●●●
●
●●●●
●●●
●● ●●●●
●●●●
●●●●
●●● ●
● ●●
● ●
●
●●
●
●
●
● ●●
●● ● ●●
●
● ●● ●●●●
●
●●
●
●● ●●●●
● ●● ●
●
●●●
●
●
●●
●
●
●●
●●
●●●●
●
●
●
● ●
●●
●●
●● ●●●●
●
●
●● ●●
●●● ●●
●● ●
●
●
●
●
●
●●
●●
●
●●
● ●
●● ●●● ●
●●●●
●
●●
●●●
●●
●●
●●●
●●
●
●●
● ●● ● ●
●●●
● ●
●
●
●
●●
●
●
●● ●●
●●
● ●
●●
●
●● ●●●● ●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●● ●●●
●● ●● ●
●●
●●●●
●
●
●
●
●●
●
●●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●
●
●●
● ●●
●
●
● ●
●
● ● ●
●●
●
●●●
● ●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
●
●●
●●● ●
●
●●
●
●
●●
●●
●
●●●
●●
●● ●●●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●● ●
●●
●
●●●
●
●
●
● ●●●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●●●
●
●●
●
●
●
●
● ●
●●●●
●
●
●
● ●
●
●●
● ●●
●
●●
●
●●
● ●●
●
●
●
●●
●
●
●
●●
●● ●●● ●●●
●
● ●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
● ●
●
●●
●●
●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●●
●
●●
● ●
●●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●●●
●●●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●
●
●
●
●●
●●
● ●●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
● ●●
●●●
●
● ●
●
●
●
●●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●
●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●●
●
● ●
●●
●
●●
●●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●● ●●●
●●●
● ●●●● ●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●●●
●●
●●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
● ●●●
●●
●
●● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●●●
● ●
●
●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●●●
●●
●●●
●●
●●
●
●
●
●
●
●
●●●●● ●
●
●
●
●●
●
●●
●●
●
●●
●
●● ●
●
●●
●●
●
● ●●
●●
●
●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●● ●
●●
●
● ●● ●
●●
●● ●
● ●●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●● ●●
●
●●
●
●● ●
●●
●
● ● ●●
●
●
●
● ●● ●●
●●
● ●●
●
●●
●
●
●●●●
●●●
●
● ●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
●● ●●
●●
●●
●
● ●●
●
●
●
●● ●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●● ●●
●●
●
●●
●●
●●●●
●●
●
●
●
● ●
●
●●
●●
●● ●
●
●
● ●
●
●●
●
●
● ●●●
●●
●●
●
●
●●●
●
●
●● ●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●● ●
●
●
●
●
●●
●●
●●
●● ●
●● ●
●● ●
●
●●
●
● ●●
●●
●
●●
●
●
●●
●
●●●●●
●● ●●●
●
●
● ●● ●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●
●●
●
●
●●●
●
●●
●
●
●
●● ●●
●●
●●●●●●●
●●
●●●●
●
●●
●
●●
●●●
●
●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●●
● ●●●
●
●
●
●
●
●●
● ●
●
●
●●
●●●
● ●●●
●●
●
●●●●
●●
●
●
●
● ●●
●
●●
●●
●●
●●
●
●
●
●●
●
●● ●
● ●●●
●●●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
● ●●
●
●
●● ●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●
● ● ●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●●
●●
●●
●●
●
●●
●●
●
●●●
●●●
●●
● ●●
●●
●●
●●
●
●
●
●●●
●
●●
●
●●●
●
●● ●
●●
●●
●●
●●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●●●
●
●●●
●●
●
●
● ●
●
●●
● ●
●
●●
● ●●●●
●●● ●
●●
●●
●●
●● ●
●●●
●
●●●
●●●
● ●
●
●
●
●●
●●●
●
●
●●
●
●
●● ●
●
●
●●
●●
●●
●●●
●
●●●
●●
●
●●
●
●
●
● ●
●
● ●●●
●
●●●
●●
●
●●
●●●
●
●● ●
● ●●
●●●
●●
●●
●
●
●
●●● ●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
● ●●
●●
●●●●
●
● ●●●●●
●
●●
●
●●●
●●●●●
●●
●●
●
●●●● ●
●●
●
●
●●●
●●●
●
●●
●
●
●●●
●● ● ●
●●
●
●
●●
●
●●●
●●●● ●●●
●
●
●
●●
●
● ●●
●
●●
●
●●
●
●
●●
●● ● ●●●●●● ●
●●●● ●●
●
●●
●●● ●●●
●●●●●
● ●
●
●●
●●
●
●
●●
● ●●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●
●
● ●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
● ●●●
● ●●●
●
●
●
●●
●● ●●
●
●
●
●●●
●
●●
●●●
●
●● ●
● ●●
●●
●●●
●●
● ●
●● ●
●● ●●●
●●●
●
●
●●
●●
●● ●●
●●
● ●●
●●●
●●
●●● ●
●
●●●
●●
●
● ●
●●●
●● ●
●●
●●
●
●
●●●
●●
●
●●●●●
●●●
●●●
●
●●●●
●
●
●●●
●● ●●
● ●●●●
●●
●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●
● ●
●●● ●
●
●●
●
●●
●
●●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●●
●●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●● ●
●
●●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●●●
● ●
●
●●
●●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●● ●
●
●
●●●●
●
●
●
●
●
●
●
●●●●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●
●●
●
●
●
●●●●
●●●
●
●
●●
●
●
●
●● ●● ●●
●● ● ●
●
●
●●●
●
●● ●●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
●
●● ●
●●● ●
●●
●●
●
●●
●
●
●
●● ●
●●
●
●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●●
● ●
● ●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●●
●●
●
●●●
●
●
●
●●●●●●●●
●●●●● ●●●
● ●●●
●● ●●●
●●●
●●●
●● ●●●
●● ●
● ●●●
●●●
●● ●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●● ●●
●●
● ●●
●●
●
●
●
●
●
●
●
●
●
● ●●●
●●●
●
●
●
●●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
● ●
● ●
●
●
●●
●●
●
●
●● ●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
● ●●●
●
●●
●
●
●
●
●
●●
● ●
●
●● ●●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●●
●
●●●
●●
●●
●
●●●●
●●
●●
●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●●
●●
●●●
●
●●
●●● ●
●●●
●●
●
● ●
●●
●
●●
●●
●● ●●
●●
●●
●●
●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●●
●●●
● ●●●
●●
●
●
●●
●● ●●
●● ●
●
●
●●●
●●
●
●
●●● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●● ●●● ●●
●
●●
●●
●●●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●●●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
● ●●●●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●●
●
●
●
●●●●
●●
●●
●●
●
●
●
●●
●●
●●
●●●
●●
●
●●
●●
●
●●●●
●●
● ●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●●
●
● ●●
● ●●
●
●
●● ●● ●
●
●● ●●
●●
●
● ●●
●●
● ●●
●
●
●
● ●●●●
●
●●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●● ●
●
●
● ●
●●●
●
●●
●●●
●●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●
● ●
●● ●
●● ●● ●●●
●●● ●
●
●
●●
●
●
●
● ●
●
●●●
●
●●
●●● ●●
●●
●
●●
●
●
●
●●
●
●●
●●●
●●
●
●
●
●
●
●●
●● ●
●
●
●●
●
●
●
●●●
●●●
●●
●
●
●
● ●
●●●●
●●
●
●
●●
● ●
● ●●
●
●
● ●
●
●●
●
●
● ●
●
●
●
●
●●●●
●●
●
●●●●●●
●
● ●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●●● ●●●●
●●
●
●
● ●
●
●
● ●●
●●●●
●●●●
●
●●
● ●
●
●
●●●
● ●●
●●
●
●●●●
●●
●
● ● ●
● ●●
●●
●●
●
●
●
●●
●●●
●●
●●
●●
●
●
●●
●●
●●
●
●
●
●●
●
●●
●
●● ●
●●
●
●●●
●
●●● ●●●●
●
●●
●●
●●
●●●●
●
●● ●
●
●
●●
●
●●●
●
●●●
●●●
●●● ●● ●●
●● ●●
● ●●
● ●
●
●
●●
● ●●
● ●●●
●● ●●●
●●
●
●● ●
●
●
● ●● ●
●●●
● ●●
●
●● ●●●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●●
●● ●
●●●●
●
●
●●
●
●●
●
●
●
●
●●●●●●
● ●●
●
●
●●
●
●●
●
●
●
●●
●●
●● ●●
●
●● ●
●
●●
●
●●
●
●
●● ●
●●
●●
●
●
●● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●● ●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●●●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●● ●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
● ●
●
●●●
●
●● ●
●
●
●
●
●
●●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●●
●●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●
● ●●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●● ●●
●
●●
●
●●
●
●
●
●●
●
●● ● ●
●
●
●
●●
●
●●
●
●●
●●
●
●
● ●●
● ●
●●
●
●
●●●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●● ●
●●
● ●●●
●
●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●●
●
●●
●●●●
●●● ●● ●●
●●●●
●
●●
●
●● ●●
●
●●●
●●●
●
●●●
●
● ● ●● ●●●
●●● ●●●●
●
●●●
●
●●
●●● ●● ●●●●●
●●● ●●
●● ●
●
●
●●●
●●●●
●
●●●
●
● ●
●
●
●● ●●●●●●●●●●●
●●●
●
●●
●
●
●●●
●●●
●●●●
●●●●●●
●●●
● ●●
●
●●
● ●●●●●●●●●
●●
●●● ●
●●
●●●●●
●●●●●
●
●●●●●
●●●●● ●● ●
●●
●
●●
●
●●
●
●● ●●●
●●●
● ●●
● ●●
●
●●●
●
●●
●● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
● ●●
●●●
●●
●
●●
●●
●●
●●
●●
●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
● ●
●
●●
●
●
●
●
●●
●
● ●●
● ●
●
● ●
●
●●●
●●
●●●
●●
● ●●●●
●●●●
●●
●
● ●●● ●
●
●●
●
●●● ●●
●
●
●●
●●●●
●
●
●
●●●
●● ● ●● ●●
●
●●
●●
● ●●●
●●●●
●●
●●
●●
●●●
●
●
●● ●
●
●
● ●●
●● ● ●
●
●
●
●●
●●
●●●●
●
●
● ●● ●
●●
● ●● ●●●●
●●●
●● ●●●●
●
●●●●●
●
●
●
●●●
● ●● ● ●●
●●●
●●●
●●●●●●● ● ●
●●● ●● ●● ●●
●●
●● ●●
●●
●
●●
● ●
●●
●●● ●
●●
●
●●
●●● ●●●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●●
●● ●●
●
● ●● ●●●
●
●
●●
●● ●●
●
●●●
●●
●
●
● ●●
●
●
●
●
●
●●
● ●●●
●● ●
●●
●
●●
●
●
●●
●
●
●●●
●●●●
●●●●
●
● ●● ●●●●
●●●●● ●●
●●
●
●●● ●●
● ●●
●
●● ●
● ●
●
●● ●
●
●● ●●
●●●●
●●
●● ●
●●
●
●
●
●
● ●
●
●
●
●●
●
● ●●
● ●●
●●● ●
●
●
●
●● ●
●
● ●●
●
●
●●
● ●●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●●●●●
●●●
●
●
●● ●●●
●
●●
●
●
●
●
●●
●●
●●
●
● ●●
●
● ●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●●●●
●●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●● ●
●●●
●●
●
●● ●●●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
●● ●●
●
●
●
●
●
●●
●●
●
●
● ●
●
●●
●
● ●●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●
●
●
●
●●●●
●
●
● ●
●
●●
●●
●●
●
● ●●●
●●●
●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●● ●●
●●●
●●●
●●●●
●●●●●●●●●
●●
●
●●
● ●●●●
●
●
●●● ●●
●●
●● ●●
●●
●●●● ●
●●● ●
●
●●
●●●
●
●●●●●
●●●●
●●
●●●
●●
●
●
●●●
●
●●
●●
●●●● ●
●
●●
●
●●●
●
●●
●
●
●●●
●
●●
●●
●
●●●
●
●
● ●● ●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●
●
● ●●●
●● ●
●●
●● ●
●●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●●
● ●
●
●
●●●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●● ●●
●
●●
●●
●
●●
●●
●
● ●● ●
●●
●●
●●
●
●●●●
●●●
●●
●● ●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
● ● ●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●● ●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●●
●
●
● ●
●
● ●●●●
●
●●●
●
●
●
●●
●●
●
●
●
●●●●●●●
● ●● ●●●
●●
●
●
●●
●● ●●●
●●●
●●●
●●
●●●●
●●●●●●●●
●● ●● ●
●●
●●●●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●●●●
●
● ●● ●●●●●
●
●
●●●
●●
●●
●●● ●●
●●●●
●● ●● ●
●
●●
● ●●●●● ●●●
●●●●●● ●●●●●● ●●
●
●●
●●●
●
●●
●●●●
●●●●●
● ●●●●
●●●●
● ●●●●●
●●●
●●
●
●
● ●●
●●●●
●●
●● ●●●●●
●● ●●●●●
●●●● ●●
●●● ●●● ●
●
●● ●●●●
●● ●●●●●●●
●●●● ●● ●
●●●
●
●
●● ●
●
●●●
● ●●●●
●●
●● ●●● ●
●●●●
●●●
●●●●● ●●●●
●
●●
●●
●●●●
●●●● ●●● ●
●
●●● ●●● ●●
●●●●●●
●
●●● ●●●●●● ●●●
● ●●●●●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●●
●
●●
●
●●
●●
●●●●
●●
●●
● ●
●
●
●●
●
●
●
●
●●
●
●
●●
●●●
●●
●●●
●● ●●
●
●●
●
●
●
●●
●●
● ●●●●● ●
●
●●●●● ●●
● ●
●
●●
●●
●●●
●
●
●
●
●●
●●●
●
● ●
●
●●●● ●
● ●●
●
●●●
●●
●
●●
●
●●●●
●
●
●●
● ●●
●
●
●
●●
●
●
●
●
●● ●● ●●
●
●
● ●
●
●●
●
●
● ●● ●● ●
●
●●
●●
●●
●● ●●●
●
●●
●●●●
●
●
●●●
●
●●
●●
●●
●
●●
●●●●
●●
●
● ●●●●●●
●● ●
●●●
●
●●
●
●●
●
●●●
●●
●●●●
●●●●●
●●●
●
●●
●
●●●
●
●● ●
●●
●
● ●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●●●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●
●●
●
●
●
●●● ● ●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●●● ●
●●
●
●
●
●
●●●
●
●
●●
●
● ●
●●●
●●
●
●●
●
● ●
●
●
●
● ●●
●
●
●● ●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●
●
●
●●
●
●●
●●●●
●●
●
●
●●● ●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●
●● ●
●
●
● ●
●●
●●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●●
●
●
●
●
●●
●
● ●●
●●
●
● ●
●
●
● ●
●
●●
● ●
●
●
●
●●●●●
●
●● ●●
●●●
●
●●
●● ●
●
●●●
●●●
● ●● ●●
●●●●
●●● ●●
●●
●
●● ●●
●●
●
●● ●
●●
●●●
●
●●●
●●●
● ●●●
● ●●●● ●●
●
●●●
●
●
●●
● ●●●●
●
●●
● ●●●
●
●●●● ●●●
●
●●●●●●
●●●
●●● ●●
●●
●
●
● ●●
●●●
●
●
●
●
●●●●●
●● ●●●●
●
●●
●● ●●
●
●● ●● ●
●●●●
●
●●●
●●
●
●●●
●●●●●●
●●● ●●
●●
●●●●●
●● ●
●
●●●● ●●
●●●
●●●
●●
●●
●●●
●●
●●
●●●
●●
●●● ●●
● ●
●● ●●●
●●
●●●
●
●●● ●●●
●
●●
●●
●●●●●
●●● ●
●●
●
●
● ●●●
●
●●
●
●●●
●●●
●
●●●
●●
●●
●●●
●
●
●●● ●
● ●●
●
● ●●
● ●●
●●●
● ●●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●●●
●
●● ●●●
●
●●
●
●● ●●
●
●
●●
●
●●
●●●
●●●
● ●●
●
●
●●
●
●
●
●
●●
●
● ●●●
●
●● ● ●●
●●
●●●●
●●
●●
●●
●●● ●●●●●
●● ●
●
●●●
●
●●
●●●●
●
●
● ●●●
●
●●●
●●
●
● ●●
●
●
●●
●●●●●●●● ●●●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●●
●●
●
●●
●
●
● ●●
●●●
●
●●
● ●
● ●●
●●●● ●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●●
●●● ●
●
●●●●
●●●●
●
●
●
●
●● ●
●
●●●●●●●
●●●
●
●
●●●●●
●●
●
●●●●
●
●
●
●
●●●●
●
●
● ●
●●
●●
●●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●●●
●●
●
●●
●●
● ●●
●●●
● ●●
●
●
●
●
●
●●
●●●●
● ● ●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●●
●●●
●
● ●●●
●
●● ●●
●●
●●●
●●●
●
●
●●
●
●
●
●●
● ●
●●●
●
●●
●
●●●●
●
●
●●●
●●
●●
●
●
●●
●●●● ●●
●
●●●●●● ●
●●●●
●●
●●●●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
12
3
4
5
α-convex hull ranking
Exploring the feature space of large time series collections Yahoo web traffic 25
01000020000300004000050000
010000200003000040000
0200040006000
010002000300040005000
0100002000030000
S10464
S7793
S8494
S1715
S7826
2015
−02
−28
2015
−03
−01
2015
−03
−02
2015
−03
−03
2015
−03
−04
2015
−03
−05
2015
−03
−06
2015
−03
−07
2015
−03
−08
2015
−03
−09
2015
−03
−10
2015
−03
−11
2015
−03
−12
2015
−03
−13
2015
−03
−14
2015
−03
−15
2015
−03
−16
2015
−03
−17
2015
−03
−18
2015
−03
−19
2015
−03
−20
2015
−03
−21
2015
−03
−22
2015
−03
−23
2015
−03
−24
2015
−03
−25
2015
−03
−26
2015
−03
−27
2015
−03
−28
2015
−03
−29
2015
−03
−30
2015
−03
−31
2015
−04
−01
date
valu
e
HDR versus α-convex hull
HDR boxplot
−5 0 5
−8
−6
−4
−2
02
46
pc1
pc2
1
2
3
45
α-convex hull
−5 0 5−
8−
6−
4−
20
24
6
●●
●● ●●
● ●●
●
●●●
●
●
●●
●●●
●●
●
●
●●
●
●●●
●●
●
●●●●● ●●
●● ●
●
●●
●●
●
●
●●
●
●
●
● ●●
●●●●
●●
●●
●● ●
●●
●●
●● ●
●●
●
● ●
●●●
●●
●
● ●
●● ●● ●●
●●
●●●
●
●
●●●
●
●
●●●
●
●●
●●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●
●
● ●●
● ●● ● ●●●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●●●●
●●●
●●●●●●● ●
●●●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●● ●●
●
●●
●●
●●●●
●
●
●
●●●
●●●
●●●●●●●●
●
●●
● ●●
●
● ●●
●
●●●
●●●● ●●●●
●●
●●
●
●
●●
●
●
●
●●●●●
●
●●
●●
●●
●●●
● ●●●
●
● ●●
●
●●
● ●
●
●
●
●
●
●●
●
●
● ●●●●
●●
●●
●
●
●
●●
●●●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
● ●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●●
● ●●● ●
● ●●
●
● ●●● ●●
●●
●●
● ●
●●
●
●●
●●
●
● ●●
●
●
●
● ●●●
●
●
●
●●
●●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●● ●
● ●
●
●●●
●
●
●
●●●
●●
●●
●●
●
●
●● ●●
●
●
●●●
●●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●● ●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●●
●
●
●●
●
●●
●
●●
●●●●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●
●●●
●●
●
●
● ●
●●
●
●
●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●●
●●
● ●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●●
●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●●
● ●
●●
●
●
●●
●
●
● ●●
●
●
●
●
●●●
●●●
●
● ●●●●
●● ●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●● ● ●
●
●
●●
●
●
●●
● ●●
●●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●●
●● ●●
●●
●
●
●
● ●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●
●
●●
●●
●●●
● ●
●●●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
● ●
●
●●
●
●●●
● ●●
●●
●
●●
●
●
●●
●
●
●●
●
●●
●●
●
● ●●●
●●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●●●
●●
●●
●
●
●●
●●●
● ●●● ●
● ●●●
●●
●●●
●●●
●●
●
● ●● ●
●●●●●
●
●●●●
●●●
●● ●●●●
●●●●
●●●●
●●● ●
● ●●
● ●
●
●●
●
●
●
● ●●
●● ● ●●
●
● ●● ●●●●
●
●●
●
●● ●●●●
● ●● ●
●
●●●
●
●
●●
●
●
●●
●●
●●●●
●
●
●
● ●
●●
●●
●● ●●●●
●
●
●● ●●
●●● ●●
●● ●
●
●
●
●
●
●●
●●
●
●●
● ●
●● ●●● ●
●●●●
●
●●
●●●
●●
●●
●●●
●●
●
●●
● ●● ● ●
●●●
● ●
●
●
●
●●
●
●
●● ●●
●●
● ●
●●
●
●● ●●●● ●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●● ●●●
●● ●● ●
●●
●●●●
●
●
●
●
●●
●
●●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●
●
●●
● ●●
●
●
● ●
●
● ● ●
●●
●
●●●
● ●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●●
●
●●
●●● ●
●
●●
●
●
●●
●●
●
●●●
●●
●● ●●●●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●● ●
●●
●
●●●
●
●
●
● ●●●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●●●
●
●●
●
●
●
●
● ●
●●●●
●
●
●
● ●
●
●●
● ●●
●
●●
●
●●
● ●●
●
●
●
●●
●
●
●
●●
●● ●●● ●●●
●
● ●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
● ●
●
●●
●●
●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●●
●
●●
● ●
●●
● ●
● ●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●●●
●●●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●
●
●
●
●●
●●
● ●●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
● ●●
●●●
●
● ●
●
●
●
●●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●
●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●●
●
● ●
●●
●
●●
●●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●● ●●●
●●●
● ●●●● ●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●●●
●●
●●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
● ●●●
●●
●
●● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●● ●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●●●
● ●
●
●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●●●
●●
●●●
●●
●●
●
●
●
●
●
●
●●●●● ●
●
●
●
●●
●
●●
●●
●
●●
●
●● ●
●
●●
●●
●
● ●●
●●
●
●● ●
●●
●
●●●
●
●
●
●
●
●●
●●
● ●
●
●
●
●●
●
●● ●
●●
●
● ●● ●
●●
●● ●
● ●●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●● ●●
●
●●
●
●● ●
●●
●
● ● ●●
●
●
●
● ●● ●●
●●
● ●●
●
●●
●
●
●●●●
●●●
●
● ●
●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●●
●
●● ●●
●●
●●
●
● ●●
●
●
●
●● ●
●
●●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●● ●●
●●
●
●●
●●
●●●●
●●
●
●
●
● ●
●
●●
●●
●● ●
●
●
● ●
●
●●
●
●
● ●●●
●●
●●
●
●
●●●
●
●
●● ●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●●
●●
●
●●
●● ●
●
●
●
●
●●
●●
●●
●● ●
●● ●
●● ●
●
●●
●
● ●●
●●
●
●●
●
●
●●
●
●●●●●
●● ●●●
●
●
● ●● ●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●
●●
●
●
●●●
●
●●
●
●
●
●● ●●
●●
●●●●●●●
●●
●●●●
●
●●
●
●●
●●●
●
●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●●
●●
●●
●
●●
●●
●●●●
● ●●●
●
●
●
●
●
●●
● ●
●
●
●●
●●●
● ●●●
●●
●
●●●●
●●
●
●
●
● ●●
●
●●
●●
●●
●●
●
●
●
●●
●
●● ●
● ●●●
●●●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
● ●●
●
●
●● ●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●
● ● ●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●●
●●
●●
●●
●
●●
●●
●
●●●
●●●
●●
● ●●
●●
●●
●●
●
●
●
●●●
●
●●
●
●●●
●
●● ●
●●
●●
●●
●●●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●●●
●
●●●
●●
●
●
● ●
●
●●
● ●
●
●●
● ●●●●
●●● ●
●●
●●
●●
●● ●
●●●
●
●●●
●●●
● ●
●
●
●
●●
●●●
●
●
●●
●
●
●● ●
●
●
●●
●●
●●
●●●
●
●●●
●●
●
●●
●
●
●
● ●
●
● ●●●
●
●●●
●●
●
●●
●●●
●
●● ●
● ●●
●●●
●●
●●
●
●
●
●●● ●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
● ●●
●●
●●●●
●
● ●●●●●
●
●●
●
●●●
●●●●●
●●
●●
●
●●●● ●
●●
●
●
●●●
●●●
●
●●
●
●
●●●
●● ● ●
●●
●
●
●●
●
●●●
●●●● ●●●
●
●
●
●●
●
● ●●
●
●●
●
●●
●
●
●●
●● ● ●●●●●● ●
●●●● ●●
●
●●
●●● ●●●
●●●●●
● ●
●
●●
●●
●
●
●●
● ●●
●
●●
●●
●●
●●
●●●
●
●●
●●●
●●
●
●
●
● ●●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
● ●●●
● ●●●
●
●
●
●●
●● ●●
●
●
●
●●●
●
●●
●●●
●
●● ●
● ●●
●●
●●●
●●
● ●
●● ●
●● ●●●
●●●
●
●
●●
●●
●● ●●
●●
● ●●
●●●
●●
●●● ●
●
●●●
●●
●
● ●
●●●
●● ●
●●
●●
●
●
●●●
●●
●
●●●●●
●●●
●●●
●
●●●●
●
●
●●●
●● ●●
● ●●●●
●●
●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●
● ●
●●● ●
●
●●
●
●●
●
●●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●●
● ●●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●●
●●●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●● ●
●
●●●
●●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●●●
● ●
●
●●
●●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●● ●
●
●
●●●●
●
●
●
●
●
●
●
●●●●
● ●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●
●●
●
●
●
●●●●
●●●
●
●
●●
●
●
●
●● ●● ●●
●● ● ●
●
●
●●●
●
●● ●●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
●
●● ●
●●● ●
●●
●●
●
●●
●
●
●
●● ●
●●
●
●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●●
● ●
● ●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●●
●●
●
●●●
●
●
●
●●●●●●●●
●●●●● ●●●
● ●●●
●● ●●●
●●●
●●●
●● ●●●
●● ●
● ●●●
●●●
●● ●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●● ●●
●●
● ●●
●●
●
●
●
●
●
●
●
●
●
● ●●●
●●●
●
●
●
●●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
● ●
● ●
●
●
●●
●●
●
●
●● ●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
● ●●●
●
●●
●
●
●
●
●
●●
● ●
●
●● ●●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●●
●
●●●
●●
●●
●
●●●●
●●
●●
●●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●●
●●
●●●
●
●●
●●● ●
●●●
●●
●
● ●
●●
●
●●
●●
●● ●●
●●
●●
●●
●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●●
●●●
● ●●●
●●
●
●
●●
●● ●●
●● ●
●
●
●●●
●●
●
●
●●● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●● ●●● ●●
●
●●
●●
●●●
●
●
●●●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●●●●
●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
● ●●●●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●●
●
●
●
●●●●
●●
●●
●●
●
●
●
●●
●●
●●
●●●
●●
●
●●
●●
●
●●●●
●●
● ●●
●●
●
●●
●●
●
●●●
●●
●●●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●●
●
● ●●
● ●●
●
●
●● ●● ●
●
●● ●●
●●
●
● ●●
●●
● ●●
●
●
●
● ●●●●
●
●●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●● ●
●
●
● ●
●●●
●
●●
●●●
●●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●
●
● ●
●● ●
●● ●● ●●●
●●● ●
●
●
●●
●
●
●
● ●
●
●●●
●
●●
●●● ●●
●●
●
●●
●
●
●
●●
●
●●
●●●
●●
●
●
●
●
●
●●
●● ●
●
●
●●
●
●
●
●●●
●●●
●●
●
●
●
● ●
●●●●
●●
●
●
●●
● ●
● ●●
●
●
● ●
●
●●
●
●
● ●
●
●
●
●
●●●●
●●
●
●●●●●●
●
● ●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●●● ●●●●
●●
●
●
● ●
●
●
● ●●
●●●●
●●●●
●
●●
● ●
●
●
●●●
● ●●
●●
●
●●●●
●●
●
● ● ●
● ●●
●●
●●
●
●
●
●●
●●●
●●
●●
●●
●
●
●●
●●
●●
●
●
●
●●
●
●●
●
●● ●
●●
●
●●●
●
●●● ●●●●
●
●●
●●
●●
●●●●
●
●● ●
●
●
●●
●
●●●
●
●●●
●●●
●●● ●● ●●
●● ●●
● ●●
● ●
●
●
●●
● ●●
● ●●●
●● ●●●
●●
●
●● ●
●
●
● ●● ●
●●●
● ●●
●
●● ●●●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●●
●● ●
●●●●
●
●
●●
●
●●
●
●
●
●
●●●●●●
● ●●
●
●
●●
●
●●
●
●
●
●●
●●
●● ●●
●
●● ●
●
●●
●
●●
●
●
●● ●
●●
●●
●
●
●● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●● ●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
● ●
●●
●
●
●●
●
●●
●
●●
●
●●●●
●●
●●
●●
●●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●●●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●● ●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●
● ●
●
●●●
●
●● ●
●
●
●
●
●
●●
●●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
● ●●
●●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
● ●
● ●●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
● ●
●
●
●●●
●
●
●
●
●
●
●●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●● ●●
●
●●
●
●●
●
●
●
●●
●
●● ● ●
●
●
●
●●
●
●●
●
●●
●●
●
●
● ●●
● ●
●●
●
●
●●●●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●● ●
●●
● ●●●
●
●
●
●●
●
●
● ●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
● ●●
●
●●
●●●●
●●● ●● ●●
●●●●
●
●●
●
●● ●●
●
●●●
●●●
●
●●●
●
● ● ●● ●●●
●●● ●●●●
●
●●●
●
●●
●●● ●● ●●●●●
●●● ●●
●● ●
●
●
●●●
●●●●
●
●●●
●
● ●
●
●
●● ●●●●●●●●●●●
●●●
●
●●
●
●
●●●
●●●
●●●●
●●●●●●
●●●
● ●●
●
●●
● ●●●●●●●●●
●●
●●● ●
●●
●●●●●
●●●●●
●
●●●●●
●●●●● ●● ●
●●
●
●●
●
●●
●
●● ●●●
●●●
● ●●
● ●●
●
●●●
●
●●
●● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
● ●●
●●●
●●
●
●●
●●
●●
●●
●●
●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●● ●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●● ●
●
●●
● ●
●
●●
●
●
●
●
●●
●
● ●●
● ●
●
● ●
●
●●●
●●
●●●
●●
● ●●●●
●●●●
●●
●
● ●●● ●
●
●●
●
●●● ●●
●
●
●●
●●●●
●
●
●
●●●
●● ● ●● ●●
●
●●
●●
● ●●●
●●●●
●●
●●
●●
●●●
●
●
●● ●
●
●
● ●●
●● ● ●
●
●
●
●●
●●
●●●●
●
●
● ●● ●
●●
● ●● ●●●●
●●●
●● ●●●●
●
●●●●●
●
●
●
●●●
● ●● ● ●●
●●●
●●●
●●●●●●● ● ●
●●● ●● ●● ●●
●●
●● ●●
●●
●
●●
● ●
●●
●●● ●
●●
●
●●
●●● ●●●
●
●●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●●
●● ●●
●
● ●● ●●●
●
●
●●
●● ●●
●
●●●
●●
●
●
● ●●
●
●
●
●
●
●●
● ●●●
●● ●
●●
●
●●
●
●
●●
●
●
●●●
●●●●
●●●●
●
● ●● ●●●●
●●●●● ●●
●●
●
●●● ●●
● ●●
●
●● ●
● ●
●
●● ●
●
●● ●●
●●●●
●●
●● ●
●●
●
●
●
●
● ●
●
●
●
●●
●
● ●●
● ●●
●●● ●
●
●
●
●● ●
●
● ●●
●
●
●●
● ●●
●●
●●
●●
●
●●
●
●●●
●
●
●●
●●
●●●●●
●●●
●
●
●● ●●●
●
●●
●
●
●
●
●●
●●
●●
●
● ●●
●
● ●
●●
●●
●●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●●●●
●●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
● ●● ●
●●●
●●
●
●● ●●●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
●● ●●
●
●
●
●
●
●●
●●
●
●
● ●
●
●●
●
● ●●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●
●
●
●
●●●●
●
●
● ●
●
●●
●●
●●
●
● ●●●
●●●
●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●● ●●
●●●
●●●
●●●●
●●●●●●●●●
●●
●
●●
● ●●●●
●
●
●●● ●●
●●
●● ●●
●●
●●●● ●
●●● ●
●
●●
●●●
●
●●●●●
●●●●
●●
●●●
●●
●
●
●●●
●
●●
●●
●●●● ●
●
●●
●
●●●
●
●●
●
●
●●●
●
●●
●●
●
●●●
●
●
● ●● ●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●●
●
●
●
●
●
● ●●●
●● ●
●●
●● ●
●●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●●
● ●
●
●
●●●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●●
●●●
●● ●●
●
●●
●●
●
●●
●●
●
● ●● ●
●●
●●
●●
●
●●●●
●●●
●●
●● ●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
● ● ●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●● ●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●●
●
●
● ●
●
● ●●●●
●
●●●
●
●
●
●●
●●
●
●
●
●●●●●●●
● ●● ●●●
●●
●
●
●●
●● ●●●
●●●
●●●
●●
●●●●
●●●●●●●●
●● ●● ●
●●
●●●●●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●●●●
●
● ●● ●●●●●
●
●
●●●
●●
●●
●●● ●●
●●●●
●● ●● ●
●
●●
● ●●●●● ●●●
●●●●●● ●●●●●● ●●
●
●●
●●●
●
●●
●●●●
●●●●●
● ●●●●
●●●●
● ●●●●●
●●●
●●
●
●
● ●●
●●●●
●●
●● ●●●●●
●● ●●●●●
●●●● ●●
●●● ●●● ●
●
●● ●●●●
●● ●●●●●●●
●●●● ●● ●
●●●
●
●
●● ●
●
●●●
● ●●●●
●●
●● ●●● ●
●●●●
●●●
●●●●● ●●●●
●
●●
●●
●●●●
●●●● ●●● ●
●
●●● ●●● ●●
●●●●●●
●
●●● ●●●●●● ●●●
● ●●●●●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●●
●
●●
●
●●
●●
●●●●
●●
●●
● ●
●
●
●●
●
●
●
●
●●
●
●
●●
●●●
●●
●●●
●● ●●
●
●●
●
●
●
●●
●●
● ●●●●● ●
●
●●●●● ●●
● ●
●
●●
●●
●●●
●
●
●
●
●●
●●●
●
● ●
●
●●●● ●
● ●●
●
●●●
●●
●
●●
●
●●●●
●
●
●●
● ●●
●
●
●
●●
●
●
●
●
●● ●● ●●
●
●
● ●
●
●●
●
●
● ●● ●● ●
●
●●
●●
●●
●● ●●●
●
●●
●●●●
●
●
●●●
●
●●
●●
●●
●
●●
●●●●
●●
●
● ●●●●●●
●● ●
●●●
●
●●
●
●●
●
●●●
●●
●●●●
●●●●●
●●●
●
●●
●
●●●
●
●● ●
●●
●
● ●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
● ●
●●●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●
●●
●
●
●
●●● ● ●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●●● ●
●●
●
●
●
●
●●●
●
●
●●
●
● ●
●●●
●●
●
●●
●
● ●
●
●
●
● ●●
●
●
●● ●
●
●●
●
●●●
●
●●●
●●●
●
●●
●
●
●
●
●●
●
●●
●●●●
●●
●
●
●●● ●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●
●● ●
●
●
● ●
●●
●●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●●
●
●
●
●
●●
●
● ●●
●●
●
● ●
●
●
● ●
●
●●
● ●
●
●
●
●●●●●
●
●● ●●
●●●
●
●●
●● ●
●
●●●
●●●
● ●● ●●
●●●●
●●● ●●
●●
●
●● ●●
●●
●
●● ●
●●
●●●
●
●●●
●●●
● ●●●
● ●●●● ●●
●
●●●
●
●
●●
● ●●●●
●
●●
● ●●●
●
●●●● ●●●
●
●●●●●●
●●●
●●● ●●
●●
●
●
● ●●
●●●
●
●
●
●
●●●●●
●● ●●●●
●
●●
●● ●●
●
●● ●● ●
●●●●
●
●●●
●●
●
●●●
●●●●●●
●●● ●●
●●
●●●●●
●● ●
●
●●●● ●●
●●●
●●●
●●
●●
●●●
●●
●●
●●●
●●
●●● ●●
● ●
●● ●●●
●●
●●●
●
●●● ●●●
●
●●
●●
●●●●●
●●● ●
●●
●
●
● ●●●
●
●●
●
●●●
●●●
●
●●●
●●
●●
●●●
●
●
●●● ●
● ●●
●
● ●●
● ●●
●●●
● ●●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
● ●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●●●●
●
●● ●●●
●
●●
●
●● ●●
●
●
●●
●
●●
●●●
●●●
● ●●
●
●
●●
●
●
●
●
●●
●
● ●●●
●
●● ● ●●
●●
●●●●
●●
●●
●●
●●● ●●●●●
●● ●
●
●●●
●
●●
●●●●
●
●
● ●●●
●
●●●
●●
●
● ●●
●
●
●●
●●●●●●●● ●●●●
●●
●
●
●●●
●●
●●
●●
●
●●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●
●●●
●●
●
●●
●
●
● ●●
●●●
●
●●
● ●
● ●●
●●●● ●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●●●
●
●
●●●
●●● ●
●
●●●●
●●●●
●
●
●
●
●● ●
●
●●●●●●●
●●●
●
●
●●●●●
●●
●
●●●●
●
●
●
●
●●●●
●
●
● ●
●●
●●
●●
●
●
●●●
●
●●●
●
●
●
●●●●
●●
●
●
●●●●
●●
●
●●
●●
● ●●
●●●
● ●●
●
●
●
●
●
●●
●●●●
● ● ●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
●●
●●●
●●●
●
● ●●●
●
●● ●●
●●
●●●
●●●
●
●
●●
●
●
●
●●
● ●
●●●
●
●●
●
●●●●
●
●
●●●
●●
●●
●
●
●●
●●●● ●●
●
●●●●●● ●
●●●●
●●
●●●●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
12
3
4
5
Exploring the feature space of large time series collections Yahoo web traffic 26
Top 5 anomalous time series
HDR0
10000200003000040000
0200040006000
01000020000300004000050000
010000200003000040000
010002000300040005000
S7793
S8494
S10464
S7833
S1715
2015
−02
−28
2015
−03
−01
2015
−03
−02
2015
−03
−03
2015
−03
−04
2015
−03
−05
2015
−03
−06
2015
−03
−07
2015
−03
−08
2015
−03
−09
2015
−03
−10
2015
−03
−11
2015
−03
−12
2015
−03
−13
2015
−03
−14
2015
−03
−15
2015
−03
−16
2015
−03
−17
2015
−03
−18
2015
−03
−19
2015
−03
−20
2015
−03
−21
2015
−03
−22
2015
−03
−23
2015
−03
−24
2015
−03
−25
2015
−03
−26
2015
−03
−27
2015
−03
−28
2015
−03
−29
2015
−03
−30
2015
−03
−31
2015
−04
−01
date
valu
e
α-convex hull0
1000020000300004000050000
010000200003000040000
0200040006000
010002000300040005000
0100002000030000
S10464
S7793
S8494
S1715
S7826
2015
−02
−28
2015
−03
−01
2015
−03
−02
2015
−03
−03
2015
−03
−04
2015
−03
−05
2015
−03
−06
2015
−03
−07
2015
−03
−08
2015
−03
−09
2015
−03
−10
2015
−03
−11
2015
−03
−12
2015
−03
−13
2015
−03
−14
2015
−03
−15
2015
−03
−16
2015
−03
−17
2015
−03
−18
2015
−03
−19
2015
−03
−20
2015
−03
−21
2015
−03
−22
2015
−03
−23
2015
−03
−24
2015
−03
−25
2015
−03
−26
2015
−03
−27
2015
−03
−28
2015
−03
−29
2015
−03
−30
2015
−03
−31
2015
−04
−01
date
valu
e
Exploring the feature space of large time series collections Yahoo web traffic 27
Outline
1 M3 competition data
2 Yahoo web traffic
3 What next?
Exploring the feature space of large time series collections What next? 28
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
What next?
Develop a more comprehensive set of featuresthat are reliable measures and fast to compute.e.g., for finance data.Consider application to functional data fromother contexts (not time series).Is PCA the right approach? Perhaps we shoulduse multidimensional scaling? Or somethingelse?Should we use more than 2 PC dimensions?Develop dynamic and interactive visualizationtools.Make methods available in an R package.
Some of the methods are already available in theanomalous package for R on github.
Exploring the feature space of large time series collections What next? 29
Further information
Exploring the feature space of large time series collections What next? 30
å Papers: robjhyndman.com
å Code: github.com/robjhyndman
å Email: [email protected]