combined estimator fourier series and spline truncated in...
TRANSCRIPT
Applied Mathematical Sciences, Vol. 9, 2015, no. 100, 4997 - 5010
HIKARI Ltd, www.m-hikari.com
http://dx.doi.org/10.12988/ams.2015.55394
Combined Estimator Fourier Series and
Spline Truncated in
Multivariable Nonparametric Regression
I. Wayan Sudiarsa
Department of Statistics, Faculty of Mathematics and Natural Sciences
Institut Teknologi Sepuluh Nopember (ITS)
Arif Rahman Hakim, Surabaya 60111, Indonesia
and
Department of Mathematics, Faculty of Mathematics and Natural Sciences,
Institut Keguruan dan Ilmu Pendidikan (IKIP) PGRI Bali, Indonesia
I. Nyoman Budiantara, Suhartono and Santi Wulan Purnami
Department of Statistics, Faculty of Mathematics and Natural Sciences
Institut Teknologi Sepuluh Nopember (ITS)
Arif Rahman Hakim, Surabaya 60111, Indonesia
Copyright © 2015 I. Wayan Sudiarsa et al. This article is distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
Multivariable additive nonparametric regression model is a nonparametric
regression model that involves more than one predictor and has additively
separable function on each predictor. There are many functions that can be
used on nonparametric regression models, such as the kernel, splines,
wavelets, local polynomial and fourier series. The purpose of this study is
to obtain an estimator of multivariable additive nonparametric regression
model. This research focuses on multivariable additive nonparametric
regression model which is a combination between fourier series and spline
truncated. The estimation method that be used to obtain the estimators is
Penalized Least Square. This method requires the estimation of smoothing
parameters in the optimization process to obtain the estimators of model. In
4998 I. Wayan Sudiarsa et al.
this study, the derivation process for obtaining the estimator of
multivariable additive nonparametric regression model has been
successfully obtained, which consists of an estimator of fourier series and
spline truncated. The results of this theoretical study shows that the
Penalized Least Square method works simultaneously for obtaining the
estimators of the smoothing parameter and nonparametric regression model
parameters as a result of combining between fourier series and spline
truncated which are additively separable. Keywords: Additive model, Fourier series, Spline truncated, Penalized
Least Square, Multivariable nonparametric regression
1. Introduction
In the last decade, nonparametric regression model has been widely
studied by many researchers. Nonparametric regression is used to model the
relationship between the response and the predictors when the functional form of
the regression curve is unknown. Due to the development of computing and some
limitations on parametric regression models, nonparametric regression model that
does not require many assumptions becomes more widely applied to solve
problems in various applied fields [1]. Fourier series nonparametric regression has
been developed by Bilodeau [2] on several predictors using the additive model.
Multivariable additive non-parametric regression model has been developed by
Hastie and Tibsirani [3], particularly in Generalized Additive Model or GAM. Up
to now, researches about multivariable additive non-parametric regression model
are limited on the same types of estimators for each predictor. One of the open
problems that arise from Bilodeau [2] is how to develop a multivariable additive
non-parametric regression model by employing different estimators for each
predictor. Hence, this research will develop a different estimator for each
predictor in multivariable additive non-parametric regression model.
Spline is one of estimators which is frequently used in nonparametric
regression because it has a good visual interpretation, flexible, and able to handle
smooth functions [4]. Moreover, the advantage of spline is able to describe the
change of the function pattern in the sub-specified interval and can handle well the
data pattern which is dramatically change by using knots [5]. Some recently
researches about the application of spline in nonparametric regression could be
found in Lestari, Budiantara, Sunaryo and Mashuri [6], Wibowo, Haryatmi and
Budiantara [7], and Fernandes, Budiantara, Otok and Suhartono [8].
Otherwise, Fourier series is also widely used in nonparametric regression
model by many researchers, such as De Jong [9], also Asrini and Budiantara [10].
Combined estimator Fourier series 4999
The main advantage of Fourier series estimator is able to handle data behavior
that follows the periodic pattern at certain intervals [2]. Additionally, other type of
estimators are also developed and applied in nonparametric regression model,
such as kernel, the local polynomial, and wavelet estimators. Some researches
about the application of kernel estimator in nonparametric regression could be
found in Manzan [11], Okumura and Naito [12], Yao [13], Kayri and Zirbhoglu
[14], and Fernandes, Budiantara, Otok and Suhartono [15]. Furthermore, some
researches about the local polynomial in non-parametric regression could be seen
in Su and Ullah [16], He and Huang [17], Martins-Filho and Yao [18], and
Oingguo [19]. Otherwise, the application of wavelet estimator in nonparametric
regression has been investigated by Qu [20], Angelini, De-Canditiis and Leblanc
[21], Rakotomamonjy, Mary and Canu [22], and Taylor [23].
This paper focuses to develop a multivariable additive non-parametric
regression model by employing different estimators for each predictor,
particularly a combination between Fourier series and spline truncated estimators.
Moreover, this research proposes Penalized Least Squares (PLS) optimization for
estimating the combination estimator.
2. Multivariable Additive Nonparametric Regression Model
In this section, we deal with the regression model involving k predictors
and the pair data is . The relationship between
and is modeled by multivariable additive nonparametric
regression as follows:
1 2, ,...,i i i pi iy x x x
1 1
2
, 1, 2,..., ,p
i j ji i
j
g x g x i n
(1)
where is a response, j jig x , 1,2,...,j p is the unknown regression curve
shape and i is variable that follows normally and independently distributed with
mean zero and variance [2].
Assume that 1 1( )g x is approached by the Fourier series function, i.e.
1 1 1 0 1
1
1cos ,
2
K
k
k
g x bx a a kx
(2)
and regression curve 2 2 3 3( ), ( ),..., ( )p pg x g x g x are truncated spline functions with
degree m and knots at 1 2, ,...,j j rjt t t as follows:
5000 I. Wayan Sudiarsa et al.
1 1
, 2,3,..., ,m r
mv
j j vj j uj j uj
v u
g x B x x t j p
(3)
where , , .0
mm j ujj uj
j uj
j uj
x tx tx t
x t
Hence, a multivariable additive nonparametric regression model in Eq. (1) can be written as follows:
1 2
2
, ,...,p
j r j
j
y Wa X t t t
(4)
where:
1,..., ,ny y y
0 1, , ,..., ,Ka b a a a
1 1,..., , ,..., ,j j mj j rjB B a a
1 ,..., n
11 11 11 11
12 12 12 12
13 13 13 13
1 1 1 1
1/ 2 cos cos 2 cos K
1/ 2 cos cos 2 cos K
1/ 2 cos cos 2 cos K ,
1/ 2 cos cos 2 cos Kn n n n
x x x x
x x x x
W x x x x
x x x x
2
1 1 1 1 1 1 1 2 1 1 1
2
2 2 2 2 1 2 2 2 2 2 2
21
3 3 3 3 1 3 3 2 3 3 3
2
1 2
,...,
m m mm
j j j j j j j j rj
m m mm
j j j j j j j j rj
m m mm
j rj j j j j j j j rj
m m mm
jn jn jn jn jn jn jn jn rjn
x x x x t x t x t
x x x x t x t x t
X t t x x x x t x t x t
x x x x t x t x t
and 2,3,..., .j p
3. Combination Estimator between Fourier Series and Spline
Truncated in Multivariable Additive Nonparametric Regression
Model
The estimator of combination between Fourier series and spline
truncated, 1 2ˆ , ,...,i i pix x x , is obtained by employing PLS optimization, i.e.
Combined estimator Fourier series 5001
1
1
2
21 (2)
1 1 1 1 10,
1 2 0
2.
p m r
pn
i i j jig C
i j
R
Min n y g x g x g x dx
(5)
Some lemmas are needed for solving this PLS optimization. Lemma 1
If the function of Fourier series is 1g as in Eq. (2), then the penalty is
.2
1
24
1
2
1
)2(
1
0
1
K
k
kakdxxggJ
Proof:
Because 1 1 1 0 1
1
1cos
2
K
k
k
g x bx a a kx
, then the second derivative of its
equation is
2
1 1 1 0 1
11 1
1cos
2
K
k
k
d dg x bx a a kx
dx dx
1
11
sinK
k
k
db ka kx
dx
2
1
1
cos .K
k
k
k a kx
So, the penalty 1( )J g becomes
2
2
1 1 1
10
2( ) cos
K
k
k
J g k a kx dx
2 2 2
1 1 1 1
10
2cos 2 cos cos .
K K
k k j
k k j
k a kx k a kx j a jx dx
Let’s assume that
2
21 1
1 0
2cos
K
k
k
A k a kx dx
and
2 21 1 1
0
22 cos cos .
K
k j
k j
B k a kx j a jx dx
5002 I. Wayan Sudiarsa et al. Then, we could calculate and show that
2
21 1
1 0
2cos
K
k
k
A k a kx dx
4 21 1
1 0
1 2sin
K
k
k
k a x kxk
4 2
1
,K
k
k
k a
(6)
and
2 21 1 1
0
4cos cos
K
k j
k j
B k a kx j a jx dx
2
1 1 1
0
4cos cos
K
k j
k j
kj a a kx jx dx
0. (7)
Hence, by using the results in Eq. (6) and (7), the penalty 1( )J g is obtained as
follows:
4 2
1
1
K
k
k
J g k a
. (8)
Lemma 2
If 1g is approached by Fourier series function as in Eq. (3) and jg for
2,3,...,j p approached by truncated spline functions as in Eq. (4), then the
goodness of fit R g is as follows:
1R g n Y Wa X Y Wa X
where
1 2, ,..., pg g g g
, 1 2, ,..., nY y y y , 0 1, , ,..., ka b a a a , and
11 2 12 2 1 1, , , , , , , , , , , .m r p mp p rpB B B B
Proof:
In general, a goodness of fit R g is defined as
2
1
1 1
1 2
( )pn
i i j ji
i j
R g n y g x g x
(9)
It could be shown that
Combined estimator Fourier series 5003
1
1 0 1 2 21 2 2
1 1 1 1
1cos
2k
n K m rm
v
i i i v u i uj
i k v u
R g n y bx a a kx B x x t
2
3 3 3 3 3
1 1 1 1
.m r m r mmv v
v i u i u vp ji up ji uj
v u v u
B x x t B x x t
Hence, a goodness of fit R g is
1R g n Y Wa X Y Wa X .
Then, estimator of combination between Fourier series and spline truncated in
multivariable additive nonparametric regression model could be derived by
applying lemma 1 and 2 as given by theorem 1.
Theorem 1
Let is a multivariable additive nonparametric regression model as in Eq. (1). If
the estimator of combination between Fourier series and spline truncated obtained
from PLS optimization as in Eq. (5), then the estimator of j jg x , 1,2,3,...,j p
is as follows:
1 1 1 0 1
1
1ˆˆ ˆ ˆ cos2
K
k
k
g x bx a a kx
,
and
1 1
ˆ ˆˆ , 2,3,..., ,m r
mv
j j vj j uj j uj
v u
g x x x t j p
where 0ˆ ˆ ˆ, , , 1,2,..., ,kb a a k K
ˆ , 1,2,...,vj v m
and ˆ , 1,2,...,uj u r ,
2,3,...,j p are obtained from equation
0 1ˆˆ ˆ ˆ ˆ, , , , ,..., Ka K t b a a a
, , ,A K t y
11 2 12 2 1 1ˆ ˆ ˆ ˆ ˆˆ ˆ ˆ ˆ, , , , , , , , , , , , , .m r p mp p rpK t
, , ,B K t y
1
1 1, , , , , , ,A K t S K W I S K W X I X X X WS K W X X X X I WS K W
1
1 1, , , , ,B K t I X X X WS K W X X X X I WS K W
1
, .S K W W n D
Hence, the estimator of combination between Fourier series and spline truncated
is
5004 I. Wayan Sudiarsa et al.
1 2 1 1
2
ˆ ˆ ˆ, ,..., .p
i i pi j j
j
x x x g x g x
Proof:
The estimator of combination between Fourier series and spline truncated is
obtained by employing optimization as in Eq. (5). By applying lemma 2, it could
be shown that
2
1
1 1
1 2
pn
i i j ji
i j
R g n y g x g x
1 .n Y Wa X Y Wa X
Then, by using lemma 1 we obtain that
2
2
1 1 1 1
0
2.J g g x dx a Da
The PLS optimization is done by combining the goodness of fit R g and penalty
1J g as follows:
1
1(0, ) 21
11
g Ca R
RR
K
p m rp m r
Min R g J g Min n Y Wa X Y Wa X a Da
2
1
a R
R
K
p m r
Min
,a
So, the estimation of a and
are obtained by using partial derivative of
,a to a and . First, consider the function ,a
as follows:
1,a n y Wa X y Wa X a Da
1 1 1 1 12 2n y y n a W y n X y n a W X n y X
1 1 .n X X a n W W D a
Then, the derivation of ,a respect to a is
1 1 1
,2 .
an W y n W X n W W D a
a
By taking ,
0a
a
, it could be shown that
Combined estimator Fourier series 5005
1 1 1ˆ ˆ.n W y n W X n W W D a
Then, multiply both sides with 1
1n W W D
and we have the result as
follows:
1
1 1 1 ˆa n W W D n W y n W X
1 ˆW W n D W y X
ˆ, .S K W y X
(10)
Moreover, the function of ,a can also be written as
1 1 1 1 1, 2a n y y n a W y n X y n y Wa n a W Wa
1 12 .n X Wa n X X a Da
To minimize ,a , we apply partial derivative of ,a with respect to
as follows:
1,
2 .a
n X y X Wa X X
By taking ,
0a
, we have
ˆ .X X X y X Wa
By multiplying both sides with 1
X X
, we get
1 1ˆ ˆ ˆ .X X X y X Wa X X X y Wa
(11)
Substituting Eq. (10) into Eq. (11) to obtain
1ˆ ˆ,X X X y W S K W y X
1 ˆ,X X X y WS K W y W X
1 ˆ, ,X X X y WS K W y WS K W X
1 1 1 ˆ, , .X X X y X X X WS K W y X X X WS K W X
Then, by subtracting both sides with 1 ˆ,X X X WS K W X
we obtain that
5006 I. Wayan Sudiarsa et al.
1 1 1ˆ ˆ, ,X X X WS K W X X X X y X X X WS K W y
1 1ˆ, ,I X X X WS K W X X X X I WS K W y
.
Therefore, the estimator of ˆ , ,K t is given by
1
1ˆ , , ,K t I X X X WS K W X
1
,X X X I WS K W y
(12)
, , ,B K t y
(13)
where
1
1 1, , , , .B K t I X X X WS K W X X X X I WS K W
Similarly, if Eq. (12) are substituted into Eq. (10), then we have
ˆˆ , , ,a K t S K W y X
1
1 1, , ,S K W I X I X X X W S K W X X X X I WS K W y
1
1 1, , , ,S K W S K W X I X X X WS K W X X X X I WS K W y
, , ,A K t y
(14)
where
1
1 1, , , , , , .A K t S K W I S K W X I X X X WS K W X X X X I WS K W
Furthermore, the estimator of combination between Fourier series and truncated spline can be written as
1 2
ˆˆ ˆ, ,..., , , , ,i i pix x x Wa K t X K t
, , , ,WA K t XB K t y
, , ,C K t y
(15)
where , , , , , , .C K t WA K t XB K t
Otherwise, the estimator of combination between Fourier series and truncated
spline could also be presented as follows:
1 2 1 1
2
ˆ ˆ ˆ, ,...,p
p j j
j
x x x g x g x
,
where
1 1 1 0 1
1
1ˆˆ ˆ ˆ cos2
K
k
k
g x bx a a kx
, and
1 1
ˆ ˆˆ ,m r
mv
j j vj j uj j uj
v u
g x x x t
2,3,..., .j p
Combined estimator Fourier series 5007
Thus, the estimator of the parameter 0ˆ ˆ ˆ, , , 1,2,..., ,kb a a k K
ˆ , 1,2,...,vj v m and
ˆ , 1,2,..., ,uj u r 2,3,...,j p are obtained from the following equations, i.e.
0 1ˆˆ ˆ ˆ ˆ, , , , ,..., Ka K t b a a a
, , ,A K t y
11 2 12 2 1 1ˆ ˆ ˆ ˆ ˆˆ ˆ ˆ ˆ, , , , , , , , , , , , , .m r p mp p rpK t
, ,B K t y .
4. Conclusion
Let is a multivariable additive nonparametric regression model as follows:
1 1
2
, 1, 2,..., .p
i j ji i
j
g x g x i n
where is a response, j jig x , 1,2,...,j p is the unknown regression curve
shape (assumed smooth) and i is variable that follows normally and
independently distributed with mean zero and variance . The regression curve
1g is assumed smooth in the space of continuous functions C(0,π) and approached
by Fourier series. Otherwise, the regression curve
, 2,3,...,jg j p
are
approached by truncated spline function. This paper focuses on theoretical
study to study further how to obtain an estimator of multivariable additive
nonparametric regression model, particularly using combination between Fourier
series and truncated spline function. The results shows that the estimator of
combination between Fourier series and truncated spline function was obtained
through the Penalized Least Square optimization, i.e.
1
1
2
21 (2)
1 1 1 1 10,
1 2 0
2
p m r
pn
i i j jig C
i j
R
Min n y g x g x g x dx
.
The estimator of combination between Fourier series and truncated spline
function is
1 2 1 1
2
ˆ ˆ ˆ, ,..., ,p
p j j
j
x x x g x g x
where
1 1 1 0 1
1
1ˆˆ ˆ ˆ cos2
K
k
k
g x bx a a kx
,
1 1
ˆ ˆˆ ,m r
mv
j j vj j uj j uj
v u
g x x x t
5008 I. Wayan Sudiarsa et al.
and 0ˆ ˆ ˆ, , , 1,2,..., ,kb a a k K ˆ , 1,2,...,vj v m , ˆ , 1,2,..., ,uj u r
2,3,...,j p are
obtained from
0 1ˆˆ ˆ ˆ ˆ, , , , ,..., Ka K t b a a a
, , ,A K t y
1
1 1, , , , ,A K t S K W I X I X X X W S K W X X X X I WS K W
11 2 12 2 1 1ˆ ˆ ˆ ˆ ˆˆ ˆ ˆ ˆ, , , , , , , , , , , , , .m r p mp p rpK t
, ,B K t y ,
1
1, , ,B K t I X X X WS K W X
1
, .X X X I WS K W
Additionally, the results also show that further research is needed to
validate this theoretical results in empirical data, both simulation and real
data. Specifically, the issue about the properties of this estimator to the
sample size and function forms are needed to study further.
References [1] M. B. Machler, Variational Solution of Penalized Likehood Problem and
Smooth Curve Estimation, Annals of Statistics, 23 (1995), 1496 - 1517. http://dx.doi.org/10.1214/aos/1176324309 [2] M. Bilodeau, Fourier Smoother and Additive Models, The Canadian
Journal of Statistics, 20 (1992), 257 - 269.
http://dx.doi.org/10.2307/3315313
[3] T. J. Hastie and R. J. Tibshirani, Generalized Additive Models, Chapman
and Hall, New York 1990.
[4] R. L. Eubank, Nonparametric Regression and Spline Smoothing, Second
Edition, New York, Marcal Dekker, Inc 1999.
[5] R. L. Eubank, Spline Smoothing and Nonparametric Regression, Marcel
Dekker, New York 1988.
[6] B. Lestari, I. N. Budiantara, S. Sunaryo and M. Mashuri, Spline Smoothing
for Multi-response Nonparametric Regression Model in case of
Heteroskedasticity of Variance, Journal of Mathematics and Statistics, 8
(2012), 377 - 384. http://dx.doi.org/10.3844/jmssp.2012.377.384 [7] W. Wibowo, S. Haryatmi and I. N. Budiantara, Modeling of Regional
Bangking Activitas using Spline Multiresponse Semiparametric Regression,
Combined estimator Fourier series 5009 International Journal of Apllied Mathematics and Statistics, 23 (2013), 102 -
110.
[8] A. A. R. Fernandes, I. N. Budiantara, B. W. Otok and Suhartono, Spline
Estimator for Bi-responses Nonparametric Regression Model for
Longitudinal Data, Journal of Mathematics and Statistics, 10 (2014) 451-
457.
[9] P. De Jong, The Fast Fourier Transform Spectral Estimator, Journal of
Statistical Soc. B 39 (1977) 327 - 330. [10] L. J. Asrini and I. N. Budiantara, Fourier Series Semiparametric Regression
Models (Case Study: The Production of Lowland Rice Irigation in Central Java), Journal of Engineering and Applied Sciences, 9 (2014) 1954 - 1960.
[11] S. Manzan and D. Zerom, Kernel Estimation of a Partially Linear Additive
Model, Statistics & Probability Letters, 72 (2005) 313 - 322. http://dx.doi.org/10.1016/j.spl.2005.02.005 [12] H. Okumura and K. Naito, Non-parametric Kernel Regression for
Multinomial Data, Journal of Multivariate Analysis, 97 (2006) 2009 - 2022. http://dx.doi.org/10.1016/j.jmva.2005.12.008 [13] F. Yao, Asymptotic Distributions of Nonparametric Regression Estimators
for Longitudinal or Functional Data, Journal of Multivariate Analysis, 98 (2007) 40 - 56. http://dx.doi.org/10.1016/j.jmva.2006.08.007
[14] M. Kayri, and G. Zirhhoglu, Kernel Smoothing Function and Choosing
Bandwitdh for Nonparametric Regression Methods, Ozean Journal of Applied Sciences, 2 (2009) 49 - 54.
[15] A. A. R. Fernandes, I. N. Budiantara, B. W. Otok, and Suhartono,
Reproducing Kernel Hibert Space for Penalized Regression Multi-Predictors: Case in Longitudinal Data, International Journal of Mathemati-
cal Analysis, 8 (2014) 1951 - 1961. http://dx.doi.org/10.12988/ijma.2014.47212 [16] L. Su and A. Ullah, Local Polynomial Estimation of Nonparametric
Simultaneous Equations Models, Journal of Econometrics, 144 (2008) 193 -218. http://dx.doi.org/10.1016/j.jeconom.2008.01.002
[17] H. He and L-S. Huang, Double-smooting for Bias Reduction in Local Linear
Regression, Journal of Statistical Planning and Inference, 139 (2009) 1056
- 1072. http://dx.doi.org/10.1016/j.jspi.2008.06.011 [18] C. Martins-Filho and F. Yao, Nonparametric Regression Estimation with
General Parametric Error Covariance, Journal of Multivariate Analysis, 100 (2009), 309 - 333. http://dx.doi.org/10.1016/j.jmva.2008.04.013
5010 I. Wayan Sudiarsa et al. [19] T. Qingguo, L1-Estimation in a Semiparametric Model with Longitudinal
Data, Journal of Statistical Planning and Inference, 140 (2010) 393 - 405. http://dx.doi.org/10.1016/j.jspi.2009.06.016 [20] L. Qu, On Semiparametric Regression via Wavelets. Ph.D., Dissertation,
Purdue University 2002. [21] C. Angelini, D. De-Canditiis and F. Leblanc, Wavelet Regression
Estimation in Nonparametric Mixed Effect Models, Journal of Multivariate Analysis, 85 (2003) 267 - 291.
http://dx.doi.org/10.1016/s0047-259x(02)00055-6 [22] A. Rakotomamonjy, X. Mary and S. Canu, Nonparametric Regression with
wavelet kernels, Appl. Statistic Models in Business and Industry., 21 (2005) 153 - 163. http://dx.doi.org/10.1002/asmb.533
[23] L. W. Taylor, Using the Haar Wavelet Transform in the Semiparametric
Specification of Time Series, Economic Modelling, 26 (2009) 392 - 403. http://dx.doi.org/10.1016/j.econmod.2008.08.001
Received: June 4, 2015; Published: July 29, 2015