Ann. Inst. Statist. Math. Vol. 45, No. 3, 499-510 (1993)

ESTIMATING FUNCTION WITH ASYMPTOTIC BIAS AND ITS ESTIMATOR

YOSHIJI TAKAGI 1 AND NOBUO INAGAKI 2

1 College of Integrated Arts and Science, University of Osaka Prefecture, Sakai, Osaka 593, Japan

2Department of Mathematical Science, Faculty of Engineering Science, Osaka University, Toyonaka, Osaka 560, Japan

(Received September 9, 1991; revised September 11, 1992)

Abstract. We introduce the estimating function with asymptotic bias and investigate the asymptotic behavior of the estimator based on it by using their relationship. The estimator based on the estimating function with asymptotic bias has the asymptotic normality with asymptotic bias. We show that this theory has several interesting applications in practical statistics.

Key words and phrases: Estimating function, asymptotic bias, asymptotic normality, order statistics, ridge estimator function, maximum posterior likeli- hood estimator.

1. Introduction

Let X~ (Xnl , Xn2, . • •, Xn~) be an observation vector for each positive inte- ger n elements of which are independent but not necessarily identically distr ibuted (i.n.i.d.) and let 0 = (0(1) , . . . , 0 (k)) be a k-dimensional unknown parameter vec- tor and there are no nuisance parameters throughout this paper. We consider the

k-dimensional est imating function ~ ( 0 ) = (gJ!~)(0),. . . , ~(k)(0)) as follows:

n

(1.1) ~ ( 0 ) = 'I2~(X~,O) = _1 E ~ni(X~i,O), n i=1

where ~ni' : ,_.~(~/~(1),... , Tni~/;(k)]J is a k-dimensional score function of observation X~i and parameter 0 which is noted to be dependent of n for i = 1 , . . . , n. We call 0~ = t?~(X~) the estimator of 0 based on ¢~(0) when

( 1 . 2 ) = o.

We also use the following notation:

(1.3) n

499

500 YOSHIJI TAKAGI AND N O B U O INAGAKI

Several important properties of the estimator, for example, consistency and asymp- totic normality, are derived from the relationship with the estimating function (see Wilks (1962), Huber (1967) and Inagaki (1973)).

Our aim of the present paper is to introduce the estimating function with asymptotic bias and to investigate the asymptotic behavior of the estimator based on it by considering the effect of the asymptotic bias. We shall discuss several interesting applications to the practical problems in statistics. Our results are closely connected with those of Inagaki (1973), where the asymptotically unbiased estimating function is discussed.

In Section 2, notations, assumptions and preliminary lemmas are stated. In Section 3, we introduce an estimating function with asymptotic bias and show that its estimator is asymptotically biased. In Section 4, we consider the applica- tions of the asymptotically biased estimating function to the practical problems in statistics.

2. Notations, assumptions and preliminary lemmas

In this section, we prepare the notations and assumptions, (A1)-(A3) and (A7) of which are fundamental but (A4)-(A6) are technical. These assumptions are similar to those in Inagaki (1973) and supposed throughout this paper. As- sumption (A2) expresses the asymptotic bias of the estimating function.

Notations. ( X , A , P): a probability space, @: a parameter space which is a subset of the k-dimensional Euclidean space

R k such that for any M > O, 0 ~ {11011 _< M} is closed, 00: the true parameter which is unknown but fixed and exists in the interior

of (~, ll-Li: the maximum norm, i.e. if011 = m&x{IS(~l,.-., Is(~)l}, ~](YI: the distribution of a random vector Y under the probability measure

P, A~: the transposed matrix of the matrix A.

ASSUMPTIONS. (A1) ~n~(X,O), i = 1 , . . . ,n are A x ~-measurable, where/3 is the a-field of

Borel subsets of O, and separable when considered as a process in 0. (A2) Expected values

An~(e) = ~A(1)ce~ A(~ ) ' , < , , , . . . , ( 0 ) ) : Z { f < ( X n ~ , e ) } i = 1 , . . . , ~ ,

exist for all 0 E • and their arithmetic mean converges:

(2.1) An(e) = _1 ~ A n i ( e ) --+ A(0), as n --+ cx~, i=1

where A(Oo) = 0 and A(O) ~ 0 if 0 ¢ 00. Furthermore, there is a k-dimensional constant vector/3 satisfying

(2.2) v ~ A ~ ( 0 o ) -+ ,s.

E S T I M A T I N G F U N C T I O N W I T H A S Y M P T O T I C BIAS 501

(A3) For each i = 1 , . . . , n, let A~i(0) be continuously differentiable in some neighborhood of 00 with the differential coefficient matrix

(2.3) o0 \ o0(~) ] ' (z,~:~,... ,k)

respectively, and their arithmetic mean be denoted by

A~(0) = _1 ~A~i (0 ) . i=1

Then, An(0) converges to A(0) uniformly in the neighborhood of 00, and A0 = A(0o) (say)is nonsingular.

(14) (a) There exists a positive constant Aoo such that

(2.4) lira lira IIA~(0)I I _> Aoo > 0. ~--+oo ii0114oo

(b) A positive function

(2.~)

satisfies

(2.6)

(c) Letting

b ~ ( O ) = m a x ( l l A ~ ( O ) l ] , ~ - )

I I ~ ( x ~ , o ) - ~i<0)11 } limoo E SoP ~ < oo.

I I ~ < x ~ , 0) - A~(0) (2.7) w~,M(x~{) = sup FI01I>M b~(O)

there exists a positive number M satisfying the following conditions:

(2.8) lim -1 ~ E{W~i ,M(X~i )} < 1, i=1

(2.9) lim 1 ~ ~ W r { W ~ , ~ ( X ~ d } = O. i=1

(A5) For each i = 1 , . . . , n, E{II@~i(X~i , O) - A,~i(0)ll 2} exists and

(2.~o) ~m E { l l ~ { ( x ~ , 0 ) - a~{(0)rl 2} -~ o.

502

(A6) Let

( 2 . n )

Y O S H I J I T A K A G I A N D N O B U O I N A G A K I

II~-0ll_<d

For every compact set C c (9, there are positive numbers do, H1, and/72 > 0 such that

n

(2.12) lim -1 E Eu~i(X~i ,O,d) < Hid, 7Z ---+ OC n

i = 1

and n

(2.13) lim -1 E E{u~i(X~i 'O'd)}2 < H2d, r ~ - ~ o o n

i = 1

the last two convergences in (2.12) and (2.13) being uniform for d _< do and 0 C C. (A7) Denote S~i = Var(~b~/(X~i,00)), the covariance matrix for each i =

1 , . . . , n , respectively. Then, their arithmetic mean S~ converges to a positive definite matrix S:

(2.14) S~ = -1 ~ S~i --+ S. n

/ = 1

Furthermore, there exists Ell~br~/(Xn/, 0o) -A~/(0o)[[ 3, i = 1 , . . . , n, and

1 (2.15) ~ i m ~ ~ E l l ~ f X ~ / , 0 0 ) - a~/(0o)ll 3 = 0.

/ = 1

The following three lemmas are straightforward extentions of the correspond- ing propositions due to Inagaki (1973) (essentially Huber (1967)) and thus, their proofs are omitted (see Takagi and Inagaki (1991)).

LEMMA 2.1. If a sequence of estimators, {0~}, satisfies the condition:

( 2 . 1 6 ) = - + 0 P a s n -+ 7~

i = 1

then it converges to Oo in P:

On ~ Oo in P as n -~ oc.

LEMMA 2.2. If { £ [ ~ ( T ~ -0o)]} is relatively compact (see Inagaki (1973)), the .following asymptotic expansion holds:

(2.17) ~(Tr~) - ~(0o) - Aox/n(T~ - 0o) ~ 0 in P, as n --~ co.

LEMMA 2.3. In order that {£[x/~(T~ - 0o)]} is relatively compact, it is nec- essary and sufficient that {£;[~(T~)]} is relatively compact.

ESTIMATING FUNCTION WITH ASYMPTOTIC BIAS 503

3. Asymptotically biased estimator

In this section, we show" the asymptotic normality of the estimator On based on the estimating function ~ (0 ) . Furthermore, we construct the asymptotically equivalent estimator to 0~ by the so-called "one-step estimator".

THEOREM 3.1. The random vector ~(0o) is asymptotically normal with asymptotic bias in the following sense:

(3.1) 12[~(0o)] -+ ;V-k(9, S) in law.

PROOF. From Assumption (A7), we have that

(3.2) 12[~(0o) - v/~),~(Oo)]

= 12 1 Z{O~i (Xn i ,Oo ) _ /~ i (0o)} --+ N~(O,S) in law i=1

(see Lo~ve (1978), p. 287), and thus, from Assumption (2.2) in (A2) that the result of this theorem holds. []

THEOREM 3.2. If a sequence of estimators, {0~}, satisfies the condition:

(3.3) {~(0,~) --+ 0 in P,

then, the estimator On is consistent and asymptotically normally distributed:

(3.4) £[,/n(O~ - 0o)] --+ Nk[ /3a, SA ] in law,

where ~A = -(Ao-1)/3 and SA ---- (Ao-1)S(Ao-1) ' •

PROOF. Since (3.3) implies (2.16), we have the consistency of 0n by Lemma 2.1. It follows from (3.3) and Lemma 2.3 that {12[x/~(0n 0o)1} is relatively compact, and therefore by Lemma 2.2 that

(3 .8) { ~ ( 0 ~ ) - { ~ ( 0 o ) - A o ~ ( 0 . - 0o) -~ 0

Thus, we have from (3.1), (3.3) and (3.5) that

C [ A o ~ / ~ ( 0 ~ - 0o)] -~ m k ( - 9 , s )

Hence, the result of this theorem is proved. []

THEOREM 3.3.

(3.6)

in P.

in law.

If an estimator T~ satisfies the condition:

12[~(T~)] --+ G, in law,

504 YOSHIJI TAKAGI AND NOBUO INAGAKI

where G is a probability distribution, then it holds that

(3.7) Z;[Aox/~(T~ - t)~)] --, G, in law.

The converse is also true.

PaOOF. Standard techniques show that the condition (3.6) or (3.7) implies the relative compactness of {£[x/~(T~ - 0o)]}. Therefore, it follows by Lemma 2.2 that T~ satisfies

(3.8) ~n(T~) - ~ ( 0 o ) - Aox/~(T~ - 0o) --+ 0 in P.

We have already in the proof of Theorem 3.2 that

- ~ ( 0 o ) - A o v ~ ( 0 ~ - 0o) ~ 0 in P. (3.9)

These lead to

(3.10) ~n(T~) - iox /n(Tn - 0n) -~ 0 in P.

Hence, either one of (3.6) and (3.7) derives the other. []

THEOREM 3.4. Suppose that {Z;[x/~(T~ - 0o)} is relatively compact. Put

(3.11) Tr~ = T~ - A - I ( T n ) ~ ( T ~ ) ,

where A-I(T~) means that Oo in A-l(0o) is taken place by Tn. Then, it holds that

(3.12) ~ ( T ~ ) --, 0 in P.

This implies by Theorem 3.3 that T~ and O~ are asymptotically equivalent:

(3.13) x/n(T£ - 0n) --~ 0

and furthermore, by Theorem 3.2 that

(3.14)

PROOF.

(3.15)

in P,

C [ v / ; ( T 2 - 0o)] SA I in law.

(3.11) is equivalent to

x/-n(~r~ - 0o) = ~/n(Tn - 0o) - A- l (Tn)~n(Tn) .

The relative compactness of {/2[x/~(T~ - 0o)]} and the continuity of A(0) imply that

(3.16) A l (Tn) ----+ A - l ( 0 o ) = Ao -1 in P,

ESTIMATING FUNCTION WITH ASYMPTOTIC BIAS 505

and thus, from (3.15) that {g[x/~(T~ - 0o)} is relatively compact. The last leads to

(3.17)

Of course, (3.8) holds, too. Therefore, it follows by (3.15) and (3.17) tha t

(3.18) ~ ( T ~ ) + {(AoA-I(T~))¢~(T~) - ~ ( 0 o ) - Ao~/n(T~ - 0o)) ~ 0 in P.

(3.8) and (3.16) implies that the par t of { } in (3.18) converges to 0 in P. This and (3.18) conclude (3.12). The proof is complete. []

By Theorem 3.4, we can construct the so-called one-step est imator T*, which

is asymptot ical ly equivalent to 0n even though it is not explicitly got ten as the solution of the est imating function.

4. Applications

4.1 Central order statistics Let observations Xi, i = 1, 2 , . . . be independent and identically dis t r ibuted

according to the distr ibution function F with the density function f . Let the parameter space O = R 1. We suppose that 0 < F(Oo) = p < 1, f ( x ) is continuous at 0o and f(Oo) > 0. Pu t

~ , ~ ( x , 0) =

p,~ = p + - - i f x > 0,

if x = 0, q~ - x/~,

/3 if x < 0 . r~ = - ( l - p ) + ~ ,

which is not depend of i. Then, we have

a . ( 0 ) = E ~ . ~ ( x ~ , 0) = p - r ( 0 ) + - -

It is easy to see that Assumptions (A1) (A7) are satisfied, where

a ( o ) = v - F ( o ) ,

A(O) = A~(O) = -f(O), s = & ( O o ) - p ( 1 - v ) .

Now, we denote the order statistics of XI,... ,X~ by Xn: 1 ~ "'" ~ Xrz:n. For a positive integer, u,~, which satisfies

- - = p + ~ + o , as n ---~ co, %

506 YOSHIJI TAKAGI AND NOBUO INAGAKI

we consider an order statistic X ~ : , . Then, we see tha t

1

1 v~ 1

v~

i = 1

- - - { p ~ ( ~ - . ~ ) + q~ + r ~ ( . ~ 1 ) }

-- - - { 1 - p + o ( ~ ) } --* 0 as n ~ o o .

Thus, by Theorem 3.2 we have the following theorem (see Serfling (1980)).

THEOREM 4.1. The u,~-th order statistic satisfying (4.1) is asymptotically normally distributed:

(4.2) c [ , / ~ ( x ~ : . ~ - 00)] - * x ( g a , & ) ,

with

fl p(1 - p ) (4.3) f l A - f(Oo) and SA- f2(Oo)

4.2 The ridge-type estimating function We consider the linear regression model, GM(Yn;X.~O, cr2I~), where Yn =

(Y~l , . . - , Y ~ ) ' is the vector of independent observations, X~ = {xij }, i = 1,. . . , n, j = 1 , . . . , k the design matrix, and 0 = (01 , . . . , 0k) t the parameter vector. Then, the normal equation for the least square estimator is X~(Y~ -X~O) = 0. It is well known tha t the least square estimator is the best linear unbiased estimator. Hoerl and Kennard (1970) propose the ridge estimator

(4.4)

as the solution of the equation X~(G~ - X~O) = ~,~0. Now, suppose

(4.5) Y n z ~-~lJ ,

1 - X ~ X ~ ---* F, n

a s n --+ o o ,

where F is nonsingular. The ridge estimating function is

% ( 0 ) = l { x ' ( Y ~ - XnO) -- , / ~ . 0 } .

Then, we have that

>,,~(e) = ~ - { x ' ~ z ~ ( e o - e) - ~ / ~ . e } ~ ; , (e ) = r ( e o - e ) ,

E S T I M A T I N G F U N C T I O N W I T H A S Y M P T O T I C BIAS

and thus, that A(00) : 0, x/nA~(00) = -~0o = fl (say), and

Furthermore, we have

n

507

(4.6) z[ , /~(T~ - Oo)] --+ N(/3a, & ) ,

with

(4.7) ,3A = F - 1 / 3 = - - P F - 1 0 0 and SA = ~ 2 F - 1 .

4.3 The maximum posterior likelihood estimator Let X1, X2, . . . be independent and identically distributed random variables

with the density function f ( x I 0). Then, the joint density function of X = (X1,..., X~) is

n

f,~(x ]0) = H f (X i ]0). i=1

We deal with the Baysian problem that the parameter 0 has the prior density function r%(0), which is supposed to satisfy the following conditions:

1 0 li~moo x/~ (900 log rr~ (0o) =/3,

(4.8) 1 02

lira - log 7r~ (0) = 0. n ---+ OO n

We consider the estimator defined by the mode of posterior distribution,

fn(x ] O)Tr~(0) >(O l x) = f A(x I o>~(o)d0'

which is called the maximum posterior likelihood estimator (MPLE). In our situ- ation, the estimating function is

qJ~(0)= 1 ~ 0 - n u~ 1~0 ~-~logf(xi 10) + - l o g ~ ( 0 ) . n

i 1

Then, under the so-called regularity conditions (for example, see Cram6r (1946)), we have

1 0

1 02 an(0) - a(o) - log~,~(O) -+ 0, as n -+ ~ ,

n 020

THEOREM 4.2. The ridge estimator T~ under Assumption (4.5) is asymptot- ically normally distributed:

508 YOSHIJI TAKAGI AND NOBUO INAGAKI

where {o } A(0)=E0o ~ l o g f ( X ] O ) ,

1(o) = EOo ~ log f ( x r o) .

Furthermore, we see A(00) = 0 and A(00) = -I(Oo), where I(O) is the Fisher information. Thus, we have the following theorems:

THEOREM 4.3. The maximum posterior likelihood estimator under Assump- tion (4.8) is asymptotically normally distributed:

(4.9) ~[~//~(Tn -- 00)] ----+ ]N/(~A, SA),

with

(4.10) flA = [ ( O 0 ) - l f l and SA = I(00) -1.

THEOREM 4.4. For the MLE O~ satisfying the likelihood equation:

n ~ logf(x i ] On) = O, i=1

we have, by Theorem 3.4, the one-step estimator T~:

(4.11) 1 0

For example, in the Bernoullis trials with the success probability 0, the least favorable prior density function is the following Beta density function:

~.(o) = 0g~/2-1(1 _ 0)~n/2-1 ,

which satisfies the conditions (4.8) with

1 - - - - 00

~ _ 2 0o(1-0o)

Since the Fisher information is I(O) = 1/0(1 - 0), we obtain that the maximum posterior likelihood estimator:

(4.12) T~ =

1 1 X~ +

2 v ~ n 1 2

1 +

ESTIMATING FUNCTION WITH ASYMPTOTIC BIAS

is asymptotically normally distributed in the following form:

(4.13) L[xf~(T,~ - Oo)] -+ N (~ - Oo,Oo(1- Oo)) .

On the other hand, by Theorem 4.4, we have the one-step estimator:

(4.14) T~*= 1 - ~ + X n + 2x/~ n"

509

5. Discussion

We have that the estimator with asymptotic bias is inferior to the estimator with asymptotically unbiased under mean square error of the asymptotic distri- bution. However, we seem that it is worth noting the justice and utility of the estimator with asymptotic bias in any practical situation where the sample size

may be large but not indefinitely large. For example, in Subsection 4.3, the maximum posterior likelihood estimator

Tn has the mean square error

( 0 - ~)2 { ( 1 - ~ ) 2 - 1 } + ~

2 ( 1 ~ n 1 + ~

On the other hand, the MLE X~ has the mean square error 0(1 - O)/n. Now T~ is better than ~-~ for value of 0 such that 2{ 2}1

(o_~) (1_~) _1 +~ o/1_~/ <

n 1 + ~

or in the interval

0 E - a, ~ + a where

1 ~ )) 1 (,+~ ~)1 a = 2 ( 1 2 )2 ( - 2 2

I+~ -I+

1/2

This interval tends to zero as n --+ oo. This fact corresponds to the fact that Tn has the asymptotic bias. But in our finite sample, T~ will be superior to X~ in

510 YOSHIJI TAKAGI AND NOBUO INAGAKI

the neighbourhood of 0 = 1/2. In fact, when n = 100, Tn is bet ter than X-~ in the interval 0 E (0.273, 0.727).

In the case of Subsection 4.2, the same fact is showed. It is well known tha t in general, the ridge estimator is a shrinkage estimator toward zero and has a bias but is superior to the least square estimator under the total mean square error. Therefore, in our situation, the ridge est imator of (4.4) has the asymptot ic bias but will be better than (X~X~)-IX~Y~ in the neighbourhood of 0 = 0.

Acknowledgements

The authors are grateful to the referees for their valuable comments and useful suggestions.

REFERENCES

Cram~r, H. (1946). Mathematical Methods of Statistics, Princeton University Press, Princeton. Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for non-orthogonal

problems, Technometrics~ 12, 55-67. Huber, P. J. (1967). The behavior of maximum likelihood estimators under nonstandard condi-

tions, Proc. Fifth Berkelay Syrup. on Math. Statist. Prob., Vol. 1, 221 233, Univ. of Califor- nia Press, Berkley.

Inagaki, N. (1973). Asymptotic relations between the likelihood estimating function and the maximum likelihood estimator, Ann. Inst. Statist. Math., 25, 1-26.

Lo~ve, M. (1978). Probability Theory, I, II, 4th. ed., Springer, New York. Serfiing, R. J. (1980). Approximation Theorems of Mathematical Statistics, Wiley, New York. Takagi, Y. and Inagaki, N. (1991). Asymptotic biased estimating function and its applications,

Reseach Reports on Statistics No. 31, Osaka University. Wilks, S. (1962). Mathematical Statistics, Wiley, New York.