A New Look at the Statistical Model Identification

HIROTUGU AI(AIKE, JIEJIBER, IEEE

IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. AC-19, KO. 6, DECEMBER 1974

Professor: Ming-Shyan Wang Student: Cai-Jia Hong

Outline

• Abstract• Introduction• Relate Work• 11. HYPOTHESITSE STINIGN TIMES ERIESA NALYSIS• IV. IIEAKL OGLIKELIHOOADS AS A MEASURE OF FIT• VI. NUMERICALE XAMPLES• VII. DISCCSSIOXS

• ACKSOTTLEDGNEST• REFERENCES

Abstracthe history of the development of statistical hypothesis testing in time series analysis is reviewed briefly and it is pointed out that the hypothesis testing procedure is not adequately defined as the procedure for statistical model identilication. The classical maximum likelihood estimation procedure is reviewed and a new estimate minimum information theoretical criterion (AIC) estimate (MAICE) which is designed for the purpose of statistical identifica- tion is introduced. When there are several competing models the MAICE is defined by the model and the maximum likelihood esti- mates of the parameters which give the minimum of AIC defined by AIC = (-2)log(maximum likelihood) + 2(number of independently adjusted parameters within the model). MAICE provides a versatile procedure for statistical model identi- fication which is free from the ambiguities inherent in the application of conventional hypothesis testing procedure. The practical utility of MAICE in time series analysis is demonstrated with some numerical examples.

Introduction(1/3) in spite of the recent, development of t.he use of statisticalconcepts and models in almost, every field of engineeringand science it seems as if the difficulty of constructingan adequate model based on the informationprovided by a finite number of observations is not fullyrecognized. Undoubtedly the subject of statistical modelconstruction or ident.ification is heavily dependent on theresults of theoret.ica1 analyses of the object. under observation.Yet. it must be realized that there is usually a big gapbetn-een the theoretical results and the pract,ical proceduresof identification. A typical example is the gap betweenthe results of the theory of minimal realizations of a linearsystem and the identifichon of a Markovian representationof a stochastic process based on a record of finiteduration.

Introduction(2/3)A minimal realization of a linear system isusually defined through t.he analysis of the rank or thedependence relation of the rows or columns of someHankel matrix [l]. In a practical situation, even if theHankel matrix is theoretically given! the rounding errorswill always make the matrix of full rank. If the matrix isobtained from a record of obserrat.ions of a real object thesampling variabilities of the elements of the matrix nil1 beby far the greater than the rounding errors and also thesystem n-ill always be infinite dimensional. Thus it can beseen that the subject of statistical identification is essentiallyconcerned with the art of approximation n-hich is abasic element of human intellectual activity.

11. HYPOTHESITSE STINIGN TIMES ERIESA NALYSIS

The studyo f t,he t,esting procedureof time series start,edwith t.he investigation of the test. of a. simple hypot,hesist,hat a. single serial correlation coefficient. is equal t.o 0.The utilit,y of this t.ype of t.est, is certa,inly t.oo limit,ed tomake it a generally useful procedure for model identification.In 1%7 Quenouille 143 int.roduced a test for thegoodness of fit of a.utoregressive (AR) models. The idea ofthe Quenouille’s test was extended by Wold [5] to a test ofgoodness of fit of moving average (31-4) models. Severalrefinements and generalizations of these test. proceduresfollowed [GI-[9] but a most, significant contribut,ion to trhesubject, of hypothesis testing in time series analysis wasmade by Whittle [lo], [ l l ] by a systematic application ofthe 7Veyma.n-Pearson likelihood ratio t.est. procedure tot.he time series situation.

IV. IIEAKL OGLIKELIHOOADS AS A MEASURE OF FIT

The well known fact that the MLE is, under regularityconditions, asymptotically efficient [29] shows that thelikelihood function tends to be a most sensitive criterion ofthe deviation of the model parameters from the truev alues.Consider the situation where ~ 1 . x. .~ .. ,x .\- are obtainedas the rcsults of 12: independent observations of a randomvariable with probability density function g(r). If nparametric famil- of density function is given by f(.&)with a vector parameter 0. the average log-likelihood. orthe log-likelihood divided by X , is given by

VI. NUMERICALE XAMPLES

Before going into t,he discussion of t,he characteristicsof MAICE it,s practical utilit,y is demonst,rated in thissection.

VII. DISCCSSIOXSWhen f(al0) is very far from g(x), S(g;f(. je)) is only asubjective measure of deviation of f(.r(e) from g(r). Thusthe general discussion of the characteristics of 3IAICEnil1 only be possible under the assumption that for atleast one family f(&) is sufficiently closed to g(r) comparedwith the expected deviation of f(sJ6) from f(rI0).The detailed analysis of thc statistical characteristics ofXXICE is only necessary when there are several familieswhich sa.tisfy this condition. As a single estimate of-2A7ES(g;f(.)8)-)2, times the log-mitximum likelihoodwill be sufficient but for the present purpose of “estimatingthe difference” of -3XES(g;f(. 18)) the introduction of theterm +2k into the definition of AIC is crucial. The disappointingresults reported by Bhansali [%I were due tohis incorrect use of the statistic. equivalent to using +X.in place of +?X- in AIC.

VIII. COWLUSIONThe practical utility of the hypothesis testing procedureas a method of statistical model building or identificationmust be considered quite limited. To develop usefulprocedures of identification more direct approach to thecontrol of the error or loss caused by the use of the identifiedmodel is necessary. From the success of thc classicalmaximum likelihood procedures t he mean log-likelihoodseems to be a natural choice as the criterion of fit of astatistical model. The XUCE procedure based on -4ICwhich is an estimate of the mean log-likelihood proyides aversatile procedure for the statistical model identification.It also provides a mathematical formulation of the principleof parsimony in the field of model construction.Since a procedure based on AIAICE can be implementedwithout the aid of subjective judgement, thc successfulnumerical results of applications suggest that the implcmentationsof many statistical identification proccdurcsfor prediction, signal detection? pattern recognition, andadaptation will be made practical 11-ith AIBICE.

ACKSOTTLEDGNEST

The author is grateful to Prof. T. Kailath, StanfordUniversity: for encouraging him to mite the presentpaper. Thanks are also duc to Prof. I<. Sato. SagasakiUniversity, for providing the brain wave data treated inSection V.

REFERENCES• [l] H. Aikaike, “Stochastic theory of minimal rea.lization,“ this• [2] E. L. Lehntan, Tcstitrg Statistical Hypothesis. Ken- York: 131 H. .lkaike.

“Inforn~stion theory and an extension of the maxiiswe, pp. 667-6i4. IViley, 1959.

• [4] 11. H. Qaenouille. “A large-sample test for the goodness of fit of antoregressive schemes,” J. Xo!y. Statist. Soc., vol. 110, pp. 12:3-120, 1947. .1.51. H. JVold. “-4I nrae-snmole test for moving averages.” J . Rou. Statist. soc., H, &I. I I , pp. 297-30>. 1949. I

• [6] 31. P. Barlert and P. €1. I)ianandn, “I<stensionz of Quenouille’s test for autoregressive scheme,” J. f?oy. Sfatid. Soc., H, vol. 12, pp. 10S-115! 19.50. [i) 31. S. Rnrtlett and I). X-. llajalabrhmsn. “Goodness of fit test for sirnnltxneous autoregrewive series,” J. Roy. Statist. Soc., B, [SI -4. 11. \Vnlker? “Sote on a generalization of the large aanlple v o l . 15, pp. 10-124, I9.X. goodness of fit test for linear antoregresive scheme,” J. Roy.

• Stafisf. Soc.. IZ, 1-01. 12, pp. 102-107, 1950.