T R A N S L A T I O N S O F

MATHEMATICAL M O N O G R A P H S V O L U M E 4 7

M. B. Nevelson R. Z. Has'minskff

Stochastic Approximation and Recursive Estimation

fl| American Mathematical Society Providence, Rhode Island

10.1090/mmono/047

CTOXACTMMECKAH AnnPOKCMMAUMf l

M PEKYPPEKTHOE OUEHMBAHM E

M. B. HEBEJlbCOH , P . 3 . XACbMMHCKM W

M3AaTejibCTBo "MayKa " MocKB a 197 2

Trans lated fro m th e R u s s i a n b y

I srae l Progra m fo r S c i e n t i f i c T r a n s l a t i o n s

Trans la t ion e d i t e d b y B . Si lve r

2000 Mathematics Subject Classification. Primar y 62L20 ; Secondary 60J05 , 93E10 .

Abstract. Thi s boo k i s devote d t o sequentia l method s o f solvin g a

c lass o f problem s t o whic h belongs , fo r example , th e proble m o f findin g a

maximum poin t o f a functio n i f eac h measure d valu e o f thi s functio n con -

tains a rando m error . Som e basi c procedure s o f stochasti c approximatio n

are investigate d fro m a singl e poin t o f view , namel y th e theor y o f Marko v

processes an d martingales . Example s ar e considere d o f application s o f

the theorem s t o som e problem s o f estimatio n theory , educationa l theor y an d

control theory , an d als o t o som e problem s o f informatio n transmissio n i n

the presenc e o f invers e feedback .

Library of Congress Cataloging in Publication Data w ^m ay

Nevel'son, Mikhai l Borisovich . Stochast ic approximatio n an d recursiv e estimation .

(Translations o f mathematica l monograph s ; v . ^7 ) Translat ion o f Stokhasticheskai a approksimatfe£ 3

i rekurrentnoe oftenivanie. Bibliography: p. Includes index . 1. Stochast i c approximation . 2 . Estimatio n

theory. I . Kha s 'minskii, Rafai l Zalmanovich , jo in t author. H . T i t l e . I I I . Ser ies . QA27lf.NWl5 519.2 76-U8298 ISBN 0-8218-1597-0

© 197 6 b y th e America n Mathematica l Society . Al l right s reserved . Reprinted b y th e America n Mathematica l Society , 2000 .

The America n Mathematica l Societ y retain s al l right s except thos e grante d t o th e Unite d State s Government .

Printed i n th e Unite d State s o f America .

@ Th e pape r use d i n thi s boo k i s acid-fre e an d fall s withi n th e guideline s established t o ensur e permanenc e an d durability .

Visit th e AM S hom e pag e a t ht tp: / /www.ams.org /

10 9 8 7 6 5 4 3 0 4 0 3 0 2 0 1 0 0

TABLE OF CONTENTS Preface 1 Introduction 4 Chapter 1 . Element s of probability and martingales 1 1

1. Probabilit y 1 1 2. Rando m variables. 1 3 3. Conditiona l probabilities and conditional expectations 1 7 4. Independence . Produc t measures 2 0 5. Martingale s and supermartingales 2 2

Chapter 2. Discret e time Markov processes 2 9 1. Marko v processes 2 9 2. Marko v processes and supermartingales 3 2 3. Recursivel y define d processe s 3 4 4. Discret e model of diffusion . 3 7 5. Exi t of sample functions fro m a domain 3 8 6. Serie s of independent rando m variables 4 1 7. Convergenc e of sample functions 4 3 8. Teachin g pattern recognition 4 9

Chapter 3. Marko v processes and stochastic equations 5 3 1. Continuou s time Markov processes 5 3 2. Stochasti c differential equations . 1 5 7 3. Stochasti c integrals 5 9 4. Stochasti c differential equations . II 6 2 5. Ito' s formula 6 4 6. Supermartingale s 6 9 7. Existenc e of solutions in the large 7 0 8. Exi t from a domain. Convergenc e of sample functions 7 3

Chapter 4. Convergenc e of stochastic approximation procedures . 1 7 9 1. Th e Robbins-Monro procedure 7 9 2. Th e Kiefer-Wolfowitz procedur e 8 3 3. Continuou s procedures 8 6 4. Convergenc e of the Robbins-Monro procedure 8 8 5. Convergenc e of the Kiefer-Wolfowitz procedur e 9 4

ui

CONTENTS

Chapter 5. Convergenc e o f stochasti c approximatio n procedures . I I 10 1 1. Introductor y remark s 10 1 2. Genera l theorems 10 2 3. Auxiliar y result s (continuous time ) 10 6 4. Auxiliar y result s (discrete time ) 11 2 5. One-dimensiona l procedure s 11 8

Chapter 6. Asymptoti c normalit y o f the Robbins-Monr o procedure 12 3 1. Preliminar y remark s 12 3 2. Asymptoti c behavior of solution s 12 9 3. Investigatio n o f the process ^(r) . 13 2 4. Investigatio n o f th e process I2{f) 13 6 5. Asymptoti c normalit y (continuou s time) 14 0 6. Asymptoti c normalit y (discrete time ) 14 7 7. Convergenc e o f moments 15 4

Chapter 7 . Som e modifications o f stochasti c approximatio n procedure s 16 1 1. Statemen t o f th e proble m 16 1 2. Genera l theore m 16 2 3. Auxiliar y result s 16 5 4. Theorem s on convergence and asymptotic normality 16 7 5. Adaptiv e Robbins-Monr o procedure s 16 9 6. Asymptoti c optimalit y 17 5

Chapter 8 . Recursiv e estimatio n (discrete time ) 18 1 1. Th e Crame'r-Rao inequality. Efficienc y o f estimate s 18 1 2. Th e Cramer-Rao inequality i n the multidimensiona l cas e 18 6 3. Estimatio n of a one-dimensional paramete r 18 9 4. Asymptoticall y efficien t recursiv e procedure 19 4 5. Estimatio n of a multidimensional parameter 19 7 6. Estimatio n with dependent observation s 20 2

Chapter 9. Recursiv e estimation (continuous time ) 20 7 1. Th e Cramer-Ra o inequalit y 20 7 2. Applicatio n o f th e Robbins-Monro procedur e 21 0 3. Time-dependen t observation s 21 2 4. Som e applications 21 3 5. A modification 22 0

Chapter 10 . Recursiv e estimation with a control parameter 22 1 1. Statemen t o f th e problem 22 1 2. Asymptoticall y optima l recursiv e plan 22 3 3. Tw o examples... . 22 6 4. Continuou s case 22 9

Notes on the literatur e 23 3 Bibliography 23 7 Subject Index 24 2 Main Notation 24 4

IV

This page intentionally left blank

NOTES ON THE LITERATURE

Chapter 1

The proo f o f al l theorem s cite d i n § § 1 - 4 ma y b e foun d i n Kolmogoro v [ 1 ] , Halmos

[ 1), an d Kolmorogo v an d Fomi n [ 1). Th e martingal e an d supermartingale concept s ar e du e

to Doob . Theorem s S. l an d 5.l ' an d thei r proof s ar e take n fro m Doo b [ 1 ] .

Chapter 2

§ 1. Fo r mor e detail s abou t propertie s o f Marko v processes an d thei r transitio n func -

tions, see Felle r [ 1 ], Dynkin [ 1 ], Loev e [ 1J, an d others .

§2. Theore m 2. 1 i s essentiall y a special cas e o f Doob' s theore m (Doo b [1 ) ) o n th e

stopping o f a supermartingale. Kolomogorov' s inequalit y fo r supermartingale s ma y als o b e

found i n Doo b [ 1 ].

§3. Theorem s 3. 1 an d 3. 2 ar e wel l known . See , fo r example , Gihman an d Skoroho d

HI. §5. Fo r related results , see, fo r example , AB R [ 1 ] , Chapter IV .

§6. Theorem s 6. 1 an d 6. 2 wer e prove d b y Kolomogoro v [1 ] i n a more genera l setting .

§7. Theore m 7. 1 i s conceptuall y simila r t o Theore m 8 i n Chapte r 4 o f AB R [ 1 ] .

§ 8. Fo r a detailed discussio n o f th e question s relate d t o thi s formulatio n o f th e prob -

lem, see AB R [ 1 ], Chapter 5 .

Chapter 3

§ 1. Fo r mor e detail s abou t th e definitio n an d propertie s o f a continuous Marko v

process, see Dynki n [ 1 ].

§2. Th e limi t passag e fro m equatio n (2.1 ) t o a continuou s Marko v process was con -

sidered b y Bernstei n [ 1 ], who, among othe r things , prove d a theorem o n th e limi t behavio r

of th e transitio n probabilit y a s h -* 0 .

§§3 an d 4 . Thi s definitio n o f th e stochasti c integra l i s du e t o ItS . Fo r a proo f o f

Theorem 4.1 , see, fo r example , Gihma n an d Skorohod [ 2 ] .

§7. A proo f o f Theore m 7. 1 ma y b e foun d i n Has'minski T [ 1 ] .

§8. Simila r results ar e presente d in th e authors ' paper [ 1 ] .

Chapter 4

§ 1. A s mentione d i n th e text , s.a . procedure s fo r locatio n o f th e root s o f regressio n

equations wer e first suggeste d i n th e 19S 1 pape r o f Robbin s an d Monr o [ 1 ] , who prove d

convergence i n mea n squar e o f th e procedur e (S.l ) unde r certai n assumptions . Thi s resul t

233

234 NOTES O N TH E LITERATUR E

was generalize d b y Wolfowit z [ 1 ], Blu m [ 1J, Chun g [ 1 ], an d others . Blu m [ 1 ), i n particular ,

presented th e firs t proo f o f convergenc e wit h probabilit y 1 . Theore m 1. 1 an d th e metho d o f

proof use d i n th e tex t ar e du e t o Gladyse v [ 1 ]. Multidimensiona l R M procedure s wer e intro -

duced b y Blu m [ 2 ] . Blu m was th e firs t autho r t o appl y th e theor y o f supermartingale s t o

investigate th e convergenc e o f s.a . procedures .

§2. Th e procedure (2.5 ) wa s propose d b y Kiefe r an d Wolfowit z [1 ] i n 1952 . Blu m

[ 2 ] , Derma n [ 1 ] , Burkholde r [1 ] an d other s studie d th e procedur e i n detai l and , i n particular ,

proved tha t i t i s convergen t wit h probabilit y 1 . N o accoun t i s given her e o f mor e genera l s.a .

procedures suc h a s those propose d i n Burkholde r [ 1 ] , Dvoretzk y [ 1 ] , etc . Mor e detaile d

bibliographies ma y b e foun d i n th e revie w article s o f Dvoretzk y [ 1 ] , Fabia n [ 3 ] , Sakriso n

[ 3 ], Schmettere r [ 1J, Logino v [ 1 ], an d others .

§ 3. Th e continuou s analo g o f th e R M procedur e wa s introduced b y Drim l an d Nedom a

[ 1 ] . The y prove d th e convergenc e o f th e procedur e unde r fairl y genera l assumption s o n th e

stochastic processe s appearin g on th e righ t o f th e equation . Unde r certain assumptions ,

Sakrison [ 1 ] prove d th e convergenc e o f th e continuou s analog of th e Kiefer-Wolfowit z pro -

cedure. Th e continuou s procedure s considere d her e wer e define d i n Has'rninsk u [1 ] an d

Nevel'son an d Has'minsk u [ 1 ].

§4. Th e firs t convergenc e theorem s fo r multidimensiona l R M procedures wer e prove d

by Blu m [ 2 ] ; see als o ABR [ 1 ] , Schmetterer [1 ] an d furthe r bibliograph y cite d there . Theo -

rem 4. 4 i s similar t o th e above-mentione d resul t o f Gladyse v [ 1 ]; Theore m 4. 5 i s du e t o

Braverman an d Rozonoe r (se e ABR [ 1 ]). Th e formulatio n o f th e las t theore m raise s th e

question a s to whethe r on e ca n als o weaken th e conditio n Lfl 2(f) < « > in th e proo f tha t

X(t) - + XQ &J . I t turn s ou t tha t thi s i s indee d possible . Fo r example , i n som e case s i t i s

sufficient t o deman d tha t £o"(f ) < °° for som e n > 0 (althoug h on e mus t the n impos e mor e

stringent condition s o n th e "noise") . Th e theorem s prove d her e fo r continuou s R M proced -

ures ar e du e t o th e author s [ 1 ] .

§ 5. Convergenc e theorem s fo r discret e multidimensiona l K W procedures wer e prove d

by Blu m [ 2 ] , Dupa£ [1 ] an d others. Fo r th e continuou s case , see Nevel'so n an d Has'minsk u

[ 1 ] .

Chapter 5

The conjectur e tha t a K W procedure canno t converg e wit h positiv e probabilit y t o

minimum point s o f th e regressio n functio n wa s advance d b y Fabia n [ 1 J, [ 2 ] . Th e result s

presented her e ar e based o n Has'minsk u (1 ] an d Nevel'son [ 1 ] , [ 2 ] . Th e tw o last-mentione d

papers als o prov e mor e genera l theorems. Se e als o Krasulin a [ 1 ] .

Chapter 6

§1 . Theore m 1. 1 fo r th e one-dimensiona l cas e i s proved , e.g. , i n Loev e [ 1 ]; for th e

multidimensional case , see Sack s [ 1 ].

§2. Condition s fo r th e validit y o f th e estimat e EX 2{t) = 0 ( l / f ) a s t - * » , wher e X(f)

is a discret e one-dimensiona l R M process , were firs t obtaine d b y Chun g [ 1 ]. A n analo g o f

Lemma 2. 1 fo r discret e multidimensiona l R M processes was prove d b y Sack s [ 1 ] unde r th e

additional assumptio n E\G(t, x, t j ) | 2 < c < « . Th e fac t tha t (2.9 ) i s a consequence o f (2.8 )

follows fro m a well-known lemm a of Chun g [ 1 ] .

§§3 an d 4 . Lemm a 3. 1 wa s prove d b y Has'minski f [ 2 ] . Lemm a 4. 3 i s du e t o Sack s

NOTES O N TH E LITERATUR E 235

§5. Theore m 5. 3 fo r M > 0 i s prove d i n Has'minski T [ 2 ] . Th e asymptoti c normalit y

of a R M procedure wa s established , unde r certai n assumptions , b y Holev o [ 1 ].

§6. Theore m 6. 1 wa s prove d b y Sack s under th e followin g additiona l assumptions :

a) th e matri x B ma y b e reduced t o diagona l for m b y a similarity transformatio n involvin g

an orthogona l matrix ; b ) th e nor m o f th e matri x A(t, x) i s bounde d fo r t > 1 an d x € Ej.

Theorem 6. 3 i s new. Asymptoti c normalit y theorem s fo r othe r s.a . procedure s wer e prove d

by Fabia n [ 5 ] , Derman [ 1 ] , Burkhoide r [ 1 ] , DupaS [1 ] an d others . Th e "truncation "

method i n asymptoti c normalit y proof s fo r s.a . procedure s was firs t applie d b y Hodge s an d

Lehman [ 1 ] .

§ 7. Th e firs t convergenc e theorem s fo r moment s o f s.a . processe s wer e prove d b y

Chung [ 1 ] i n th e one-dimensiona l case . Fo r the multidimensiona l case , the resul t o f Theore m

7.2 i s wel l know n (see , fo r example , Schmettere r [ 1J) fo r X > *ta . Bu t i f Conditio n (a ) o f

§ 7 hold s with A < Via, the onl y resul t know n t o u s o n convergenc e o f moment s i s tha t o f

Sakrison [ 2 ] . Constructin g a certai n modificatio n o f th e R M procedure , Sakrison prove d

that th e matri x o f secon d moment s o f th e proces s *Ji(X{i) — XQ) converges t o th e correspon -

ding covarianc e matri x o f a normal law , provided al l moment s E|G(f , x, (J))P t p = 1 , 2 , . . . ,

are bounded .

Chapter 7

§ § 1 - 4 . Alber t an d Gardne r [1 ] establishe d convergenc e an d asymptoti c normalit y

for th e procedur e (1.1 ) an d a more genera l on e i n whic h th e facto r a{t) i s allowe d t o depen d

on pas t observations . Thes e author s als o considere d a multidimensional modificatio n o f (1.1) .

Recently, Cslb i [ 1 ] , [2 ] ha s prove d convergenc e o f truncate d procedure s fo r a broad rang e

of truncations .

§5. Ther e i s a considerable literatur e o n th e optima l choic e o f parameter s fo r R M

procedures an d thei r adaptiv e modifications . Amon g others , we mentio n Chun g [ 1 ],

Dvoretzky [ 1 ] , Keste n [ 1 ] , Cypkin [ 1 ] , [2],an d Stratonovic ' [ 2 ] . Ou r approac h i s base d

on th e wor k o f Vente r [ 1 ] , who prove d Theore m 5. 1 unde r slightly mor e restrictiv e assump -

tions.

§6. Theore m 6. 1 slightl y generalize s a result o f Vente r [ 1 ] ; Theorem 6. 2 i s new .

Chapter 8

§1. Theore m 1. 1 follow s fro m result s o f Chapma n an d Robbin s [1 ] an d Kaga n [ 1 ] .

Theorem 1. 2 wa s prove d b y Ibragimo v and Has'minski i [ 2 ] . Thes e paper s als o deriv e inequal -

ities fo r biase d estimates .

§2. Theore m 2. 1 an d it s analo g fo r biase d estimate s wer e prove d b y Ibragimo v an d

Has'minskii [ 2 ] .

§ 3. Th e us e o f R M procedure s fo r parametri c estimatio n i s discusse d i n Alber t an d

Gardner [ 1 ], wher e a larger clas s o f estimates , no t necessaril y formin g Marko v processes , i s

considered.

§§4 an d 5 . Theorem s 4. 1 an d 4. 5 wer e prove d b y Sakriso n [ 2 ] , [3 ] unde r stronge r

conditions (an d without th e convergenc e o f distributions) . Othe r optimalit y criteri a wer e

considered b y Alber t an d Gardne r [ 1 ] , Cypkin [ 1 ] , and Stratonovic ' [ 2 ] , but thei r procedure s

generally depen d o n pas t observation s (see , fo r example , th e procedur e (0.9 ) i n th e Intro -

duction).

236 NOTES ON THE LITERATURE

Chapter 9

§1. Fo r inequality (1.7) , see, for example, Van Trees [1]. §2. A procedure equivalent to (2.4) for m linear in the parameter (m(*0) = mxg ,

where m is a nonsingular matrix) was considered by Holevo [1] . H e also considered the case that m is not a square matrix but mm* is nonsingular. Fo r the nonlinear case, Holevo proposed a procedure more complicated than (2.4). Unde r certain assumptions, he established asymptotic normality o f the estimation procedure.

§3. Recursiv e procedures similar to (3.1) ma y be derived from the Bayesian approach, using linearization. Fo r linear systems, similar estimation procedures may be found in Kal-man and Bucy [1] , Lipcer and Sirjaev [1] , and elsewhere.

§4. Th e problem of estimating x0 base d on observations of type (4.7) (estimation o f linear regression coefficients) ha s been considered from other points of view in many papers.

§5. Wit h suitable initial conditions, the procedure (5.3) , (5.4) for a linear function m{t, x) = m{t)x coincide s with the Kalman-Buc y optimal linear filtering equations . I n other words, this system is satisfied, in particular, by the conditional expectation of XQ given Y(s), s < f , provide d the a priori distribution of XQ is Gaussian.

BIBLIOGRAPHY

M. A . AIZERMAN , fe. M. BRAVERMA N AN D L L ROZONOE R (ABR ) 1. The method of potential functions in machine learning theory, "Nauka" , Moscow ,

1970. (Russian ) ARTHUR E . ALBER T AN D LELAN D A . GARNDER , JR .

1. Stochastic approximation and nonlinear regression, MI T Press , Cambridge, Mass., 1967. M R 3 6 #7261 .

SERGE BERNSTEI N 1. Principes de la theorie des equations differentiates stochastiques, Trud y Fiz.-Mat .

Inst. Steklov . Otd . Mat . S (1934) , 95-124 . DAVID BLACKWEL L AN D M . A . GIRSHIC K

1. Theory of games and statistical decisions, Wiley , Ne w York ; Chapman an d Hall , London, 1954 . M R 16 , 1135 .

JULIUS R . BLU M 1. Approximation methods which converge with probability one, Ann . Math. Statist .

25 (1954) , 382-386 . M R 15 , 973 . 2. Multidimensional stochastic approximation methods, Ann . Math . Statist . 25 (1954) ,

737-744 . M R 16 , 382 . D. L. BURKHOLDER

1. On a class of stochastic approximation processes, Ann . Math . Statist . 27 (1956) , 1044-1059. M R 1 9 , 7 1 .

DOUGLAS G . CHAPMA N AN D HERBERT ROBBIN S 1. Minimum variance estimation without regularity assumptions, Ann . Math. Statist .

22 (1951) , 581-586 . M R 13 , 367 . K. L CHUN G

1. On a stochastic approximation method, Ann . Math. Statist . 2 5 (1954) , 463—483 . MR 16 , 272 .

HARALD CRAME R 1. Mathematical methods of statistics, Princeto n Univ . Press , Princeton , N . J. , 1946 .

MR 8 , 39 . S. CSIBI

1. Simple and compound processes in iterative machine learning, Lecture s delivere d a t the Internationa l Summe r School , Trieste , 1971 .

2. Learning optimal decision functions recursively, Secon d Internat . Sympos . Infor-mation Theor y (Tsahkadsor , 1971) , Abstract s o f papers , Izdat . Akad . Nauk SSSR , Moscow, 1971 , pp . 2 9 2 - 2 9 4. CC A 7 #12990 .

JA. Z. CYPKI N 1. Adaptation and learning in automatic systems, "Nauka" , Moscow , 1968 ; English

transl., Academic Press , New York , 1971 . 2. Foundations of the theory of learning systems, "Nauka" , Moscow , 1970 ; English

transl., Academi c Press , New York , 1973 . CYRUS DERMA N

1. An application of Chung's lemma to the Kiefer-Wolfowitz stochastic approximation procedure, Ann . Math. Statist . 2 7 (1956) , 532-536 . M R 17 , 1218 .

A. G. D'JACKOV AND M. S. PlNSKER 1. 77ie optimal linear method of transmission over a memoryless Gaussian stationary

channel with complete feedback, Problem y Peredac i Informaci i 7 (1971) , vyp. 2 , 3 8 - 4 6 = Problem s of Information Transmissio n 7 (1971), 123-129 . M R 46 #8741 .

237

238 BIBLIOGRAPHY

J. L. DOOB 1. Stochastic processes, Wiley , New York ; Chapma n an d Hall , London, 1953 . M R

15 ,445 . MlLOSLAV DRIM L AN D JlR I NEDOM A

1. Stochastic approximations for continuous random functions, Trans . Second Pragu e Conf. Information Theory , Publ . House Czechoslova k Acad . Sci. , Prague , 1960 ; Academic Press , New York , 1961 , pp . 145-158 . M R 2 3 #A4171 .

VACLAV DUPA C 1. On the Kiefer-Wolfowitz approximation method, Casopi s Pest . Mat. 82 (1957) , 4 7 -

75. (Czech ; Russian an d Englis h summaries ) M R 19 , 693 . 2. A dynamic stochastic approximation method, Ann . Math. Statist . 3 6 (1965) , 1 6 9 5 -

1702. M R 3 3 #1939 . ARYEH DVORETZK Y

1. On stochastic approximation, Proc . Thir d Berkele y Sympos . Math . Statist.and Prob -ability (1954-55) , vol . I , Univ . o f Californi a Press , Berkeley , Cal. , 1956 , pp . 3 9 -55. M R 18 , 946 .

E B . DYNKIN 1. Markov processes, Fizmatgiz , Moscow , 1963 ; English transl. , Springer-Verlag, Berlin ,

Academic Press , New York , 1965 . M R 3 3 #1886 , 1887 . VACLAV FABIA N

1. Stochastic approximation methods, Czechoslova k Math . J . 1 0 (85 ) (1960) , 123 — 159. M R 22 # 4 0 9 1 .

2. A new one-dimensional stochastic approximation method for finding a local mini-mum of a function, Trans . Third Pragu e Conf . Informatio n Theory , Statistica l Decision Functions , Rando m Processe s (Liblice , 1962) , Publ . House Czech . Acad . Sci., Prague , 1964 , pp . 85-105 . M R 2 9 #6569 .

3. A survey of deterministic and stochastic approximation methods for the minimi-zation of functions, Kybernetik a (Prague ) 1 (1965) , 499-523 . (Czech ; English summary) M R 3 2 #5378 .

4. On the choice of design in stochastic approximation methods, Ann . Math . Statist . 39 (1968) , 457-465 . M R 3 7 # 1 0 3 8 .

5. On asymptotic normality in stochastic approximation, Ann . Math. Statist 3 9 (1968) , 1327-1332. M R 3 7 #6984 .

A. A . FEL'DBAU M 1. Optimal control systems, 2n d rev . ed. , "Nauka" , Moscow , 1966 ; English transl . o f

1st ed. , Academi c Press , New York , 1965 . M R 3 5 #2656 ; 3 2 #9075 . WILLIAM FELLE R

1. An introduction to probability theory and its applications. Vol . II , Wiley, Ne w York, 1966 . M R 3 5 #1048 .

G. M . FIHTENGOL' C 1. A course of differential and integral calculus. Vol . II , 4th ed. , Fizmatgiz , Moscow ,

1959; Germa n transl. , VE B Deutsche r Verla g de r Wissenschaften , Berlin , 1966 . MR 3 9 # l b .

1.1. GlHMAN AND A. V. SKOROHOD 1. Introduction to the theory of random processes, "Nauka" , Moscow, 1965 ; English

transl., Saunders , Philadelphia , Pa. , 1969 . M R 3 3 #6689 ; 4 0 # 9 2 3 . 2. Stochastic differential equations, "Naukov a Dumka" , Kiev , 1968 ; English transl. ,

Springer-Verlag, Ne w York , 1972 . M R 4 1 #7777 ; 4 9 #11625 . E G . GLADYSE V

1. On stochastic approximation, Teor . Verojatnost . i Primenen . 1 0 (1965) , 297 -30 0 = Theor . Probabilit y Appl . 1 0 (1965) , 275-278 . M R 3 2 #3184 .

PAUL R . HALMO S 1. Measure theory, Va n Nostrand , Princeton , N . J. , 1950 . M R 11 , 504 .

BIBLIOGRAPHY 239

R.Z.HAS'MINSIUI 1. Stability of systems of differential equations under random perturbations of their

parameters, "Nauka" , Moscow , 1969 . (Russian ) M R 41 #3925 . 2. The behavior of the processes of stochastic approximation for large time-variables,

Problemy Peredac i Informaci i 8 (1972) , vyp. 1 , 81-91 = Problem s o f Informatio n Transmission 8 (1972) , 6 2 - 7 0 . M R 4 8 # 1 3 2 1 .

J. L . HODGES , JR., AND E . L . LEHMAN N 1. Two approximations to the Robbins-Monro process, Proc . Third Berkele y Sympos .

Math. Statist , an d Probabilit y (1954-1955) , vol . I , Univ . of Californi a Press , Berke -ley, Ca!. , 1956 , pp . 95 -104 . M R 18 , 947 .

A. S. HOLEVO 1. Estimates for the drift parameters for a diffusion process by the method of sto-

chastic approximation, Studie s i n th e Theor y o f Self-adjustin g System s (V. G . Sragovic, editor) , Vycisl . Centr Akad . Nau k SSSR , Moscow , 1967 , pp . 179-200 . (Russian) M R 3 8 #1800 .

I. A . IBRAGIMO V AN D R . Z . HAS'MINSK H 1. The asymptotic behavior of certain statistical estimates in the smooth case. I : In-

vestigation of the likelihood ratio, Teor . Verojatnost . i Primenen . 1 7 (1972) , 4 6 9 -486 = Theor . Probabilit y Appl . 1 7 (1972) , 445-462 . M R 4 6 #10106 .

2. Information inequalities and superefficient methods, Dokl . Akad . Nau k SSSR 20 4 (1972), 1300-130 2 = Sovie t Math . Dokl . 1 3 (1972) , 821-824 . M R 4 7 # 2 7 3 3 .

3. On the approximation of statistical estimators by sums of independent variables, Dokl. Akad . Nau k SSS R 21 0 (1973) , 1273-127 6 = Sovie t Math . Dokl . 1 4 (1973) , 883-887 . M R 4 9 #10020 .

KlYOSI IT O 1. On a formula concerning stochastic differentials, Nagoy a Math . J . 3 (1951) , 55 -65 .

MR 13 , 363 . KlYOSI IT O AN D MAKIK O NlSI O

1. On stationary solutions of a stochastic differential equation, J . Math . Kyot o Univ . 4 (1964) , 1-75 . M R 3 1 # 1719 .

A.M.KAGAN 1. On the theory of Fisher's information quantity, Dokl . Akad. Nau k SSS R 15 1

(1963), 277-27 8 = Sovie t Math . Dokl . 4 (1963) , 991-993 . M R 2 7 #2032 . R. E . KALMA N AN D R . S . BUC Y

1. New results in linear filtering and prediction theory, Trans . ASME Ser . D . J . Basi c Engrg. 83 (1961) , 9 5 - 1 0 8 . M R 3 8 #3076 .

HARRY KESTE N 1. Accelerated stochastic approximation, Ann . Math . Statist . 29 (1958) , 4 1 - 5 9 . M R

20 # 3 7 1 . J. KlEFE R AN D J . WOLFOWIT Z

1. Stochastic estimation of the maximum of a regression function, Ann . Math . Statist . 23 (1952) , 462-466 . M R 14 , 299 ; erratum, M R 14 , 1278 .

A. N . KOLMOGORO V 1. Grundbegriffe der Wahrscheinlichkeitsrechnung, Springer-Verlag , Berlin, 1933 ; Eng-

lish transl. , Chelsea, Ne w York , 1950 ; 2nd ed. , 1956 . M R 11 , 374 ; 18 , 155 . A. N . KOLMOGOROV AN D S . V. FOMI N

1. Elements of the theory of functions and of functional analysis, 2n d rev . ed. , "Nauka", Moscow , 1968 ; English transl . o f 1s t ed. , vols . 1 , 2 , Grayloc k Press , Al -bany, N . Y. , 1957 , 1961 . M R 3 8 #2559 ; 19 , 44; 22 #9565a .

N. N . KRASOVSK U 1. Certain problems in the theory of stability of motion, Fizznatgiz , Moscow , 1959 ;

English transl. , Stability of motion. Application of Lyapunov fs second method to differential systems and equations with delay, Stanfor d Univ . Press , Stanford, Cal. , 1963. M R 2 1 #5047 ; 2 6 # 5 2 5 8 .

240 BIBLIOGRAPHY

T. P . KRASULIN A 1. The Robbins-Monro process in the case of several roots, Teor . Verojatnost . i

Primenen. 1 2 (1967) , 386-39 0 = Theor . Probabilit y Appl . 1 2 (1967) , 333-337 . MR 3 5 # 5 0 8 5 .

L M . LiBKIND 1. On the probability of large deviations for a quadratic functional of a stationary

Gaussian sequence, Problem y Peredac i Informaci i 5 (1969) , vyp . 2 , 7 2 - 7 8 = Prob -lems o f Informatio n Transmissio n 5 (1969) , no. 2 , 56—60 .

R. S. LlPCER AND A. N. SlRJAEV 1. Nonlinear filtering of Markov diffusion processes, Trud y Mat . Inst. Steklov . 10 4

(1968), 135-18 0 = Proc . Steklov Inst . Math . 10 4 (1968) , 163-21 8 (1971) . M R 41 #6342 .

2. The absolute continuity with respect to Wiener measure of measures that corre-spond to processes of diffusion type, Izv . Akad . Nau k SSS R 3 6 (1972) , 847-88 9 = Math . USS R Izv . 6 (1972) , 839-882 . M R 4 7 # 1 1 1 9 .

MICHEL LOEV E 1. Probability theory, 2n d ed. , Va n Nostrand , Princeton , N . J. , 1960 . M R 2 3 #A670 .

N . V . LOGINO V 1. Methods of stochastic approximation, Avtomat . i Telemeh . 2 7 (1966) , no . 4 , 1 8 5 -

204 = Automat . Remot e Contro l 2 7 (1966) , 706 -728 . I. G. MALKIN

1. Theory of stability of motion, 2n d rev . ed., "Nauka" , Moscow , 1966 ; Englis h transl. o f 1s t ed. , AE C Transl . #3352 , U . S . Atomic Energ y Commission , Oa k Ridge, Tenn. , 1958 . M R 3 4 #6228 ; erratum, 43 , p . 1695 .

M. B . NEVEL'SO N 1. Certain properties of continuous procedures of stochastic approximation, Teor .

Verojatnost. i Primenen . 1 7 (1972) , 310-31 9 = Theor . Probabilit y Appl . 1 7 (1972) , 298-308 . M R 4 5 #6124 .

2. The convergence of continuous and discrete Robbins-Monro procedures in the case of several roots of the regression equation, Problem y Peredac i Informaci i 8 (1972) , vyp. 3 , 48—57 = Problem s o f Informatio n Transmissio n 8 (1972) , 215—223. M R 48 #1320 .

M B . NEVEL'SO N AN D R . Z . HAS'MINSK H 1. Continuous procedures of stochastic approximation, Problem y Peredac i Informaci i

7 (1971) , vyp. 2 , 58-6 9 = Problem s o f Informatio n Transmissio n 7 (1971) , 1 3 9 -148. M R 4 8 #1319 .

2. On the convergence of the moments of the Robbins-Monro procedure, Avtomat . i Telemeh. 1973 , no. 1 , 96 -10 0 = Automat . Remot e Contro l 3 4 (1973) , 8 5 - 9 0 .

M. S. PlNSKER 1. The probability of error in block transmission in a memoryless Gaussian channel

with feedback, Problem y Peredac i Informaci i 4 (1968) , vyp. 4 , 3—1 9 = Problem s of Informatio n Transmissio n 4 (1968) , no. 4 , 1—14 .

C. RADHAKRISHN A RA O 1. Linear statistical inference and its applications, Wiley , New York , 1965 . M R

36 #4668 . HERBERT ROBBIN S AN D SUTTO N MONR O

1. A stochastic approximation method, Ann . Math . Statist . 2 2 (1951) , 400—407 . MR 13 , 144 .

JEROME SACK S 1. Asymptotic distribution of stochastic approximation procedures, Ann . Math. Statist .

29 (1958) , 373-405 . M R 2 0 #4886 . DAVID J . SAKRISO N

1. A continuous Kiefer-Wolfowitz procedure for random processes, Ann . Math. Statist . 35 (1964) , 590-599 . M R 2 8 #4650 .

BIBLIOGRAPHY 241

2. Efficient recursive estimation; application to estimating the parameters of a co-variance function, Internat . J . Engrg . Sci . 3 (1965) , 461-483. M R 3 1 #6306 .

3. Stochastic approximation: A recursive method for solving regression problems, Advances i n Communication Systems : Theor y an d Application s (A . V . Balakrishnan , editor), Vol . 2 , Academic Press , New York , 1966 , pp . 51-106 .

J. PlETE R M SCHALKWIJ K 1. Recent developments in feedback communication, Proc . IEE E 5 7 (1969) , 1242 —

1249. J. PlETE R M . SCHALKWIJ K AN D T . KAILAT H

1. A coding scheme for additive noise channels with feedback. I : No bandwidth con-straint, IEE E Trans . Informatio n Theor y IT-1 2 (1966) , 172-182 . M R 4 0 # 3 9 9 1 .

L SCHMETTERE R 1. Multidimensional stochastic approximation, Multivariat e Analysis , I I (Proc . Secon d

Internat. Sympos. , Dayton , Ohio , 1968 ) (P . R . Krishnaian , editor) , Academi c Press , New York , 1969 , pp . 443-460 . M R 4 1 #2844 .

C. E . SHANNO N 1. A mathematical theory of communication, Bel l Syste m Tech . J . 27 , (1948) , 3 9 7 -

423, 623-656 . M R 10 , 133 . A. V . SKOROHO D

1. Studies in the theory of random processes, Izdat . Kiev . Univ . Kiev , 1961 ; English transl., Addison-Wesley , Reading , Mass. , 1965 . M R 3 2 #3082a,b .

R. L STRATONOVI C 1. Conditional Markov processes and their application to the theory of optimal control,

Izdat. Moskov . Univ. , Moscow, 1966 ; English transl. , Amer . Elsevier , Ne w York , 1968. M R 3 3 # 5 3 9 1 ; 3 6 #4912 .

2. Does there exist a theory of synthesis of optimal adaptive, self-learning, and self-adaptive systems'*. Automat , i Telemeh . 1968 , no . 1 , 96-107 = Automat . Remot e Control 2 9 (1968) , 83 -92 . M R 3 9 #5210 .

HARRY L VA N TREE S 1. Detection, estimation and modulation theory. Par t I , Wiley, New York , 1968 .

J. H . VENTE R 1. An extension of the Robbins-Monro procedure , Ann . Math . Statist . 3 8 (1967) ,

181-190 . M R 3 4 #5225 . M. T . WASA N

1. Stochastic approximation, Cambridg e Univ . Press , New York , 1969 . M R 4 0 # 9 7 5 . J. WOLFOWIT Z

1. On the stochastic approximation method of Robbins and Monro, Ann . Math . Statist . 23 (1952) , 4 5 7 - 4 6 1 . M R 14 , 299 .

JOHN M. WOZENCRAF T AN D IRWI N MARK JACOB S 1. Principles of communication engineering, Wiley , Ne w York , 1965 .

K. S . ZlGANGIRO V 1. Transmission of messages in a binary Gaussian channel with feedback, Problem y

Peredaci Informaci i 3 (1967) , vyp. 2 , 98—10 1 = Problem s o f Informatio n Trans -mission 3 (1967) , no . 2 , 7 4 - 7 8 .

SUBJECT INDE X

Admissible control , 22 1 asymptotically optimal , 23 0

Asymptotic efficiency, 18 6 normality, 12 3 variance, 18 5

Borel set , 1 2 Chapman-Kolmogorov relation , 3 0 Condition (A) , 3 6

(B), 7 2 (a), 15 5

Control parameter , 22 1 Convergence

almost sure , 1 5 in distribution , 1 5 in probability , 1 5

Diffusion coefficient, 3 8 matrix, 6 7

Distribution o f a vector, 1 4 Drift

coefficient, 3 8 vector, 6 7

Estimate, 18 1 asymptotically efficient , 186 , 21 0

strongly, 185 , 188 , 20 9 weakly, 186 , 18 8

asymptotically unbiased , 18 2 efficient, 185 , 18 8 maximum likelihood , 5 recursive, 190 , 21 0 strongly consistent , 18 2 unbiased, 18 2 weakly consistent , 18 2

Expectation, 1 5 conditional, 17 , 1 8

relative t o a random vector , 1 9 finite, 1 5

Fisher information , 18 2 matrix, 18 6

Function Borel, 1 3 characteristic (o f a set), 1 7 distribution, 1 4 Ljapunov, 8 9 measurable, 1 3

Gaussian whit e noise , 6 3 Generating operator , 3 1

differential, 6 7 Independent

events, 2 0 random variable an d a-algebra , 2 0 sets o f rando m vectors , 2 0 0-algebras, 20 vectors, 2 0

Inequality Cebysev's, 1 5 CrameV-Rao (information) , 18 2 Kolmogorov's (fo r supermartingales) , 33 -3 4

Information content, 6 , 18 2 Fisher, 18 2 inequality, 18 2 matrix (Fisher) , 18 6

Integral, Lebesgue , 1 4 Itd's formula fo r stochasti c differentials , 66—6 7 Kiefer-Wolfowitz procedure , 83ff . Lebesgue integral , 1 4 Lemma

Fatou's, 1 6 Ljapunov's, 13 3

Martingale, 22 , 23 , 6 9 Measurable space , 1 2 Measure, 1 2

absolutely continuous , 1 6 complete, 1 2 finite, 12 Lebesgue, 1 2 probability, 1 3 a-finlte, 1 2

Observation process , 20 7 Plan o f experiment , 22 1

asymptotically optimal , 22 3 Probability, conditional , 17 , 1 8

relative t o a random vector , 1 9 Probability space , 1 3 Product

of measures , 2 1 of spaces , 2 0

Random variable, 1 3 independent o f th e future , 6 1 vector-valued, 1 3

242

SUBJECT INDE X 243

Recursive procedure , 19 0 truncated, 16 2

Regularity condition , 7 1 Regularization, 3 1 Robbins-Monro procedure , 4 , 79ff . Sample functio n o f a stochastic process , Separating hype r plane, 5 0 Set o f elementar y events , 1 2 o-algebras, 1 2

Borel, 1 2 generated b y a random vector , 1 9 imbedding o f on e i n another , 1 2 intersection of , 1 2 minimal, 1 2

Signalling rate , 21 6 Space o f elementar y events , 11 , 1 2 Stable

matrix, 13 2 point, 10 6

Standard Wiene r process , 5 5 with value s i n £ j , 5 6

Statistic, 18 1 Stochastic

approximation procedure s 4 , 79ff . Kiefer-Wolfowitz, 83ff . Robbins-Monro, 4 , 79ff .

differential (Ito) , 6 3 equivalence, 1 4

of stochasti c processes , 5 4 strong, 6 2

integral (Ito's) , 6 0 22, 5 3 process , 5 3

discrete, 2 2 Markov, 30 , 5 4

homogeneous, 5 4 measurable, 5 3 regular, 7 1 separable, 5 3 stationary (i n th e narro w sense) , 14 4

Supermartingale, 23 , 6 9 Theorem

Fubini's, 2 1 Ito's, 6 6 Kolmogorov's, 5 6 Lebesgue's, 1 6 Radon-Nikodym, 1 6

Transition densit y o f a Markov process , 5 4 Transition function , 30 , 5 4

homogeneous, 3 1 temporally homogeneous , 5 4

Transition probabilit y o f a Markov process , 2 9

MAIN NOTATIO N

11*11 = yfcijbff- nor m of matri x B = (tyy)) , p . 12 5

C2: clas s o f function s V(t, x) continuousl y differentiabl e wit h respec t t o t an d twic e con -

tinuously differentiabl e wit h respec t t o x, p . 5 7

C2' se t o f function s V(x) € C 2 wit h bounde d second-orde r partia l derivatives , p . 9 2

Dr: domai n o f definitio n o f generatin g operator , p . 3 1

D£ ' x ' : domai n o f definitio n o f generatin g operato r L a t poin t (t, x), p . 3 1

Eji euclidea n /-space , p . 1 2

7;: monoton e famil y o f a*lgebras , pp . 36 , 5 9

/(x): Fishe r informatio n matrix , pp . 182 , 18 6

/ : identit y matrix , p . 12 4

L: generatin g operator , pp . 31 , 6 7

1*2fo b]: se t o f measurabl e rando m function s /(f , u>) , t € [a, b], a ; e ft, ^-measurabl e fo r

every f , suc h tha t / £ / (f , w)d f < « • with probabilit y 1 , p . 5 9

)lf: a-algebr a o f event s generate d b y rando m variable s X(u), u < t, pp . 30 , 7 3

U: closur e o f se t U, p . 16 2

U€(B): e-neighborhoo d o f se t B , p . 3 9

U€.RW = K €<*> n <* : W < * * ' P - 4 0

K€(B) = EJ\U €{B), p . 3 9

1: domai n o f value s o f unknow n paramete r x, p . 18 1

X(t) ~9l(m , 5) : asymptoti c normalit y o f X(t) 9 p . 12 3

*W = <y x y w ) ,P-202 2: domai n o f value s o f contro l paramete r z , p . 22 1 Vg/Cx): vector wit h coordinate s \f(x + ce/ ) - / ( x — ce/)]/2c, p . 8 6

dv/bx: matri x wit h element s dty/dxy , p . 10 7

r^: firs t exi t tim e fro m domai n G , pp . 32 , 7 3

4>(B): clas s o f function s define d o n p . 4 0

244