OPTIMAL DESIGN OFCONTROL SYSTEMS

PURE AND APPLIED MATHEMATICS

A Program of Monographs, Textbooks, and Lecture Notes

EXECUTIVE EDITORS

Earl J. Taft Zuhair NashedRutgers University University of Delaware

New Brunswick, New Jersey Newark, Delaware

EDITORIAL BOARD

M. S. Baouendi Anil NerodeUniversity of California, Cornell University

San DiegoDonald Passman

Jane Cronin University of Wisconsin,Rutgers University Madison

Jack K. Hale Fred S. RobertsGeorgia Institute of Technology Rutgers University

S. Kobayashi Gian-Carlo RotaUniversity of California, Massachusetts Institute of

Berkeley Technology

Marvin Marcus David L. RussellUniversity of California, Virginia Polytechnic Institute

Santa Barbara and State University

W. S. Massey Walter SchemppYale University Universitat Siegen

Mark TeplyUniversity of Wisconsin,

Milwaukee

MONOGRAPHS AND TEXTBOOKS INPURE AND APPLIED MATHEMATICS

1. K. Yano, Integral Formulas in Riemannian Geometry (1970)2. S. Kobayashi, Hyperbolic Manifolds and Holomorphic Mappings (1970)3. V. S. Vladimirov, Equations of Mathematical Physics (A. Jeffrey, ed.; A. Littlewood,

trans.) (1970)4. B. N. Pshenichnyi, Necessary Conditions for an Extremum (L. Neustadt, translation

ed.; K. Makowski, trans.) (1971)5. L. Narici et a/., Functional Analysis and Valuation Theory (1971)6. S. S. Passman, Infinite Group Rings (1971)7. L. Domhoff, Group Representation Theory. Part A: Ordinary Representation Theory.

Part B: Modular Representation Theory (1971,1972)8. W. Boothbyand G. L. Weiss, eds., Symmetric Spaces (1972)9. Y. Matsushima, Differentiate Manifolds (E. T. Kobayashi, trans.) (1972)

10. L. E. Ward, Jr., Topology (1972)11. A. Babakhanian, Cohomological Methods in Group Theory (1972)12. R. Gilmer, Multiplicative Ideal Theory (1972)13. J. Ye/7, Stochastic Processes and the Wiener Integral (1973)14. J. Barros-Neto, Introduction to the Theory of Distributions (1973)15. R Larsen, Functional Analysis (1973)16. K. Yano and S. Ishihara, Tangent and Cotangent Bundles (1973)17. C. Procesi, Rings with Polynomial Identities (1973)18. R. Hermann, Geometry, Physics, and Systems (1973)19. N. R. Wallach, Harmonic Analysis on Homogeneous Spaces (1973)20. J. Dieudonne, Introduction to the Theory of Formal Groups (1973)21. /. Vaisman, Cohomology and Differential Forms (1973)22. B.-Y. Chen, Geometry of Submanifolds (1973)23. M. Marcus, Finite Dimensional Multilinear Algebra (in two parts) (1973,1975)24. R. Larsen, Banach Algebras (1973)25. R. O. Kujala and A. L Vitter, eds., Value Distribution Theory: Part A; Part B: Deficit

and Bezout Estimates by Wilhelm Stall (1973)26. K. B. Stolarsky, Algebraic Numbers and Diophantine Approximation (1974)27. A. R. Magid, The Separable Galois Theory of Commutative Rings (1974)28. B. R. McDonald, Finite Rings with Identity (1974)29. J. Satake, Linear Algebra (S. Koh et al., trans.) (1975)30. J. S. Go/an, Localization of Noncommutative Rings (1975)31. G. Klambauer, Mathematical Analysis (1975)32. M. K. Agoston, Algebraic Topology (1976)33. K. R. Goodearl, Ring Theory (1976)34. L. E. Mansfield, Linear Algebra with Geometric Applications (1976)35. N. J. Pullman, Matrix Theory and Its Applications (1976)36. B. R. McDonald, Geometric Algebra Over Local Rings (1976)37. C. W. Groetsch, Generalized Inverses of Linear Operators (1977)38. J. E. Kuczkowski and J. L. Gersting, Abstract Algebra (1977)39. C. O. Christenson and W. L. Voxman, Aspects of Topology (1977)40. M. Nagata, Field Theory (1977)41. R. L. Long, Algebraic Number Theory (1977)42. W. F. Pfeffer, Integrals and Measures (1977)43. R. L Wheeden and A. Zygmund, Measure and Integral (1977)44. J. H. Curtiss, Introduction to Functions of a Complex Variable (1978)45. K. Hrbacek and T. Jech, Introduction to Set Theory (1978)46. W. S. Massey, Homology and Cohomology Theory (1978)47. M. Marcus, Introduction to Modern Algebra (1978)48. £ C. Young, Vector and Tensor Analysis (1978)49. S. B. Nadler, Jr., Hyperspaces of Sets (1978)50. S. K. Sega/, Topics in Group Kings (1978)51. A. C. M. van Rooij, Non-Archimedean Functional Analysis (1978)52. L. Corwin and R. Szczarba, Calculus in Vector Spaces (1979)53. C. Sadosky, Interpolation of Operators and Singular Integrals (1979)54. J. Cronin, Differential Equations (1980)55. C. W. Groetsch, Elements of Applicable Functional Analysis (1980)

56. /. Vaisman, Foundations of Three-Dimensional Euclidean Geometry (1980)57. H. /, Freedan, Deterministic Mathematical Models in Population Ecology (1980)58. S. B. Chae, Lebesgue Integration (1980)59. C. S. Rees ef a/., Theory and Applications of Fourier Analysis (1981)60. L Nachbin, Introduction to Functional Analysis (R. M. Aron, trans.) (1981)61. G. Orzech and M. Orzech, Plane Algebraic Curves (1981)62. R. Johnsonbaugh and W. E. Pfaffenberger, Foundations of Mathematical Analysis

(1981)63. W. L. Voxman and R. H. Goetschel, Advanced Calculus (1981)64. L J. Corwin and R. H. Szczarba, Multivariable Calculus (1982)65. V. /. Istratescu, Introduction to Linear Operator Theory (1981)66. R. D. Jarvinen, Finite and Infinite Dimensional Linear Spaces (1981)67. J. K. Been? and P. E. Ehriich, Global Lorentzian Geometry (1981)68. D. L Armacost, The Structure of Locally Compact Abelian Groups (1981)69. J. W. Brewer and M. K. Smith, eds., Emmy Noether: A Tribute (1981)70. K. H. Kim, Boolean Matrix Theory and Applications (1982)71. T. W. Wieting, The Mathematical Theory of Chromatic Plane Ornaments (1982)72. D. B.Gauld, Differential Topology (1982)73. R. L. Faber, Foundations of Euclidean and Non-Euclidean Geometry (1983)74. M. Carmeli, Statistical Theory and Random Matrices (1983)75. J. H. Carruth et a/., The Theory of Topological Semigroups (1983)76. R. L. Faber, Differential Geometry and Relativity Theory (1983)77. S. Bamett, Polynomials and Linear Control Systems (1983)78. G. Karpilovsky, Commutative Group Algebras (1983)79. F. Van Oystaeyen and A. Verschoren, Relative Invariants of Rings (1983)80. /. Vaisman, A First Course in Differential Geometry (1984)81. G. W. Swan, Applications of Optimal Control Theory in Biomedicine (1984)82. T. Petrie andJ. D. Randall, Transformation Groups on Manifolds (1984)83. K. Goebel and S. Reich, Uniform Convexity, Hyperbolic Geometry, and Nonexpansive

Mappings (1984)84. T. Albu and C. Nastasescu, Relative Finiteness in Module Theory (1984)85. K. Hrbacek and T. Jech, Introduction to Set Theory: Second Edition (1984)86. F. Van Oystaeyen and A. Verschoren, Relative Invariants of Rings (1984)87. B. R. McDonald, Linear Algebra Over Commutative Rings (1984)88. M. Namba, Geometry of Projective Algebraic Curves (1984)89. G. F. Webb, Theory of Nonlinear Age-Dependent Population Dynamics (1985)90. M. R. Bremner et ai, Tables of Dominant Weight Multiplicities for Representations of

Simple Lie Algebras (1985)91. A. E. Fekete, Real Linear Algebra (1985)92. S. B. Chae, Holomorphy and Calculus in Normed Spaces (1985)93. A. J. Jem, Introduction to Integral Equations with Applications (1985)94. G. Karpilovsky, Projective Representations of Finite Groups (1985)95. L. Narici and E. Beckenstein, Topological Vector Spaces (1985)96. J. Weeks, The Shape of Space (1985)97. P. R. Gribik and K. O. Kortanek, Extremal Methods of Operations Research (1985)98. J.-A. Chao and W. A. Woyczynsk!, eds., Probability Theory and Harmonic Analysis

(1986)99. G. D. Crown et a/.. Abstract Algebra (1986)

100. J. H. Carruth et a/., The Theory of Topological Semigroups, Volume 2 (1986)101. R. S. Dora/7 and V. A. Belfi, Characterizations of C*-Algebras (1986)102. M. W. Jeter, Mathematical Programming (1986)103. M. Altman, A Unified Theory of Nonlinear Operator and Evolution Equations with

Applications (1986)104. A. Verschoren, Relative Invariants of Sheaves (1987)105. R. A. Usmani, Applied Linear Algebra (1987)106. P. Blass and J. Lang, Zariski Surfaces and Differential Equations in Characteristic p >

0(1987)107. J. A. Reneke et ai, Structured Hereditary Systems (1987)108. H. Busemann and B. B. Phadke, Spaces with Distinguished Geodesies (1987)109. R. Harte, Invertibility and Singularity for Bounded Linear Operators (1988)110. G. S. Ladde et at., Oscillation Theory of Differential Equations with Deviating Argu-

ments (1987)111. L. Dudkin et a/., Iterative Aggregation Theory (1987)112. T. Okubo, Differential Geometry (1987)113. D. L. Stand and M. L. Stand, Real Analysis with Point-Set Topology (1987)

114. T. C. Card, Introduction to Stochastic Differential Equations (1988)115. S. S. Abhyankar, Enumerative Combinatorics of Young Tableaux (1988)116. H. Sfracfe and R. Famsteiner, Modular Lie Algebras and Their Representations (1988)117. J. A. Huckaba, Commutative Rings with Zero Divisors (1988)118. W. D. Wallis, Combinatorial Designs (1988)119. W. W;{?s/aw, Topological Fields (1988)120. G. Karpilovsky, Field Theory (1988)121. S. Caenepeel and F. Van Oystaeyen, Brauer Groups and the Cohomology of Graded

Rings (1989)122. W. Kozlowski, Modular Function Spaces (1988)123. E. Lowen-Colebunders, Function Classes of Cauchy Continuous Maps (1989)124. M. Pave/, Fundamentals of Pattern Recognition (1989)125. V. Lakshmikantham et a/., Stability Analysis of Nonlinear Systems (1989)126. R. Sivaramakrishnan, The Classical Theory of Arithmetic Functions (1989)127. N. A. Watson, Parabolic Equations on an Infinite Strip (1989)128. K. J. Hastings, Introduction to the Mathematics of Operations Research (1989)129. B. Fine, Algebraic Theory of the Bianchi Groups (1989)130. D. N. Dikranjan et a/., Topological Groups (1989)131. J.C. Morgan II, Point Set Theory (1990)132. P. Biter and A. Witkowski, Problems in Mathematical Analysis (1990)133. H. J. Sussmann, Nonlinear Controllability and Optimal Control (1990)134. J.-P. Florens et a/., Elements of Bayesian Statistics (1990)135. N. Shell, Topological Fields and Near Valuations (1990)136. B. F. Doolin and C. F. Martin, Introduction to Differential Geometry for Engineers

(1990)137. S. S. Holland, Jr., Applied Analysis by the Hilbert Space Method (1990)138. J. Okninski, Semigroup Algebras (1990)139. K. Zhu, Operator Theory in Function Spaces (1990)140. G. B. Price, An Introduction to Multicomplex Spaces and Functions (1991)141. R. B. Darst, Introduction to Linear Programming (1991)142. P. L Sachdev, Nonlinear Ordinary Differential Equations and Their Applications (1991)143. T. Husain, Orthogonal Schauder Bases (1991)144. J. Foran, Fundamentals of Real Analysis (1991)145. W. C. Brown, Matrices and Vector Spaces (1991)146. M. M. RaoandZ. D. Ron, Theory of Oriicz Spaces (1991)147. J. S. Go/an and T. Head, Modules and the Structures of Rings (1991)148. C. Small, Arithmetic of Finite Fields (1991)149. K. Yang, Complex Algebraic Geometry (1991)150. D. G. Hoffman et a/., Coding Theory (1991)151. M. O. Gonzalez, Classical Complex Analysis (1992)152. M. O. Gonzalez, Complex Analysis (1992)153. L W. Baggett, Functional Analysis (1992)154. M. Sniedovich, Dynamic Programming (1992)155. R. P. Agarwal, Difference Equations and Inequalities (1992)156. C. Brezinski, Biorthogonality and Its Applications to Numerical Analysis (1992)157. C. Swartz, An Introduction to Functional Analysis (1992)158. S. 8. Nadler, Jr., Continuum Theory (1992)159. M. A. AI-Gwaiz, Theory of Distributions (1992)160. E. Perry, Geometry: Axiomatic Developments with Problem Solving (1992)161. E. Castillo and M. R. Ruiz-Cobo, Functional Equations and Modelling in Science and

Engineering (1992)162. A. J. Jerri, Integral and Discrete Transforms with Applications and Error Analysis

(1992)163. A. Charlieret a/., Tensors and the Clifford Algebra (1992)164. P. Bilerand T. Nadzieja, Problems and Examples in Differential Equations (1992)165. E. Hansen, Global Optimization Using Interval Analysis (1992)166. S. Guerre-Delabriere, Classical Sequences in Banach Spaces (1992)167. Y. C. Wong, Introductory Theory of Topological Vector Spaces (1992)168. S. H. Kulkami and B. V. Limaye, Real Function Algebras (1992)169. W. C. Brown, Matrices Over Commutative Rings (1993)170. J. LoustauandM. Dillon, Linear Geometry with Computer Graphics (1993)171. W. V. Petryshyn, Approximation-Solvability of Nonlinear Functional and Differential

Equations (1993)172. E. C. Young, Vector and Tensor Analysis: Second Edition (1993)173. T. A. Bick, Elementary Boundary Value Problems (1993)

174. M. Pave/, Fundamentals of Pattern Recognition: Second Edition (1993)175. S. A. Albeverio et al., Noncommutative Distributions (1993)176. W. Fulks, Complex Variables (1993)177. M. M. Rao, Conditional Measures and Applications (1993)178. A. Janicki and A. Weron, Simulation and Chaotic Behavior of a-Stable Stochastic

Processes (1994)179. P. Neittaanmaki and D. Tiba, Optimal Control of Nonlinear Parabolic Systems (1994)180. J. Cronin, Differential Equations: Introduction and Qualitative Theory, Second Edition

(1994)181. S. HeikkilS and V. Lakshmikantham, Monotone Iterative Techniques for Discontinuous

Nonlinear Differential Equations (1994)182. X. Mao, Exponential Stability of Stochastic Differential Equations (1994)183. B. S. Thomson, Symmetric Properties of Real Functions (1994)184. J. E. Rub/o, Optimization and Nonstandard Analysis (1994)185. J. L Bueso et al., Compatibility, Stability, and Sheaves (1995)186. A. N. Michel and K. Wang, Qualitative Theory of Dynamical Systems (1995)187. M. R. Darnel, Theory of Lattice-Ordered Groups (1995)188. Z. Naniewicz and P. D. Panagiotopoulos, Mathematical Theory of Hemivariational

Inequalities and Applications (1995)189. L. J. Co/w/n and R. H. Szczarta, Calculus in Vector Spaces: Second Edition (1995)190. L. H. Erbe et al., Oscillation Theory for Functional Differential Equations (1995)191. S. Agaian etal.. Binary Polynomial Transforms and Nonlinear Digital Filters (1995)192. M, I. Gil', Norm Estimations for Operation-Valued Functions and Applications (1995)193. P. A. Grillet, Semigroups: An Introduction to the Structure Theory (1995)194. S. Kichenassamy, Nonlinear Wave Equations (1996)195. V. F. Krotov, Global Methods in Optimal Control Theory (1996)196. K. /. Beidaret al.. Rings with Generalized Identities (1996)197. V. I. Amautov et al., Introduction to the Theory of Topological Rings and Modules

(1996)198. G. S/erfcsma, Linear and Integer Programming (1996)199. R. Lasser, Introduction to Fourier Series (1996)200. V. Sima, Algorithms for Linear-Quadratic Optimization (1996)201. D. Redmond, Number Theory (1996)202. J. K. Beem et al., Global Lorentzian Geometry: Second Edition (1996)203. M. Fontana et al., Priifer Domains (1997)204. H. Tanabe, Functional Analytic Methods for Partial Differential Equations (1997)205. C. Q. Zhang, Integer Flows and Cycle Covers of Graphs (1997)206. E. Spiegel and C. J. O'Donnell, Incidence Algebras (1997)207. B. Jakubczyk and W. Respondek, Geometry of Feedback and Optimal Control (1998)208. T. W. Haynes et al., Fundamentals of Domination in Graphs (1998)209. T. W. Haynes et al., Domination in Graphs: Advanced Topics (1998)210. L. A. D'Alotto et al., A Unified Signal Algebra Approach to Two-Dimensional Parallel

Digital Signal Processing (1998)211. F. Halter-Koch, Ideal Systems (1998)212. N. K. Goviletal., Approximation Theory (1998)213. R. Cross, Multivalued Linear Operators (1998)214. A. A. Martynyuk, Stability by Liapunov's Matrix Function Method with Applications

(1998)215. A. FaviniandA. Yagi, Degenerate Differential Equations in Banach Spaces (1999)216. A. Illanes and S. Nadler, Jr., Hyperspaces: Fundamentals and Recent Advances

(1999)217. G. Kato and D. Struppa, Fundamentals of Algebraic Microlocal Analysis (1999)218. G. X.-Z. Yuan, KKM Theory and Applications in Nonlinear Analysis (1999)219. D. Motreanu and N. H. Pavel, Tangency, Flow Invariance for Differential Equations,

and Optimization Problems (1999)220. K. Hrbacek and T. Jech, Introduction to Set Theory, Third Edition (1999)221. G. £ Kolosov, Optimal Design of Control Systems (1999)222. A. I. Prilepko et al., Methods for Solving Inverse Problems in Mathematical Physics

(1999)

Additional Volumes in Preparation

OPTIMAL DESIGN OFCONTROL SYSTEMS

Stochastic andDeterministic Problems

G. E. KolosovMoscow University ofElectronics and MathematicsMoscow, Russia

M A R C E L

MARCEL DEKKER, INC. NEW YORK • BASEL

Library of Congress Cataloging-in-Publication Data

Kolosov, G. E. (Gennadil Evgen'evich)Optimal design of control systems: stochastic and deterministic problems / G. E.

Kolosov.p. cm.— (Monographs and textbooks in pure and applied mathematics; 221)

Includes bibliographical references and index.ISDN 0-8247-7537-6 (alk. paper)1. Control theory. 2. Mathematical optimization. I. Title. II. Scries.

QA402.3.K577 1999629.8312—dc21 99-30940

CIP

This book is printed on acid-free paper.

HeadquartersMarcel Dekker, Inc.270 Madison Avenue, New York, NY 10016tel: 212-696-9000; fax: 212-685-4540

Eastern Hemisphere DistributionMarcel Dekker AGHutgasse 4, Postfach 812, CH-4001 Basel, Switzerlandtel: 41-61-261-8482; fax: 41-61-261-8896

World Wide Webhttp://www.dekker.com

The publisher offers discounts on this book when ordered in bulk quantities. For more infor-mation, write to Special Sales/Professional Marketing at the headquarters address above.

Copyright © 1999 by Marcel Dekker, Inc. AH Rights Reserved.

Neither this book nor any part may be reproduced or transmitted in any form or by anymeans, electronic or mechanical, including photocopying, microfilming, and recording, orby any information storage and retrieval system, without permission in writing from thepublisher.

Current printing (last digit):1 0 9 8 7 6 5 4 3 2 1

PRINTED IN THE UNITED STATES OF AMERICA

PREFACE

The rise of optimal control theory is a remarkable example of interactionbetween practical needs and mathematical theories.

Indeed, in the middle of this century the development of various auto-matic control systems in technology and of systems for control of motionof mechanical objects (in particular, of flying objects such as airplanes androckets) gave rise to specific mathematical problems concerned with findingthe conditional extremum of functions or functionals, which could not besolved by means of the methods of classical mathematical analysis and thecalculus of variations.

Extreme urgency of these problems for practical needs stimulated theefforts of mathematicians to develop methods for solving these new prob-lems. At the end of the fifties and at the beginning of the sixties, theseefforts were crowned with success when new mathematical approaches suchas Pontryagin's maximum principle, Bellman's dynamic programming, andlinear and convex programming (developed somewhat earlier by L. Kan-torovich, G. Dantzig, and others) were established. These new approachesgreatly affected the research carried out in control theory at that time. Itshould be noted that these approaches have played a very important rolein the process of formation of optimal control theory as an independentbranch of science. One can say that the role of the maximum principle anddynamic programming in the theory of optimal control is as significant asthat of Maxwell's equations in electromagnetic theory in physics.

Optimal control theory evolved most intensively at the end of the sixtiesand during the seventies. This period showed a very high degree of coop-eration and interaction between mathematicians and all those dealing withapplications of control theory in technology, mechanics, physics, chemistry,biology, etc.

Later on, a gap between the purely mathematical and the practical ap-proach to solving applied problems of optimal control began to emerge andis now apparent. Although the appearance of this gap can be explained byquite natural reasons, nevertheless, the further growth of this trend seemsto be undesirable. The author hopes that this book will to some extentreduce the gap between these two branches of research.

iii

IV Preface

This book is primarily intended for specialists dealing with applicationsof control theory. It is well known that the use of such approaches as, say,the maximum principle or dynamic programming often leads to optimalcontrol algorithms whose implementation for actual real-time plants en-counters great (sometimes insurmountable) difficulties. This is the reasonthat for solving control problems in practice one often employs methodsbased on various simplifications and heuristic concepts. Naturally, thisresults in losses in optimality but makes it possible to obtain control al-gorithms that allow simple technological implementations. In some casesthe use of simplifications and heuristic concepts can also result in signif-icant deviations of the system performance index from its optimal value(Chapter VI).

In this book we describe ways for constructing simply realizable algo-rithms of optimal (suboptimal) control, which are based on the dynamicprogramming approach. These algorithms are derived on the basis of exact,approximate analytical, or numerical solutions of differential and functionalBellman equations corresponding to the control problems considered.

The book contains an introduction and seven chapters. Chapter I dealswith some general concepts of control theory and the description of math-ematical approaches to solving problems of optimal control. We considerboth deterministic and stochastic models of controlled systems and discussthe distinguishing features of stochastic models, which arise due to possibleambiguous interpretation of solutions to stochastic differential equationsdescribing controlled systems with white noise disturbances.

We define the synthesis problem as the principal problem of optimalcontrol theory and give a general scheme of the dynamic programming ap-proach. The Bellman equations for deterministic and stochastic controlproblems (for Markov models and stochastic models with indirect obser-vations) are studied. For problems with infinite horizon we introduce theconcept of stationary operating conditions, which is widely used in furtherchapters of the book.

Exact methods of synthesis are considered in Chapter II. We describe theexceptional cases in which the Bellman equations have exact solutions, andhence the optimal control algorithms can be obtained in explicit analyticalforms.

First (in §2.1), we briefly discuss some well-known results concerned withsolution of the so-called LQ-problems. Next, in §§2.2-2.4, we write exact so-lutions for three specific problems of optimal control with bounded controlactions. We consider deterministic and stochastic problems of control ofthe population size and the problem of constructing an optimal servomech-anism. In these systems, the optimal controllers are of the "bang-bang"form, and the switch point coordinates are given by finite formulas.

Preface v

The following four chapters are devoted to the description of approxi-mate methods for synthesis. In this case, the design of suboptimal controlsystems is based, as a rule, on using the approximate solutions of the cor-responding Bellman equations. To obtain these approximate solutions, wemainly use various versions of small parameter methods or successive ap-proximation procedures.

In Chapter III we study weakly controlled systems. We consider controlproblems with bounded controls and assume that the values of admissiblecontrol actions are small. This stipulates the appearance of a small param-eter in the nonlinear term in the Bellman equation. This, in turn, makes itpossible to propose a natural successive approximation procedure for solv-ing the Bellman equation, and thus the synthesis problem, approximately.This procedure is a modification of the well-known Picard and Bellmanprocedures which provide a way for obtaining approximate solutions ofnonlinear differential equations by solving a sequence of linear equations.

Chapter III is organized as follows. First (in §3.1), we describe thegeneral scheme of approximate synthesis for controlled systems under sta-tionary operating conditions. Next (in §3.2), by using this general scheme,we calculate a suboptimal controller for an oscillatory system with one de-gree of freedom. Later (in §3.3 and §3.4), we generalize our approach tononstationary problems and to the case of correlated disturbances; then weestimate the error obtained. In §3.5 we prove that the successive approx-imation procedure in question converges asymptotically. Finally (in §3.6),we apply this approach to an approximate design of a stochastic systemwith distributed parameters.

Chapter IV is about stochastic controlled systems with noises of smallintensities. In this case, the diffusion terms in the Bellman equation con-tain small coefficients. Under certain assumptions this allows us to replacethe initial stochastic problem by a sequence of auxiliary deterministic prob-lems of optimal control whose solutions (i) can be calculated more easilyand (ii) give a way for designing suboptimal control systems (with respectto the initial stochastic problem). This approach is used for calculatingsuboptimal controllers for two specific servomechanisms.

In Chapter V we consider a class of controlled systems whose dynamicsare quasiharmonic. The trajectories of such systems are close to harmonicoscillations, and this is the reason that the well-developed techniques of thetheory of nonlinear oscillations can be effectively applied for studying thesesystems. By using polar coordinates as the phase variables, we describethe system state in terms of slowly changing amplitude and phase. Thepresence of a small parameter on the right-hand sides of the differentialequations for these variables allows us to elaborate different versions ofapproximate solutions for the various problems of optimal control. These

vi Preface

solutions are based on the use of appropriate asymptotic expansions of theperformance index, the optimal control algorithm, etc. in powers of thesmall parameter.

We illustrate these techniques by solving four specific problems of op-timal damping of deterministic and stochastic oscillations in a biologicalpredator-prey system and in a mechanical system with oscillatory dynam-ics.

In Chapter VI we discuss some special asymptotic methods of synthesiswhich do not belong to the classes of control problems studied in Chap-ters III-V. We consider the problems of control of plants with unknownparameters (the adaptive control problems), in which the a priori uncer-tainty of their values is small. In addition, we study stochastic controlproblems with bounded phase variables and a problem of optimal controlof the population size whose behavior is governed by a stochastic logisticequation with a large value of the medium capacity. We use small parameterapproaches for solving the problems mentioned above. For the constructionof suboptimal controls, we employ the asymptotic series expansions for theloss functions and the optimal control algorithms. The error obtained isestimated.

Numerical methods of synthesis are covered in the final Chapter VII.We discuss the problem of the assignment of boundary conditions to gridfunctions and propose some different schemes for solving specific problemsof optimal control. The numerical methods proposed are used for solvingspecific synthesis problems.

The presentation of all the approaches studied in the book is accompa-nied by numerous examples of actual control problems. All calculationsare carried out up to the accuracy level sufficient for comparatively simpleimplementation of the optimal (suboptimal) algorithms obtained in actualdevices. In many cases, the algorithms are presented in the form of analo-gous circuits or flow charts.

The book can be helpful to students, postgraduate students, and special-ists working in the field of automatic control and applied mathematics. Thebook may be of interest to mechanical and electrical engineers, physicistsand biologists. Only knowledge of the foundations of probability theory isrequired for assimilating the subject matter of the book.

The reader should be acquainted with basic notions of probability theorysuch as random events and random variables, the probability distributionfunction and the probability density of random variables, the mean valueof a random variable, inconsistent and independent random events andvariables, etc. It is not compulsory to know the foundations of the theoryof random processes, since Chapter I provides all necessary facts about themethods for describing random processes that are encountered further in

Preface vii

the book. This makes the book accessible to a wide circle of students andspecialists who are interested in applications of optimal control theory.

The author's intention to write this book was supported by R. L. Stra-tonovich, who was the supervisor of the author's Ph.D thesis and for manyyears till his sudden death in 1997 remained the author's friend.

The author wishes to express his deep gratitude to V. B. Kolmanovskii,R. S. Liptser, and all participants of the seminar "Stability and Control" atthe Moscow University of Electronics and Mathematics for useful remarksand advice concerning the contents of this book.

The author's special thanks go to M. A. Shishkova for translating themanuscript into English and keyboarding.

G. E. Kolosov

CONTENTS

Preface vIntroduction 1

Chapter I. Synthesis Problems for ControlSystems and the Dynamic ProgrammingApproach 7

1.1. Statement of synthesis problems for optimal controlsystems 7

1.2. Differential equations for controlled systems withrandom functions 32

1.3. Deterministic control problems. Formal scheme ofthe dynamic programming approach 48

1.4. The Bellman equations for Markov controlled pro-cesses 57

1.5. Sufficient coordinates in control problems with indi-rect observations 75

Chapter II. Exact Methods for Synthesis Prob-lems 93

2.1. Linear-quadratic problems of optimal control (LQ-problems) 93

2.2. Problem of optimal tracking a wandering coordinate 1032.3. Optimal control of the population size 1232.4. Stochastic problem of optimal fisheries management 133

Chapter III. Approximate Synthesis of Stochas-tic Control Systems With Small ControlActions 141

3.1. Approximate solution of stationary synthesis prob-lems 144

3.2. Calculation of a quasioptimal regulator for the os-cillatory plant 154

IX

Contents

3.3. Synthesis of quasioptimal controls in the case of cor-related noises 164

3.4. Nonstationary problems. Estimates of the qualityof approximate synthesis 175

3.5. Analysis of the asymptotic convergence of successiveapproximations (3.0.6)-(3.0.8) as k —>• oo 188

3.6. Approximate synthesis of some stochastic systemswith distributed parameters 199

Chapter IV. Synthesis of Quasioptimal Systemsin the Case of Small Diffusion Terms in theBellman Equation 219

4.1. Approximate synthesis of a servomechanism withsmall-intensity noise 221

4.2. Calculation of a quasioptimal system for tracking adiscrete Markov process 233

Chapter V. Control of Oscillatory Systems 2475.1. Optimal control of a quasiharmonic oscillator. An

asymptotic synthesis method 2485.2. Control of the "predator-prey" system. The case of

a poorly adapted predator 2675.3. Optimal damping of random oscillations 2765.4. Optimal control of quasiharmonic systems with noise

in the feedback circuit 298

Chapter VI. Some Special Applications ofAsymptotic Synthesis Methods 311

6.1. Adaptive problems of optimal control 3126.2. Some stochastic control problems with constrained

phase coordinates 3286.3. Optimal control of the population size governed by

the stochastic logistic model 341

Chapter VII. Numerical Synthesis Methods 3557.1. Numerical solution of the problem of optimal damp-

ing of random oscillations 3567.2. Optimal control for the "predator-prey" system (the

general case) 368

Conclusion 383References 387Index 401

INTRODUCTION

The main problem of the control theory can be formulated as follows.In the design of control systems it is assumed that each control sys-

tem (see Pig. 1) consists of the following two principal parts (blocks orsubsystems): the subsystem P to be controlled (the plant) and the con-trolling subsystem C (the controller). The plant P is a dynamical system(mechanical, electrical, biological, etc.) whose behavior is described by awell-known operator mapping the input (controlling) actions u(t) into theoutput trajectories x(t}. This operator can be denned by a system of ordi-nary differential, functional, functional-differential, or integral equations orby partial differential equations. It is important that the operator (or, intechnical terms, the structure or the construction) of the plant P is assumedto be given and fixed from the outset.

x(t)

FIG. 1

As for the controller C, no preliminary restrictions are imposed on itsstructure. This block must be constructed in such a way that the outputtrajectories { x ( t ) : 0 < t < T] (the case T = +00 is not excluded) possess,in a sense, sufficiently "good" properties.

Whether the trajectories are "good" or not depends on the specificationsimposed on the control system in question. These assumptions are oftenstated by using the concept of a support (or standard) trajectory x ( t ) , andthe control system itself is constructed so that the deviation x(t) — x(t)\on the time interval 0 < t < T does not exceed a value given in advance.

If the "quality" of an individual trajectory {x(t): 0 < t < T} can be es-timated by the value of some functional /[»(<)] of this trajectory, then thereis a possibility to find an optimal trajectory x* (t) on which the functional

CU(t)

P

2 Introduction

I [ x ( t ) } attains its extremum value (in this case, the extremum type (mini-mum or maximum) is determined by the character of the control problem).The functional /[cc(t)] used for estimating the control quality is often calledthe optimality criterion or the performance index of the control systemdesigned.

If there are no random actions on the system, the problem of finding theoptimal trajectory K*(i) amounts to finding the optimal control program{u*(t): 0 < t < T} that ensures the plant motion along the extremum tra-jectory { x f ( t } : 0 < t < T}. The optimal control w*(t) can be calculatedby using methods of classical calculus of variations [64], or, in more generalsituations, Pontryagin's maximum principle [156], or various approximatemethods [138] based on these two fundamental approaches. Different meth-ods for calculating the optimal control programs are discussed in [137].

If an optimal control system is constructed without considering stochas-tic effects, then the system can be open (as in Fig. 1), since the plant tra-jectory { x ( t ) : 0 < t < T} and hence the value of the optimality criterion/[x(t)] are determined uniquely for a chosen realization {u(t): 0 < t < T} ofcontrol actions. (Needless to say that the equation of the plant is assumedto have a unique solution for a given initial state x(Q) = XQ and a giveninput function u(t].}

y(t} Cu(t)

Px(

FIG. 2

The situation is different if the system is subject to noncontrolled ran-dom actions. In this case, to obtain an effective control, one needs someinformation about the actual current state x(t) of the plant, that is, theoptimal system must be a closed-loop (or feedback) system. For example,all servomechanisms are designed according to this principle (see Fig. 2).In this case, in addition to the operator of the plant P, it is necessary totake into account the properties of a source of information, which deter-mines the required value y(t) of the output parameter vector x(t) at eachinstant t (examples of specific servomechanisms can be found in [2, 20,38, 50]). The block C measures the current values of the input y(t) andoutput x(t) variables and forms controlling actions in the form of the func-tional u(t) — y>(i/o, XQ) of the observed trajectories y*0 = {y(s): 0 < s < t},xf

0 = { x ( s ) : 0 < s < t} so that the equality x ( t ) = y(t) holds, if possible,

Introduction 3

for 0 < t < T. However, the stochastic nature of the assigning action (com-mand signal) y(t) on one side, and the inertial properties of the plant P onthe other side do not allow to ensure the required identity between the in-put and output parameters. Therefore, a problem of optimal control arisesin a natural way.

Hence, just as in the deterministic case, the optimality criterion /[|y(t) —x(t)\] is introduced, which is a measure of the "distance" between the func-tions y ( t ) and x(t) on the time interval 0 < t < T. The final statementof the problem depends on the type of assumptions on the properties ofthe assigning action y(t). Throughout this book, we use the probabilitydescription for all random actions on the system. This means that all as-signing actions are treated as random functions with known (completelyor partially) probability characteristics. In this approach, the optimal con-trol law that determines the structure of the block C can be found fromthe condition that the mean value of the criterion j[|j/(<) — £(£)|] attainsits minimum. Another approach in which the regions of admissible valuesof perturbations rather than their probability characteristics are specifiedand the optimal system is constructed by methods of the game theory isdescribed in [23, 114, 115, 145, 195].

If the servomechanism shown in Fig. 2 is significantly affected by noisesarising due to measurement errors, instability of voltage sources in electri-cal circuits, varying properties of the medium surrounding the automaticsystem, then the block diagram in Fig. 2 becomes more complicated andcan be of the form shown in Fig. 3.

FIG. 3

Here £(<) and rj(t) denote random perturbations distorting informationon the command signal y ( t ) and the state x ( t ) of the plant to be controlled;the random function £(t) describes the perturbing actions on the plant P.By '!' and '2' we denote the blocks in which useful signals and noises arecombined. It is usually assumed that the structure of such blocks is known.

4 Introduction

In this book we do not consider control systems whose block diagrams aremore complicated than that shown in Fig. 3. All control systems studiedin the sequel are special cases of the system shown in Pig. 3.

The main emphasis of this book is on the methods for calculating theoptimal control algorithms

(*) «»(<) = <f>*(t,XQ,yo),

which determine the structure of the controller C and guarantee the optimalbehavior of the feedback control system shown in Fig. 3. Since the methodsstudied in this book are oriented to solving applied control problems inmechanics, engineering, and biology, much attention is paid to obtaining(*) in a form such that it can easily be used in practice. This means that alloptimal control algorithms described in the book for specific problems aresuch that the functional (mapping) ip* in (*) has either a finite analytic formor can be implemented by sufficiently simple standard modeling methods.

From the mathematical viewpoint, all problems of optimal control arerelated to finding a conditional extremum of a functional (the optimal-ity criterion), i.e., are problems of calculus of variations [28, 58, 64, 137].However, a distinguishing feature of many optimal control problems is thatthey are "nonclassical" due restrictions imposed on the admissible valuesof controlling actions u(i). For instance, this often leads to discontinuousextremals inadmissible in the classical theory [64]. Therefore, problems ofoptimal control are usually solved by contemporary mathematical methods,the most important being the Pontryagin maximum principle [156] and theBellman dynamic programming approach [14]. These methods develop andgeneralize two different approaches to variational problems in the classicaltheory: the Euler method and the Weierstrass variational principle used forconstructing a separate extremal and the Hamilton-Jacobi method basedon the consideration of the entire field of extremals, which leads to partialdifferential equations for controlled systems with lumped parameters or toequations with functional derivatives for controlled systems with distributedparameters.

The maximum principle, which is a rigorously justified mathematicalmethod, can be used in general for solving both deterministic and stochasticproblems of optimal control [58, 116, 156]. However this method, based onthe consideration of individual trajectories of the control process, leadsto certain technical difficulties when one needs to find the structure ofthe controller C in feedback stochastic systems (see Figs. 2 and 3). Inthis situation, the dynamic programming approach looks more attractive.This method however suffers some flaws from the accuracy viewpoint (forexample, it is well known that the Bellman differential equations cannot be

Introduction 5

used in some cases of deterministic time-optimal control problems [50, 137,156]).

In systems with lumped parameters where the behavior of the plant Pis governed by ordinary differential equations, the dynamic programmingapproach allows the reduction of optimal control problem to solving a non-linear partial differential equation (the Bellman equation). In this case, thestructure of the controller C (and hence the form of the function (map-ping) iff in (*)) is determined simultaneously with solving this equation.Thus this method provides a straightforward solution of the main problemin control theory, namely, the synthesis of a closed-loop automatic controlsystem. As for the possibility to use this method, so far it has been rig-orously proved that the Bellman differential equations are valid and formthe basis for solving the synthesis problems for a wide class of stochasticand deterministic control systems [113, 175]. Therefore, the dynamic pro-gramming approach is widely used in this book and underlies practicallyall methods developed for calculating optimal (or quasioptimal) controls.As noted above, these methods constitute the dominant bulk of the subjectmatter of this book.

As is known, the functional and differential Bellman equations can beused effectively only if the controlled process (or, in more general cases,the system phase trajectory in some state space) is a process without af-tereffects, that is, a Markov type process. In deterministic problems, thisMarkov property of trajectories readily follows from the corresponding exis-tence and uniqueness theorems for the solutions of the Cauchy problem. Toensure the Markov property of trajectories in stochastic control problems,it is necessary to impose some restrictions on the class of random functionsused as mathematical models of random disturbances on the system. Tothis end, throughout this book, it is assumed that all random actions onthe system are either "white noise" type processes or Markov stochasticprocesses.

When the perturbations are of white noise type, the controlled processx(t) itself can be Markov. If the noises are of Markov type, then the processx(t) is, generally speaking, a component of a partially observable Markovprocess of larger dimension. Therefore, to solve the synthesis problem ef-fectively in this case, one needs to use a special state space formed bysufficient statistics, so that the time evolution of these statistics possessesthe Markov property. In this case, the controller C consists of two parts: ablock that forms sufficient statistics (coordinates) and an actual controllerwhose structure can be found by solving the Bellman equation.

These topics are studied in more detail in Chapter I.

CHAPTER I

SYNTHESIS PROBLEMS FOR CONTROL SYSTEMSAND THE DYNAMIC PROGRAMMING APPROACH

§1.1. Statement of synthesisproblems for optimal control systems

In synthesis problems it is required to find the structure of the controlblock (controller) C in a feedback control system (see Figs. 2 and 3). Fromthe mathematical viewpoint, this problem is solved if we know the form ofthe mapping

u = <p(x,y) (I-1-!)

that determines a single-valued correspondence between the input func-tions1 ? = {x(t) : 0 < t < T} and y = ( y ( t ) : 0 < t < T] and the controlvector-function u = {u(t) : 0 < t < T} (the system is considered on the timeinterval [0,T]). The conditions under which algorithm (1.1.1) can physi-cally be implemented impose some restrictions on the form of the mapping<f> in (1.1.1). Usually, it is assumed that the current values of the controlvector u(i) = (ui(t], . . . , ur(t)j at time t are independent of the future val-ues x(t') and y(t'}^ t1 > t. Therefore, the mapping (1.1.1) can be writtenas follows (see (*) in Introduction):

u(t) = y>(*,x£,y0'), 0 < t < T , (1.1.2)

where XQ = {x(s) : 0 < s < t} and y£ = { y ( s ) : 0 < s < t} denote thefunctions x and y realized at time t.

In simpler situations (say, in the case of the servomechanism shown inFig. 2), the synthesis function ip may depend only on the current values ofthe input processes

(1.1.3)

or even may be of the form

u(t)=<p(t,x(t)) (1.1.4)

lPThe functions x and y are input functions for the controller C.

8 Chapter I

if the command signal y(t) is either absent or a known deterministic functionof time.

The explicit form of the synthesis function ip is determined by the char-acter of the optimal control problem.

To state the synthesis problem for an optimal control system mathemat-ically, we need to know:

(1) the dynamic equations of the controlled plant;(2) the goal of control;(3) the restrictions (if any) on the domain of admissible values of control

actions w, on the domain of the phase variables x, etc.;(4) the probability characteristics of the stochastic processes that affect

the system.

Obviously, in problems of deterministic optimal control we need only thefirst three objects.

1.1.1. Dynamic equations of the controlled plant. The presentmonograph, except for §3.6, deals with control systems in which the plant Pcan be described by a system of ordinary differential equations in the normalform

x = g(t,x,u), (1.1.5)

where x = x(t) G Rn and u = u(t) 6 Rr are the current values of ann-dimensional vector of output parameters (the phase variables) and ofan r-dimensional control vector, g(t, £, u): R x Rn x Rr i—>• Rn is a givenvector-function, and the dot over a letter denotes the derivative with respectto time (that is, x is an n-vector with components dxi/dt, i — l , . . . ,n).Here and in the sequel, R& denotes the Euclidean space of fc-dimensionalvectors.

If, in addition to the control u, the controlled plant experiences uncon-trolled random perturbations (see Fig. 3), then its behavior is described bythe equation

x=g(t,x,u,t(t}), (1.1.6)

where £(t) is an m-vector of random functions ( f , i ( t ) , . . . ,{,m(t)). Differ-ential equations of the form (1.1.6) with random functions on the right-hand sides are called stochastic differential equations. In contrast with the"usual" differential equations of the form (1.1.5), they have some specialproperties, which we consider in detail in the next section.

The form of the vector-functions g(t, x, u) and g(t, x, u, £(t)) on the rightin (1.1.5) and (1.1.6) is determined by the physical nature of the plant. Inthe subsequent chapters, we consider various special cases of Eqs. (1.1.5)

Synthesis Problems for Control Systems 9

and (1.1.6) and solve some specific control problems for mechanical, techni-cal, and biological objects. In the present chapter, we only discuss generalrestrictions that we need to impose on the function </(•) in (1.1.5) and (1.1.6)to obtain a well-posed mathematical statement of the problem of optimalcontrol synthesis.

The most important and, in fact, the only restriction on the function g(-)is the existence of a unique solution to the Cauchy problem for Eqs. (1.1.5)and (1.1.6) with any given control function u(t] chosen from a functionclass that is called the class of admissible controls. This means that thetrajectory x ( t ) of system (1.1.5) or (1.1.6) is uniquely determined2 on thetime interval t0 < t < to + T by the initial state x(to) = *o and a chosenfunction {u(t}: t0 < t < t0 + T}.

The uniqueness of the solution x(t) of system (1.1.5) with the initialcondition x(to) = XQ is guaranteed by well-known existence and uniquenesstheorems for systems of ordinary differential equations [137]. The followingtheorem [156] presents very general sufficient conditions for the existenceand uniqueness of the solution of system (1.1.5) with the initial conditionx(to) = XQ (the Cauchy problem).

THEOREM. Let a vector-function g(t,x,u) be continuous with respect toall variables (t, x, u) and continuously differentiate with respect to the com-ponents of the vector x = (xi,..., xn), and let the vector-function u — u(t]be continuous with respect to time. Then there exists a number T > 0 suchthat a unique continuous vector-function x(t) satisfies system (1.1.5) withthe initial condition x(to) = XQ on the interval to < t < to + T.

If T —>• oo, that is, if the domain of existence of the unique solutionis arbitrary large, then the solution of the Cauchy problem is said to beinfinitely continuable to the right.

It should be noted that the functions g(-) and u need not be continuouswith respect to t. The theorem remains valid for piecewise continuous andeven for bounded functions <?(•) and u that are measurable with respectto t. In the last case, the solution x(t): to < t0 + T of system (1.1.5) is anabsolutely continuous function [91].

The assumption that the function g(-) is smooth with respect to thecomponents of the vector x is much more essential. If this condition is notsatisfied, then we can encounter situations in which system (1.1.5) does nothave any solutions in the "common" classical sense (for example, for someinitial vectors x(to) = XQ, it may be impossible to construct a function

The solution of the stochastic differential equation (1.1.6) is a stochastic processx(t). The uniqueness of the solution to (1.1.6) is understood in the sense that the initialcondition x(to) = #o and the control function u(t): t0 < t < t + T uniquely determinethe probability characteristics of the random variables x(t) for all t £ (to, to + T].

10 Chapter I

x(t) that identically satisfies (1.1.5) on an arbitrarily small finite intervalt0 <t<t0 + T).

It is significant that we cannot exclude such seemingly "exotic" casesfrom our consideration. As was already noted, the control function u onthe right-hand side of (1.1.5) can be defined either as a controlling program(that is, as a function of time) or in the synthesis form, for example, in theform u = <f>(t, x(t)) like in (1.1.4). It is well known (this will be illustratedby numerous special examples considered later) that many problems ofoptimal control with control constraints often result in control algorithmsu* — <p(t,x(t)) in which the synthesis function (p is discontinuous withrespect to the phase variables x. In this case the assumptions of the above-cited theorem may be violated even if the vector-function g(t, x, u) in (1.1.5)is continuously differentiable with respect to x.

Now let us generalize the notion of the solution to the case of discontin-uous (with respect to x) right-hand sides of Eqs. (1.1.5). Here we discussonly the basic ideas for constructing generalized solutions. The detailedand rigorous theory of generalized solutions of equations with discontinu-ous right-hand sides can be found in Filippov's monograph [54].

We assume that in (1.1.5) the control function u has the synthesis form(1.1.4). Then, by setting <?(t, x) = g(t, x, <p(t, z)), we can rewrite (1.1.5) asfollows:

x = g(t,x). (1.1.7)

In the space of variables (i, x), we choose a domain D on which we need toconstruct the solution of system (1.1.7). Suppose that a twice continuouslydifferentiable surface S divides the domain D into two domains Z3_j_ and D_and some vector-functions gL|_ and g_ continuous in t and continuouslydifferentiable in x-i, X2, • • • , xn are defined on D+ + S and on D_ + S sothat g = <7+ in D+ and ~g — §_ in D-. In this case, the solution of (1.1.7) onthe domain D_ can uniquely be continued till the surface S. If the vector gis directed towards the surface S in £)_ and away from the surface S in D+,then the solution goes from £>_ to D+, intersecting the surface S only once(Fig. 4). But if the vector ~g is directed towards the surface S in £>_ andin D+, then the solution, once coming to S, can leave it neither to Z>_nor to D+. Therefore, there is a problem of continuation of this solution.In [54] it is assumed that after the solution x(t) comes to the surface S, thesubsequent motion of system (1.1.7) is realized along the surface S withvelocity

x = go(t,x) = ag+(t,x) + (1 - a)g_(t, x), (1.1.8)

where x (E S and the number a (0 < a < 1) are chosen so that the vector<7o(£, x) is tangent to the surface S at the point x.

The vector ^(t, xj in (1.1.8) can be constructed in the following way.

Synthesis Problems for Control Systems 11

x(t)D

FIG. 4

At the point x G 5 we construct the vectors cj+(t, x) and g-(t,x) andconnect their endpoints with a straight line. The point of intersection ofthis straight line with the plane tangent to 5 at the point x is the endpointof the desired vector g0(t,x) (Fig. 5).

FIG. 5

A function x(t) satisfying Eq. (1-1.7) in D+ and in D_ and satisfyingEq. (1.1.8) on the surface 5 is called the generalized solution of Eq. (1.1.7)or a solution in the sense of Filippov.

This definition makes sense, since a solution in the sense of Filippov isthe limit of a sequence of classical solutions to Eq. (1.1.7) with smoothed (in

12 Chapter I

x) right-hand sides gk(t,x) ifgk(t,x) — >• g(t, x) as k — )• oo. Moreover, thesequence xk(t) of classical solutions of equations with retarded argument

uniquely converges to the same limit if the delay Tk — >• 0 as k — )• oo (see [54]).We also note that, in practice, solutions in the sense of Pilippov can

be realized in some technical, mechanical, and other systems of automaticcontrol, which are sometimes called systems with variable structure [46]. Insuch systems, the plant is described by Eq. (1.1.5), and the control vector umakes a jump when the phase vector x(t) intersects a given switching sur-face S. In such systems, if the motion is along the switching surface, thecritical segments of the trajectory can be realized by infinitely fast switch-ing of control. In the theory of automatic control such regimes are called"sliding modes" [2, 46].

Generalized solutions in the sense of Filippov allow us to constructthe unique solution of the Cauchy problem for Eq. (1.1.5) with functiong(t, x,u) piecewise continuous in x.

Now let us consider the stochastic differential equations (1.1.6). Wehave already pointed out that these equations substantially differ from or-dinary differential equations of the form (1.1.5); the special properties ofEqs. (1.1.6) are studied in §1.2. Here we only briefly dwell on the nature ofspecial properties of these equations.

The stochastic differential equations (1.1.6) have the following funda-mental characteristic property. If the random function £ ( t ) on the right-hand side of (1.1.6) is a stochastic process of the "white noise" type, thenthe Cauchy problem for (1.1.6) can have an infinite (larger than a count-able) set of different solutions. Everything depends on how we understandthe solution of (1.1.6) or, in other words, on how we construct the randomfunction x ( t ) that satisfies the corresponding Cauchy problem for (1.1.6).It turns out that in this case we can propose infinitely many well-definedsolutions of equation (1.1.6).

This situation gives an impression that the differential equations (1.1.6)do not make any sense. However, since control systems perturbed by a whitenoise play an important role, it is necessary to specify how the dynamicsof a system is described in this case and in which sense Eq. (1.1.6) must beunderstood if it is still used.

On the other hand, the existence and uniqueness of the solution to theCauchy problem for equations of the forms (1.1.5) and (1.1.6) is the basicassumption that allows us to use the dynamic programming approach forsolving problems of optimal control synthesis.

In §1.2 we discuss these and some other topics.

Synthesis Problems for Control Systems 13

1.1.2. Goal of control. The requirements imposed on a designedcontrol system determine the form of the functional (the optimality crite-rion) , which is a numerical estimate of the control process. Let us considersome typical problems of optimal control and write out the cost functionalsneeded to state these problems.

We begin with deterministic problems in which the plant is described bythe system of differential equations (1.1.5). First, we assume that the timeinterval 0 < t < T (on which we consider the control process) is fixed andthe initial position of the plant is given, that is, x(0) = Zo, where x0 is avector of some given numbers. Such problems are called control problemswith variable right endpoint of the trajectory. Suppose that it is requiredto construct an optimal servomechanism (see Fig. 2) such that the inputcommand signal y(t): 0 < t < T is a known function of time. If the goalof the servomechanism shown in Fig. 2 is to reproduce the input functiony(t) via the output function x(t): 0 < t < T most closely, then one ofpossible criteria for estimating the performance of this servomechanism isthe integral

\x(t)- y ( t } \ p d t , (1.1.9)/Jowhere p is a given positive number, and a denotes the Euclidean norm of

__ 1 /">

a vector o, that is, a = (S™=i a?) • ^n an "ideal" servomechanism, thecontrolled output process is identically equal to the command signal, thatis, x(t) = y(t), 0 < t < T, and the functional (1.1.9) is equal to zero, whichis the least possible value. In other cases, the value of (1.1.9) is a numericalestimate of the proximity between the input and output processes.

It may happen that much "effort" is required to ensure a sufficient prox-imity between the processes x(t] and y(t), that is, the control action u(t)needs to be large at the input of the plant P. However, it is undesirable touse too "large" controls in many actual devices both from the energy andeconomy viewpoints, as well as from the reliability considerations. In thesecases, instead of (1.1.9), it is better to use, for example, the cost functional

rT[ \ x ( t ) - y(t)\'+ a\U(t)\«] dt, (1.1.10)

where a, q > 0 are some given numbers. This functional takes into accountboth the proximity between the output process x(t) and a given inputprocess y(t) and the total "cost" of control on the time interval [0,T].

Of course, the functionals (1.1.9) and (1.1.10) do not exhaust all meth-ods for stating integral optimality criteria that are used in problems ofsynthesis of optimal servomechanisrns (Fig. 2). The most general form