Contributions to the Theoryof Optimal Control

R. E. KALMAN

THIS is one of the two ground-breaking papers by Kalmanthat appeared in 1960—with the other one (discussed next) be-ing the filtering and prediction paper. This first paper, whichdeals with linear-quadratic feedback control, set the stage forwhat came to be known as LQR (Linear-Quadratic-Regulator)control, while the combination of the two papers formed thebasis for LQG (Linear-Quadratic-Gaussian) control. Both LQRand LQG control had major influence on researchers, teachers,and practitioners of control in the decades that followed.

The idea of designing a feedback controller such that the in-tegral of the square of tracking error is minimized was first pro-posed by Wiener [17] and Hall [8], and further developed in theinfluential book by Newton, Gould and Kaiser [12]. However,the problem formulation in this book remained unsatisfactoryfrom a mathematical point of view, but, more importantly, thealgorithms obtained allowed application only to rather low ordersystems and were thus of limited value. This is not surprisingsince it basically took until theH2-interpretation in the 1980s ofLQG control before a satisfactory formulation of least squaresfeedback control design was obtained. Kalman’s formulation interms of finding the least squares control that evolves from anarbitrary initial state is a precise formulation of the optimal leastsquares transient control problem.

The paper introduced the very important notion ofcontrolla-bility, as the possibility of transfering any initial state to zero bya suitable control action. It includes the necessary and sufficientcondition for controllability in terms of the positive definitenessof the Controllability Grammian, and the fact that the lineartime-invariant system withn states,

d

dtx = Fx + Gu

is controllable if and only if the matrix [G, FG, . . . , Fn−1G]has rankn. As is well known, this concept of controllability, itsimplications (e.g., in pole placement and stabilization), and gen-eralizations to nonlinear or infinite-dimensional systems becameone of the mainleitmotivsin control research. Controllability isindeed one of the compelling notions that is truly endogenousto the field of control.

The paper also introduced the notion ofobservability, but as amere “dual” of controllability. Contemporaneously Kalman pro-vided an alternative, more satisfactory definition in [10], whereobservability is defined in a more intrinsic way in terms of thepossibility of deducing the state trajectory from input/outputmeasurements.

Kalman actually states (p. 102, fourth paragraph) that he viewsthe introduction of the notions of controllability and observ-ability, and their exploitation in the regulator problem, as theprincipal contribution of the present paper. Controllability andobservability are shown in the paper to be of central impor-tance in the analysis of the least squares control problem overan infinite horizon. They are also used to obtain the asymptoticproperties of the Riccati differential equation [Equation (6.3)of the paper—henceforth referred to as RDE], and the stabilityproperties of its limiting solution. This paper was in fact the firstto introduce the RDE as an algorithm for computing the statefeedback gain of the optimal controller for a general linear sys-tem with a quadratic performance criterion. RDE had emergedearlier in the study of the second variations in the calculus ofvariations, but its use in general linear systems, where the opti-mal trajectory needs to be generated by a control input, was new.

The analysis throughout the paper concentrates on time-varying systems, and uses the Hamilton-Jacobi theory to arriveat RDE and to deduce optimality of the LQ control gain. Wenow know, however, that an alternative way to prove optimalityin least squares is by showing how RDE allows one to “completethe square” (see, e.g., [5], [18]).

Almost immediately after its appearance, the LQ-problemwas included in influential textbooks [2], [5], [11], [1], andextended in a number of directions. For example, the case ofindefinite cost is treated in [18], which requires a more del-icate analysis, and later had many applications, in particular,in H∞-control; and extensions to zero-sum and nonzero-sumdifferential games are discussed in [9] and [16], where againRDE-based feedback policies arise, albeit with more generalstructures [3]. Kalman’s paper actually also motivated and ledthe way to a great deal of research on RDE, particularly on itsalgebraic version, and algorithms for solving it appeared very

147

soon [13]. Most computer packages that aim at linear systemsand control implement today a Riccati equation solver. A wealthof information and references on the work on the LQ problemcan be found in [15] and [4].

Kalman’s paper deals only with state feedback. The formula-tion of the output feedback version of the LQ problem requireseither introducing, as in LQG, stochastic disturbances, or a for-mulation in terms of theH2-norm. It is worth noting, however,that Kalman’s paper contains (p. 104) an informal, unproven butquite precise statement of the separation theorem and the cer-tainty equivalence principle, which are further discussed in thisvolume in the preamble to Feldbaum’s papers.

The LQ problem and its sister, LQG, received enormous atten-tion, and were sometimes presented as a panacea for solving lin-ear control problems. Critics [14], [16] pointed to the excessivebandwidth and poor robustness properties of the LQG controller.In fact, Doyle’s paper [6] about guaranteed margin of LQG-controllers had the abstract “There are none”—fortuitously mak-ing reading of the article superfluous. This lack of robustness ledto theH∞ control problem which was formulated by Zames [19]in part as a reaction to the LQG problems (see the comments inthe preamble to [19], the last paper in this volume). However,it turns out that also in theH∞-problem it was the Riccati equa-tion that finally prevailed [7]! The introduction of the Riccatiequation, along with the notions of controllability and observ-ability, are among the many gems in the crown of original ideasin this seminal paper by Kalman.

REFERENCES

[1] B.D.O. ANDERSON AND J.B. MOORE, Linear Optimal Control, PrenticeHall (Englewood Cliffs, NJ), 1971.

[2] M. A THANS AND P.L. FALB, Optimal Control, McGraw-Hill (New York),1966.

[3] T. BASAR AND G.J. OLSDER,Dynamic Noncooperative Game Theory, Clas-sics in Applied Mathematics, SIAM (Philadelphia), 1999.

[4] S. BITTANTI , A.J. LAUB, AND J.C. WILLEMS, edts,The Riccati Equation,Springer Verlag (Berlin), 1991.

[5] R.W. BROCKETT, Finite Dimensional Linear Systems, Wiley (New York),1970.

[6] J.C. DOYLE, “Guaranteed margins for LQG regulators,”IEEE Trans. Au-tomat. Contr., AC-23:756–757, 1978.

[7] J.C. DOYLE, K. GLOVER, P.P. KHARGONEKAR AND B.A. FRANCIS, “State-space solutions to standardH2 and H∞ control problems,”IEEE Trans.Automat. Contr., AC-34:831–847, 1989.

[8] A.C. HALL , The Analysis and Synthesis of Linear Servomechanisms, TheTechnology Press, M.I.T. (Cambridge, MA), 1943.

[9] Y.C. HO, A.E. BRYSON, JR.AND S. BARON, “Differential games and optimalpursuit-evasion strategies,”IEEE Trans. Automat. Contr., AC-10(4):385–389, 1965.

[10] R.E. KALMAN , “On the general theory of control systems,” inProc.First Internat. Congress Automat. Contr., pp. 481–491, Moscow,1960.

[11] H. KWAKERNAAK AND R. SIVAN , Linear Optimal Control Systems, Wiley(New York), 1972.

[12] G.C. NEWTON, JR., L.A. GOULD AND J.F. KAISER, Analytical Design ofLinear Feedback Controls, Wiley (New York), 1957.

[13] J.E. POTTER, “Matrix quadratic solutions,”SIAM J. Appl. Math., 14:496–501, 1964.

[14] H.H. ROSENBROCK ANDP.D. MCMORRAN, “Good, bad, or optimal?,”IEEETrans. Automat. Contr., AC-16(6):552–554, 1971.

[15] Special Issue on the Linear-Quadratic-Gaussian Estimation and ControlProblem, IEEE Trans. Automat. Contr., AC-16(6), 1971.

[16] A.W. STARR AND Y.C. HO, “Nonzero-sum differential games,”J. Optimiz.Theory Appl., 3(3):184–206, 1969.

[17] N. WIENER, Extrapolation, Interpolation, and Smoothing of StationaryTime Series, MIT Press (Cambridge, MA), 1949.

[18] J.C. WILLEMS, “Least squares stationary optimal control and the alge-braic Riccati equation,”IEEE Trans. Automat. Contr., AC-16:621–634,1971.

[19] G. ZAMES, “Feedback and optimal sensitivity: Model reference transforma-tions, multiplicative seminorms, and approximate inverses,”IEEE Trans.Automat. Contr., AC-26:301–320, 1981.

W.S.L. & J.C.W.

148

149

Reprinted with permission from Boletin de la Sociedad Matematica Mexicana, R. E. Kalman,“Contributions to the Theory of Optimal Control,” Vol. 5, 1960, pp. 102–119.