Download - UCLA Chemical Engineering
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Constrained In�nite�Time Nonlinear Quadratic
Optimal Control
V� Manousiouthakis D� Chmielewski
Chemical Engineering Department
UCLA
���� AIChE Annual Meeting
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Outline
Unconstrained In�nite�Time Nonlinear Quadratic�Optimal Control
� Hamilton�Jacobi�Bellman Equation
� State Dependent Riccati Equation and the Curl Condition
� Closed�Loop Stability
� Example
Constrained In�nite�Time Nonlinear Quadratic�Optimal Control
� Family of Finite Time Problems
� Equivalence to the In�nite Time Problem
� Example
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
In�nite�time Nonlinear Quadratic Optimal Control
Problem
minu
Z�
�
�xTQ�xx uTR�xu
�dt
s�t� �x � f�x B�xu
The solution can be found by solving the HJB equation
�J�x
Tf�x xTQ�xx�
�
�J�x
TB�xR���xBT �x�J
�x� �
The optimal policy is given by u��x � ���R���xBT �x�
�J�x
�T
The determination of J�x is nearly impossible in all but the simplest cases�
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
In�nite�time Nonlinear Quadratic Optimal Control
J�x can be written as� J�x � xTP �xx� where P �x is symmetric�
In this case�
�J�x
� ��
B�P �x �
�xT�P
�x�
CAx
De�ne
��x ��P �x �
�xT�P
�x
Then the HJB becomes�
xT�
��xA�x AT �x��x Q�x� ��xB�xR���xBT �x��x�
x � �
where A�x s�t� f�x � A�xx�
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Assumptions
�H� R�x � �� � x � �n� and Q�x � �� � x � D � �n
�H� A�x and B�x are analytic matrix valued functions � x � D
�H� The pair �A�x� B�x is controllable �in the linear system sense � x � D
�H The gradients ��A�x�x�
�x
and �bi�x�
�x
exist and are continuous and bounded
� x � D �where bi�x is the ith column of B�x
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
State Dependent Riccati Equation
If ��x satis�es the SDRE�
��xA�x AT��x� ��xB�xR���xBT �x��x Q�x � �
then the HJB will appear to be satis�ed� In this case� the optimal policy is
u��x � �R���xBT �x��xx
and the optimal value function isJ�x � xTP �xx
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Gradient of a Scalar Function
For the HJB to actually be satis�ed
�xT��x must equal�J
�x
The conditions for a vector function� v�x� to be a gradient are �
�vi�xj
��vj
�xi� � i� j
Assumption� �H� A�x is s�t� the solution to the SDRE� ��x� satis�es�
curl�xT��x � � � � x � D
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Existence of a ��x�
The following guarantees the existence of a SDRE such that the curl
condition is satis�ed �Lu and Huang� ���
If there exists a solution to the HJB equation�
J�x�� then there exists A�x� s�t� the solution
to the SDRE equation� ��x�� satis�es
�J�x
� xT��x��
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Stability and Optimality
Theorem��Manousiouthakis and Chmielewski� ����
Let �H���H� hold� Then the scalar function
J�x � xTZ �
� ����xd�
x
satis�es the necessary conditions for optimality �HJB for all x � D�
Furthermore� the feedback policy
u�x � �R���xBT �x��xx
is asymptotically stabilizing�
Corollary� Let �H���H� hold� and assume D � �n�
Then the feedback policy is globally asymptotically stabilizing�
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
An Inverse Method
Consider ��x and A�x to be design functions�
First we must guarantee
��x � � � curl�xT��x � � � A�xx � f�x
Then simply calculate Q�x from the SDRE�
Q�x � ��xB�xR���xBT �x��x� AT �x��x� ��xA�x
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
An Inverse Method �continued�
Let Z� be de�ned as �a positively invariant set
Z� ��
x � �njJ�x � � � J�x � xT�Z �
� ����xd�
x�
De�ne �� s�t�
xTQ�xx uTR�xu � xT�
Q�x ��xB�xR���xBT �x��x�
x
� � � x � Z��
Then a su�cient condition closed�loop stability is that x�� � Z���
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example
Consider a CSTR Reactor with reaction �Aka
��kb
B�
���������x�
�x��
������� ��
���������kax�
� ��
kaCAss FV
�x� �kbx�
kax�
� �kaCAssx� ��
FV
kb�
x�
��������
��������F
V�
�������� u
x� � CA � CAss � x� � CB � CBss � u � CAin � CAinss
CBss �
ka
FV
kbC�Ass �
��
�CAinss � CAss
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example �continued�
The following SS design parameters are used �all in SI units
ka � ��� kb � ��� FV
� ���
CAinss � �� CAss � ��� CBss � ��
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example �continued�
Let ��x � Po where Po satis�es
ATo Po PoAT
o Qo � PoBR��BTPo � �
where R � �VF ��
Ao ��
��� ��
kaCAss FV
�
�kb
�kaCAss ��
FV
kb�
���� and Qo �
���� qo
�� �
� qo���
���
Clearly this choice of ��x satis�es the �Curl� condition�
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example �continued�
Q�x is calculated� via the SDRE� to be�
Q�x ��
�������qo�� �kax���p�� � p�� kax��p�� � �p��
kax��p�� � �p�� qo��
��������
Then Q�x PoBR��BTPo is positive de�nite i�
qo�� �kax���p�� � p�� p��� � �
�qo�� �kax���p�� � p�� p���
�qo�� � �kax��p�� � �p��p��p��� � �
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example �continued�
For qo�� � �� and qo�� � ��
x� � � ��� � ��� � Q�x PoBR��BTPo � �
The largest positively invariant set
Z� ��
� � ��j�TPo� � ��
such that Q�x PoBR��BTPo � � � x � Z�� corresponds to
� � ����
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
10
5
0
-5
Conce
ntr
atio
n o
f B
1050-5-10
Concentration of A
Q(x
) +
PoB
R-1
BTP
o >
0
Q(x) + PoB R-1BTP
o >0
Example 1: Closed-Loop Stability Region
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
5
4
3
2
1
0
Conce
ntr
atio
n
250200150100500
Time
CA
CBCB
Nominal Inputs Optimal Control
Example 1: Closed-Loop Simulations
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Constrained ITNQOC Problem
��� � infu
�Z�
�
�xTQx �T �uR��u
�dt
�
s�t� �x � A�xx B�x��u� x�� � �
where ��� � �M U � is de�ned as
��u � arg min��Ujju� jj
Further Assumptions�
�H� U is convex and contains the origin in its interior
�H� � � Xo �� f� � �nj u s�t� ��� ��g
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Constrained FTNQOC Problem
Consider the following family of �nite�time optimal control problems�
�T �� � infu
�Z T�
�xTQx �T �uR��u
�dt J�x�T
�
s�t� �x � A�xx B�x��u� x�� � �
where
J�x � xT�Z �
� ����xd�
x
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Equivalence of Constrained and Unconstrained
Problems
De�ne
�X � f� � �nj K��� � Ug
O� ���
� � �nj x�t � �X � t � ��
where K�x is the unconstrained optimal feedback gain�
Lemma� Let K�x be bounded for all x in a neighborhood of the origin� Then
� � intfUg �� � � intfO�g
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Finite�Time Solution to In�nite�Time Problem
Lemma� Let �H��H� hold and assume the solution to �T is unique� Then
�i �� �� limT���T exists for all � � Xo
�ii �T � �� � �T � �T�� � � � � � u�T � u�T�� � � � � � x�T �T � O�
�iii If T � � such that x�T �T � O� then �T � �
Theorem� Let �H��H� hold and assume the solution to �T is unique� Then
� � � Xo T s�t� x�T �T � O�
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Equivalence of Initial Condition Sets
De�ne the set of constrained stabilizable initial conditions as
Xmax ���
�� � �nj u s�t� limt��
jjx�tjj � ����
�
Theorem� Let �H��H� hold and assume the solution to �T is unique� Then
Xmax � Xo
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example �
Consider the CSTR of Example � with kb � �
�x� � ��kax�
� ��
B� kaCAss F
V�
CAx� F
Vu
with objective function �r � �FV �
Z�
�
�x��t� ru�t��
dt
and constraints
u�t CAinss � ��� ��� � t � � Xmax � �������
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example � �continued�
The resulting SDRE is�
� ���kax� � �kaCAss ��x� �� ���x� � �
Which has solution
��x� � �
kax� kaCAss r
�kax� kaCAss� �
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
Example � �continued�
The Unconstrained Optimal Control Policy is
u�x � �V
F
kax� kaCAss r
�kax� kaCAss� �
x�
and
O� � ���� ��
Finally� the Closed�Loop System is
�x � �x�r
�kax� kaCAss� �
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
1.0
0.8
0.6
0.4
0.2
0.0
Conce
ntr
atio
n o
f A
2520151050
Time
No Constraints Constrained Input Nominal Input
Example 2: Optimal State Trajectories
Process & Control Systems Engineering Laboratory
UCLA Chemical Engineering
60
40
20
0
CA
in
2520151050
Time
No Constraints Constrained Input
Example 2: Optimal Input Policies