mathematical fundamentalsmocha-java.uccs.edu/ece5590/ece5590_ch2_f15.pdf · 2019. 4. 24. · in...

ECE5590: Model Predictive Control 2–1

Mathematical Fundamentals

Vector Spaces

! Simplest example of a vector space is the Euclidian n-dimensionalspace, Rn, e.g.,

u D .u1; u2; :::; un/ with ui 2 R for i D 1; 2; : : : ; n

! Rn exhibits some special properties that lead to a general definition ofa vector space:

– Rn is an additive abeliean group

– Any n-tuple ku is also in Rn

– Given any k 2 R and u 2 Rn we can obtain by scalar multiplicationthe unique vector ku

In abstract algebra, an abelian group, also called acommutative group, is a group in which the result of ap-plying the group operation to two group elements doesnot depend on the order in which they are written (theaxiom of commutativity). Abelian groups generalize thearithmetic of addition of integers. They are named afterNiels Henrik Abel. [Jacobson, Nathan, Basic Algebra I,2nd ed., 2009]

! Some further results:

Lecture notes prepared by M. Scott Trimboli. Copyright c" 2010-2015, M. Scott Trimboli

ECE5590, Mathematical Fundamentals 2–2

– k .u C v/ D ku C kv (i.e., scalar multiplication is distributive underaddition)

– .k C l/ u D ku C lu

– .kl/ u D k .lu/

– 1 # u D u

! To define a vector space in general we need the following fourconditions:

1. A set V which forms an additive abeliean group2. A set F which forms a field3. A scalar multiplication of k 2 F with u 2 V to generate a unique

ku 2 V

4. Four scalar multiplication axioms from above.

Then we say that V is a vector space of F .

Examples:

! All ordered n-tuples, u D .u1; u2; : : : ; un/ with ui 2 C forms a vectorspace Cn over C

! The set of all n $ m real matrices Rn$m is a vector space over R

Linear Spans, Spanning Sets and Bases

! Let S be the set S=fu1; u2; : : : ; umg 2 V where V is a vector space ofsome F

! Define L.S/ Dfv 2 V W v D a1u1 C a2u2 C : : : C amum with ai 2 F g! Thus L.S ) is the set of all linear combinations of the ui

! Then,



– L.S/ is a subspace of V (note, it is closed under vector additionand scalar multiplication)

– L.S/ is the smallest subspace containing S

! We say that L.S/ is the linear span of S , or that L.S/ is spanned by S

! Conversely, S is called a spanning set for L.S/

IMPORTANT: A spanning set v1; v2;: : : ; vm of V is a basis if the vi arelinearly independentNOTE: If dim.V / D m, then any set v1; v2;: : : ; vn 2 V with n > m must belinearly dependent

Linear Transformations, Matrix Representation & Matrix Algebra

Linear Mappings

! Let X; Y be vector spaces over F and let T be a mapping of X into Y ,

T W X ! Y

! Then T 2 L.X; Y /; the set of linear mappings (or transformations) ofX into Y , if

T .u C v/ D T .u/ C T .v/

andT .ku/ D k # T .u/; for all u; v 2 X and k 2 F

) This defines a Linear Tranform

Matrix Respresentation of Linear Mappings

! Linear mappings can be made concrete and represented by matrixmultiplications...



– once x 2 X and y 2 Y are both expressed in terms of theircoordinates relative to appropriate basis sets

! We can now show that A is a matrix representation of the lineartransform T :

– y D T .x/ is equivalent to y D Ax

! Note that when dealing with actual vectors in Rk or Ck, an all-timefavorite basis set is the standard basis set, which consists of vectorsei which are zero everywhere except for a “1” appearing in the i th

position

Change of Basis

! Let e01; e

02;: : : ; e

0m be a new basis for X

! Then each e0i is in X and can be written as a linear combination of the

ei :

e0i D

nXj D1

pij ej

E0 D EP

where P is a square and invertible matrix with elements pji

! Thus we can write,

E D E0P %1

meaning that P %1 transforms the old coordinates into the newcoordinates.

! We define a similarity transformation as

T D P %1TP



Image, Kernal, Rank & Nullity

Isomorphism. A linear transformation is an isomorphism if the mappingis “one-to-one”

Image. The image of a linear transformation T is,Image.T / D fy 2 Y W T .x/ D y; x 2 Xg

Kernel. The kernel of a linear transformation T is,ker.T / D fx 2 X W T .x/ D 0g

! Image.T / is a subspace of Y (the co-domain) and ker.T / is asubspace of X (the domain)

Example 1.1

A W R3 ! R2 with A D"

1 0 1

0 1 1

#

! Rank of a Matrix

! The rank of a matrix is defined as the number of linearly independentcolumns (or equivalently the number of linearly independent rows)contained in the matrix

! Our definition of the rank of T is consistent with this because in thecase of T 2 L.X; Y / defined by x! Ax, the dimension of Image.T / isnothing other than the dimension of the linear span of the columns ofA, which obviously is equal to the number of linearly independentcolumn vectors

! Note that an alternative but equivalent and perhaps more convenientway to define the rank of a square matrix is by way of minors:

– If an n $ n matrix A has a non-zero r $ r minor, while every minorof order higher than r is zero, then A has rank r .



Example 1.2

A D

266664

1 1 0

0 %1 1

1 1 0

3 2 1

377775

! Since the dimension of A is 4 $ 3, it’s rank can be no larger than 3

! But since column 2 plus column 3 equals column 1, the rank reducesto 2

Corrollary: A square n $ n matrix is full rank (non-singular) if itsdeterminant is non-zero.

! Some helpful properties of determinants:

Let A; B 2 Rn$n

1. det.A/ D 0 if and only if the columns (rows) of A are linearlydependent

2. If a column (row) of A is zero, then det.A/ D 0

3. det.A/ D det.AT /

4. det.AB/ D det.A/ # det.B/

5. Swapping two rows (columns) changes the sign of det.A/.

(a) Scaling a row (column) by k, scales the determinant by k.(b) Adding a multiple of one row (column) onto another does not

affect the determinant.

6. If A D diagonalfaiig or lower triangular, or A D upper triangular withdiagonal elements aii ; then det.A/ D a11; a22; : : : ; ann

Define the adjoint of a square matirx A as follows:



! Let !i;j D .%1/iCj det.Mi;/ where Mi;j is the matrix A with the i throwand j thcolumn removed.

! Then,

adj.A/ D !Ti;j

! Then the matrix inverse is defined as:

A%1 D 1

det.A/# adj.A/

Eigenvalue & Singular Value Decompositions

! Often in engineering we have to deal with scalar functions of a matrix

! For example

G.s/ D C.sI % A/%1B

where A is a square real matrix, represents the transfer function matrix ofa dynamic system with many inputs and many outputs.

! Equally,

y D eAtxo

is the form of solution of n 1st order ordinary differential equations.

! We know how to handle terms like1

s % a, whose inverse Laplace

Transform is just eat .

! But how do we handle .sI % A/%1? And how do we evaluate eAt?

– Remember that an n $ n matrix A can be thought of as a matrixrepresentation of a linear tranformation on Rn, relative to somebasis (e.g., the standard basis).



– Yet we know that a change of basis, described in matrix form byE

0 D EP , will transform A to P %1AP .

! Suppose then that it were possible to choose our new basis such thatP %1AP were a diagonal matrix D.

! This transforms general matrix problems into “diagonal” problems –they consist now of a collection of scalar problems, 1=.s%di / and edi t ,which we can easily solve.

! Such a basis exists for almost all A0s and is defined by the set ofeigenvectors of A.

! The corresponding diagonal elements di of D turn out to be theeigenvalues of A.

Eigenvalue Decomposition

! A linear transformation T maps x ! y D Ax where in general x, y

have different directions

! There exist directions w however which are invariant under themapping

Aw D "w where " is a scalar:

! Such A-invariant directions are called eigenvectors of A and thecorresponding "’s are called eigenvalues of A; eigen in Germanmeans “self”.

! The above equation can be re-arranged as:

."I % A/ w D 0

and thus implies that w lies in the kernel of ."I % A/:



! A non-trivial solution for w can be obtained if and only if,ker."I % A/ ¤ f0g, which is only possible if ."I % A/ is singular, i.e.,

det."I % A/ D 0

! This equation is called the characteristic equation of A, and definesall possible values for the eigenvalues "

! The " may assume one of the n roots, "i of the characteristicequation, and are thus called characteristic roots or simply theeigenvalues of A.

! Interesting and important properties of the eigenvalues of a matrix A:

– Trace.A/ & sum of the diagonal elements of A DnX

iD1

"i D sum of

the eigenvalues

– det.A/ & determinant of A DnY

iDi

"i D product of the eigenvalues

! Once the values of " have been determined, the correspondingeigenvectors may be calculated by solving the set of homogeneousequations:

."I % A/

0BBBB@

w1

w2:::

wn

1CCCCA

D 0 where wi denotes the i th element of w

NOTE: If Aw D w , then A .kw/ D kAw D k"w D " .kw/, so that if w isan eigenvector of A, then so also will be any scalar multiple of w – theimportant property of w is its direction, not its scaling or length.

Example:



A D

264

1 %3 5

0 1 2

0 %1 4

375

so that,det."I % A/ D "3 % 6"2 C 11" % 6 D 0

IMPORTANT EIGENVECTOR/EIGENVALUE PROPERTIES:

1. The eigenvalues of a diagonal or triangular matrix are equal to thediagonal elements of the matrix.

2. The eigenvectors of a diagonal matrix (with distinct diagonalelements) are the standard basis vectors.

3. The eigenvalues of the identity matrix are all "1" and the eigenvectorsare arbitrary.

4. The eigenvalues of a scalar matrix kI are all k and the eigenvectorsare arbitrary.

5. Multiplying a matrix by a scalar k has the effect of multiplying all itseigenvalues by k , but leaves the eigenvectors unaltered.

6. (Shift Theorem) Adding onto a matrix a scalar matrix, kI , has theeffect of adding k to all its eigenvalues, but leaves the eigenvectorsunaltered.

7. The eigenvectors of a matrix and its transpose are the same.

8. The eigenvalues/eigenvectors of a real matrix are real or appear incomplex conjugate pairs.



Characteristic Decomposition

! By convention, we collect all the eigenvectors of an n $ n squarematrix A as the column vectors of the eigenvector matrix, W :

W D Œw1; w2; # # # ; wn#

! Upon inversion we can obtain the dual set of left eigenvectors as therow vectors of the dual eigenvector matrix, V D W %1:

V D W %1 D

266664

vt1

vt2:::

vtn

377775

! Combining the eigenvector and dual eigenvector matrices togetherwe get the eigenvalue/eigenvector decomposition (or characteristicdecomposition):

AW D Wƒ

where ƒ is a diagonal matrix containing the eigenvalues, "i , of A.Post-mulitplying the above by W %1 we obtain:

A D WƒW %1 D WƒV

) This is also known as the spectral decomposition of A.

! The eigen-decomposition can be ill-conditioned with respect to itscomputation as shown in the following example.

Example

A D"

1 $

0 1:001

#

where by inspection, "1 D 1 and "2 D 1:001.



! It is easy to see that,"

1 $

0 1:001

#1

0D 1 #

"1

0

#

! Hence the first eigenvector is w1 D"

1

0

#.

! From,"

1 $

0 1:001

#"a

b

#D"

a C $b

1:001b

#D .1:001/

"a

b

#

we get,

$b D :001a

1:001b D 1:001b

which gives,a D 1000$b

! Letting b D 1, then the second eigenvector is w2 D"

1000$

1

#.

! The w2 eigenvector has a component which varies 1000 times fasterthan $ in the A matrix.

! In general, eigenvectors can be extremely sensitive to small changesin matrix elements when the eignvalues are clustered closelytogether.

Some Special Matrices

REAL SYMMETRIC MATRIX: ST D S

! Here S has real eigenvalues, "i and hence real eigenvectors, wi ,where the eigenvectors are orthonormal (i.e., wt

j ! wi D 0, for all



i ¤ j ) . Therefore its eigenvalue/eigenvector decompositionbecomes:

S D RƒRT ;

where RT R D RRT D I .

Corrollary: Any quadratic form,X

aij xixj can be written as xtSx and ispositive definite (i.e., is positive for x ¤ 0 ) if and only if "i .S/ > 0 forall i .

NORMAL MATRIX: N 'N D NN '

If ."i ; wi / is an eigen-pair of N , then! N"i ; wi

"is an eigen-pair of N '

The eigenvectors of N are orthogonal.

HERMITIAN MATRIX: H ' D H

H has real eigenvalues and orthogonal eigenvectors.

UNITARY MATRIX: U 'U D U U ' D I

The eigenvalues of U have unit modulus and the eigenvectors areorthogonal.

Jordan Form

! If the n $ n matrix A cannot be diagonalized, then it can always bebrought into Jordan form via a similarity transformation,

T %1AT D J D

264

J1 0: : :

0 Jq

375

where



Ji D

266664

"i 1 0

"i: : :: : : 1

0 "i

377775

2 Rni$ni

is called a Jordan block of size ni with eigenvalue "i . Note that J isblock-diagonal and upper bi-diagonal.

Singular Value Decomposition

! Despite its extreme usefulness, the eignenvalue/eigenvectordecomposition suffers from two main drawbacks:

1. It can only handle square matrices2. Its computation may be sensitive to even small errors in the

elements of a matrix

! To overcome both of these we can instead use the Singular ValueDecomposition (when appropriate).

! Let M be any matrix of dimension p $ m. It can then be factorized as:

M D U †V '

where U U ' D Ip and V V ' D Im. Here, † is a rectangular matrix ofdimension p $ m which has non-zero entries only on its leading diagonal:

† D

266664

%1 0 # # # 0 0 # # # 0

0 %2 # # # 0 0 # # # 0::: ::: : : : ::: ::: : : : :::

0 0 # # # %N 0 # # # 0

377775

and %1 ( %2 ( # # # ( %N ( 0.

! This factorization is called the Singular Value Decomposition (SVD).



! The % ’s are called the singular values of M .

! The number of positive (non-zero) singular values is equal to the rankof the matrix M .

! The SVD can be computed very reliably, even for very large matrices,although it is computationally heavy.

– In MATLAB it can be obtained by the function svd: [U,S,V] =svd(M).

! The largest singular value of M , %1, (also denoted, N%) is an inducednorm of the matrix M :

%1 D supx

kM xkkxk

! It is important to note that,

MM ' D U †2U '

andM 'M D V †2V '

! These are eigen-decompositions since U 'U D I and V 'V D I with† diagonal.

SOME PROPERTIES

! %i Dp

"i .M 'M/ Dp

"i .MM '/= i thsingular value of M

! vi D wi .M'M / = i th input principal direction of M

! ui D wi .MM '/ = i th output principal direction of M

Example (Golub & Van Loan):



A D"

0:96 1:72

2:28 0:96

#D U †V T

D"

0:6 0:8

0:8 %0:6

#"3 0

0 1

#"0:8 %0:6

0:6 0:8

#T

! Any real matrix, viewed geometrically, maps a unit-radius(hyper)sphere into a (hyper)ellipsoid.

! The singular values %i give the lengths of the major semi-axes of theellipsoid.

! The output principal directions ui give the mutually orthogonaldirections of these major axes.

! The input principal directions vi are mapped into the ui vectors withgain %i such that, Avi D %iui .

v1

v2

u2

3× u1

Domain of Matrix Input Image of Matrix Output

! Numerically, the SVD can be computed much more accurately thanthe eigenvectors, since only orthogonal transformations are used inthe computation.



Vector & Matrix Norms

Vector Norms

A vector norm on C is a function f W C ! R with the following properties:

! f .x/ ( 0 8 x 2 Cn

! f .x/ D 0 , x D 0

! f .x C y/ ) f .x/ C f .y/ 8 x; y 2 Cn

! f .˛x/ D j˛jf .x/ 8 ˛ 2 C; x 2 Cn

The p-norms are defined by

kxkp D

nXiD1

jxi jp!1=p

Three norms of particular interest to control theory are:

! 1-NORM

kxk1 DnX

i%1

jxi j

! 2-NORM

kxk2 D

vuut nXi%1

jxi j2 Dp

x'x

! 1-NORM

kxk1 D maxi

jxi j

The following relationships hold for 1, 2, and 1 norms for vectors:

! kxk2 ) kxk1 )p

n kxk2



! kxk1 ) kxk2 )p

n kxk1

! kxk1 ) kxk1 ) n kxk1

Matrix Norms

Matrix norms satisfy the same properties outlined above for vectornorms. The matrix norms normally used in control theory are also the1-norm, 2-norm and 1-norm.

! Another useful norm is the Frobenius norm, defined as follows:

kAkF D

0@

mXiD1

nXj D1

jaij j21A

1=2

8 A 2 Cm$n

Here aij denotes the element of A in the i th row, j th column.

! The p-norms for matrices are defined in terms of induced norms, thatis they are induced by the p-norms on vectors.

! One way to think of the norm kAkp is as the maximum gain of thematrix A as measured by the p-norms of vectors acting as inputs tothe matrix A:

kAkp D supx¤0

kAxkp

kxkp

8 A 2 Cm

! The matrix norms are computed by the following expressions:

kAk1 D maxj

mXiD1

jaij j



kAk1 D maxi

nXj D1

jaij j

kAk2 D N%.A/

SOME USEFUL RELATIONSHIPS:

! kABkp ) kAkp kBkp

! kAk2 ) kAkF )p

n kAk2

! maxi;j

jaij j ) kAk2 )p

mn maxi;j

jaij j

! kAk2 )p

kAk1 kAk1

! 1pn

kAk1 ) kAk2 )p

m kAk1

! 1pm

kA1k ) kAk2 )p

n kAk1

Toeplitz & Hankel Matrix Forms

Model-based Predictive Control formulations make extensive use ofToeplitz and Hankel matrices to simplify much of the algebra.Consider the polynomial n.z/:

n.z/ D n0 C n1z%1 C # # # C nmz%m

Then we define the Toeplitz matrices, &n, Cn for n.z/ from the followingmatrix form:

&n D#

Cn

Mn

$

where,



Cn D

26666664

n0 0 0 # # # 0

n1 n0 0 # # # 0

n2 n1 n0 # # # 0::: ::: ::: ::: :::

nm nm%1 nm%2:::

37777775

and

Mn D

264

0 nm nm%1:::

::: ::: ::: ::: :::

0 0 0 # # # n0

375

Define the Hankel matrix Hn as

Hn D

2666666666666664

n1 n2 n3 # # # nm%1 nm

n2 n3 n4 # # # nm 0

n3 n4 n5 # # # 0 0::: ::: ::: ::: ::: :::

nm%1 nm 0 ::: 0 0

nm 0 0 ::: 0 0

0 0 0 ::: 0 0::: ::: ::: ::: ::: :::

3777777777777775

Note: The dimension of matrices &n; Cn; Hn are not defined here as theyare implicit in the context in which they are used.

Toeplitz matrices defined in this manner are used for changingpolynomial convolution into a matrix-vector multiplication. As might beguessed, the relationship between matrix/vector multiplication andpolynomial operations is very useful for analysis of predictions.



Analytic Functions of Matrices

Let A be an n $ n matrix with eigenvalues "1; "2; : : : ; "n, and let p.x/ bean infinite series in a scalar variable x,

p.x/ D a0 C a1x C a2x2 C : : : C akxk C : : :

Such an infinite series may converge or diverge depending on the valueof x.

EXAMPLES

1 C x C x2 C x3 C : : : C xk C : : :

! This geometric series converges for all jxj < 1 .

1 C x C x2

2ŠC x3

3ŠC # # # C xk

kŠC # # #

! This series converges for all values of x and is in fact, ex.

These results have analogies for matrices.

Let A be an n $ n matrix with eigenvalues "i . If the infinite series p.x/

defined above is convergent for each of the n values of "i , then thecorresponding matrix inifinite series

p.A/ D a0I C a1A C a2A2 C # # # C akAk C # # # D

1XkD0

akAk

converges.

DEFINITION: A single-valued function f .z/, with z a complex scalar, isanalytic at a point z0 if and only if its derivative exists at every point insome neighborhood of z0: Points at which the function is not analytic arecalled singular points.



RESULT. If a function f .z/ is analytic at every point in some circle ( inthe complex plane, then f .z/ can be represented as a comnvergentpower series (Taylor series) at every point z inside (.

IMPORTANT RESULT. If f .z/ is any function which is analytic withing acircle in the complex plane which contains all the eigenvalues "i of A,then a corresponding matrix function f .A/ can be defined by aconvergent power series.

EXAMPLE:

The function e˛x is analytic for all values of x. Therefore it hasconvergent series representation:

e˛x D 1 C ˛x C ˛2x2

2ŠC ˛3x3

3ŠC # # # C ˛kxk

kŠC # # #

The corresponding matrix function is

e˛A D I C ˛A C ˛2A2

2ŠC ˛3A3

3ŠC # # # C ˛kAk

kŠC # # #

EXAMPLE:

eAt # eBt D e.ACB/t

if and only if, AB D BA.

OTHER USEFUL REPRESENTATIONS:

sin A D A % A3

3ŠC A5

5Š% # # #

cos A D I % A2

2ŠC A4

4Š% # # #



sinh D A C A3

3ŠC A5

5ŠC # # #

cosh D I C A2

2ŠC A4

4ŠC # # #

sin2 A C cos2 A D I

sin A D ejA % e%jA

2jand cos A D ejA C e%jA

2

Cayley-Hamilton Theorem

If the characteristic polynomial of a matrix A isj"I % Aj D .%"/n C cn%1"

n%1 C cn%2"n%2 C # # # C c1" C c0 D 4."/

then the corresponding matrix polynomial is4.A/ D .%1/nAn C cn%1A

n%1 C cn%2An%2 C # # # C c1A C c0I

CAYLEY-HAMILTON THEOREM. Every matrix satisfies its owncharacteristic equation.

4.A/ D 0

The proof of this follows by applying a diagonalizing similaritytransformation to A and substituing into 4.A/.

EXAMPLE:

A D"

3 1

1 2

#

j"I % Aj D .3 % "/.2 % "/ % 1 D "2 % 5" C 5

4.A/ D A2 % 5A C 5I D"

10 5

5 5

#% 5 #

"3 1

1 2

#C 5 #

"1 0

0 1

#D"

0 0

0 0

#


mathematical fundamentalsmocha-java.uccs.edu/ece5590/ece5590_ch2_f15.pdf · 2019. 4. 24. · in...

Documents