indr 262 introduction to o ptimization methods

INDR 262INTRODUCTION TO

OPTIMIZATION METHODS

LINEAR ALGEBRA

INDR 262 Metin Türkay 1

Matrices

• m and n are positive integers• Order of matrix: mxn• The number in the ith row and jth column of A is

called the ijth element of A and is written aij.

mnmm

n

n

aaa

aaaaaa

A

..........

..

..

21

22221

11211


Example

761

987654321

31

23

11

aaa

A


Equal Matrices • Two matrices A and B are equal if and only if A

and B are of the same order and for all i and j, aij=bij.

• If A=B, then x=1, y=2, w=3, z=4

1 2 B

3 4x y

Aw z


Vectors• A list of n real numbers, say (a1, a2, …, an) is called an n-

dimensional vector. An n-dimensional vector also maybe displayed as a 1 by n matrix.

321 ,21


Scalar Product of Two Vectors

• The scalar product of vectors

u = [u1 u2 … un] and

is u1v1+u2v2+…+unvn

nv

vv

v..2

1


Example

• uv = 1x2+2x1+3x2 = 10

212

321 vu


Notes• If u = [1 2 3] and , then uv is

not defined because the vectors are of different dimensions.

• Two vectors are perpendicular to each other if and only if their scalar product is equal to 0.

• E.g., u = [1 -1] and .

43

v

11

v


Scalar Multiple of a Matrix

• Given any matrix A and any scalar c, the scalar multiple of matrix A, cA, is obtained from the matrix A by multiplying each element of A by c.

0363

3 0121

AA


Addition of two Matrices

• Let A=[aij] and B=[bij] be two matrices with the same order (say mxn). Then, the matrix C=A+B is defined to be the mxn matrix whose ijth element is aij+bij.

002000

111120332211

112321

110321

BAC

BA


The Transpose of a Matrix• Given any mxn matrix

the transpose of A (written AT) is the nxm matrix

mnmm

n

n

aaa

aaaaaa

A

..........

..

..

21

22221

11211

nmnn

n

n

T

aaa

aaaaaa

A

..........

..

..

21

22212

12111


Example• For any matrix A, (AT)T=A.

654321

635241

654321 T TTAAA


Matrix Multiplication

• Given two matrices A and B, the matrix product of A and B is defined if and only if the number of columns in A is equal to the number of rows in B. The matrix product C=AB is determined as follows:

cij = scalar product of (row i of A and column j of B)


Properties of Matrix Multiplication1. Matrix multiplication is associative, i.e.,

A(BC)=(AB)C.

2. Matrix multiplication is distributive, i.e., A(B+C)=AB+AC.


Example

531

12

421

12

431

11

321

11

22

21

12

11

c

c

c

c

ABC

BA

3211

1211

5443

ABC


Matrices and Systems of Linear Equations

• Consider a system of linear equations given by

• x1, x2, …, xn are referred to as variables• aij’s and bi’s are constants• A set of equations like above is called a linear system of m

equations in n variables

nnmnmm

nn

nn

bxaxaxa

bxaxaxabxaxaxa

..............................

...

...

2211

22222121

11212111


Solution

• A solution to a linear system of m equations in n unknowns is a set of values for the unknowns that satisfy each of the systems m equations.


Example x1 + 2x2 = 52x1 - x2 = 0

21

x

13

x

00 0 ? 2)1(2

55 5 ? )2(21

05 0 ? 1)3(2

55 5 ? )1(23

Solution

Not a solution


Matrix Representation of Systems of Linear Equations

Ax = b

nnmnmm

n

n

b

bb

b

x

xx

x

aaa

aaaaaa

A...

,...

,

...............

...

...

2

1

2

1

21

22221

11211

mmnmm

n

n

b

bb

aaa

aaaaaa

bA...

...............

...

...

2

1

21

22221

11211

Augmented matrix


The Gauss-Jordan Method for Solving Systems of Linear Equations

• Gauss-Jordan method is used to find solution(s) to systems of linear equations. A system of linear equations must satisfy one of the following cases:

• Case 1: The system has no solution• Case 2: The system has a unique solution• Case 3: The system has an infinite number of solutions


Elementary Row Operations• An ero transforms a given matrix A into a new matrix A’ via one

of the following operations.• Type 1 ero: A’ is obtained by multiplying any row of A by a

nonzero scalar.• Type 2 ero: Begin by multiplying any row of A (say, row i) by a

nonzero scalar c. For some ji, let row j of A’ = c(row i of A) + row j of A, and let the other rows of A’ be the same as the rows of A.

• Type 3 ero: Interchange any two rows of A.


Facts• If the matrix A’ is obtained from A via an ero, A’ and A are

equivalent.• If the augmented matrix [A’b’] is obtained from [Ab] via

an ero, the systems Ax=b and A’x=b’ are equivalent.• Any sequence of ero’s performed on the augmented matrix

[Ab] corresponding to the system Ax=b will yield an equivalent linear system.


Gauss-Jordan Method• The Gauss-Jordan method solves a linear system of

equations by utilizing ero’s in a systematic fashion.

Step 1 To solve Ax=b, write down the augmented matrix [Ab].

Step 2 At any stage, define a current row, current column, and a current entry. Begin with row 1 as the current row, column 1 as the current column, and a11 as the current entry.


Gauss-Jordan Method

a If a11 (the current entry) is nonzero, use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.

b If a11 (the current entry) equals 0, then do a Type 3 ero involving the current row and any row that contains a nonzero entry in the current column. Use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.

c If there are no nonzero numbers in the first column, obtain a new current column and entry by moving one column to the right. Then go to Step 3.


Gauss-Jordan Method

Step 3a If the new current entry is nonzero, use ero’s to transform it to 1 and the

rest of the current column’s entries to 0. When finished, obtain a new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.

b If the current entry is 0, do a Type 3 ero with the current row and any row that contains a nonzero entry in the current column. Then, use ero’s to transform column entry to 1 and the rest of the current column’s entries to 0. When finished, obtain the new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.

c If the current column has no nonzero numbers below the current row, obtain a new current column and entry and repeat Step 3. If it is impossible, stop.


Gauss-Jordan Method

Step 4 Write down the system of equations A’x=b’ that corresponds to the matrix [A’b’] obtained when Step 3 is completed. Then, A’x=b’ will have the same set of solutions as Ax=b.

• The Gauss–Jordan method converts the augmented matrix [Ab] into [A’b’] such that

'

'2

'1

...1...00............0...100...01

''

mb

bb

bA''

22'11 ..., , , mn bxbxbx


Example

• Solve the following system of linear equations using the Gauss-Jordan method.2x1 + 2x2 + x3 = 92x1 - x2 + 2x3 = 6 x1 - x2 + 2x3 = 5


Solution

569

211212122

bA

562

9

2112122

111

11 bA

532

9

2111302

111

22 bA

21

32

9

23201302

111

33 bA


Solution

2112

9

2320

31102

111

44 bA

2112

7

2320

31106

501

55 bA

2512

7

6500

31106

501

66 bA

312

7

1003

1106

501

77 bA


Solution

321

100010001

99 bA

311

1003

110001

88 bA

x1 = 1 x2 = 2 x3 = 3


Analysis of the Solutions to Systems of Linear Equations

• For any linear system, a variable that appears with a coefficient of 1 in a single equation and a coefficient of 0 in all other equations is called a basic variable. Any variable that is not a basic variable is called a nonbasic variable.

x set of all of the variables in the system Ax=bxB set of all of the basic variables in the system Ax=bxN set of all of the nonbasic variables in the system

Ax=bx = xB xN


Analysis of the Solutions to Systems of Linear Equations

• The solution to A’x=b’ can be categorized in one of the three cases:

Case 1: A’x=b’ has at least one row of the form [0 0 … 0c] and c0. Then, Ax=b has no solution.

Case 2: When case 1 does not apply and xN=, then Ax=b has a unique solution.

Case 3: When case 1 does not apply and xN, then Ax=b has infinite number of solutions.


Example

• Case 1 does not apply since there are no rows of the form [0 0 … 0c] and c0.

• Case 2 does not apply since,xB={x1, x2, x3}xN={x4, x5}

• There are infinite number of solutions.

0123

00000101000201011001

'' bA


Example

• Assign arbitrary values to the variables in xN; x4=c, x5=k.• Write down the equations in [A’b’],

x1 + c + k = 3 x1 = 3 - c – kx2 + 2c = 2 x2 = 2 - 2cx3 + k = 1 x3 = 1 - k

• It is easy to see that there are infinite number of values of c and k that will satisfy this system of equations.

0123

00000101000201011001

'' bA


Linear Combination• A linear combination of the vectors in V is any vector of the

form c1v1+c2v2+…+ckvk, where c1, c2, …, ck are arbitrary scalars.

• Example: V={[1,2], [2,1]}

2v1-v2 = 2([1 2]) – [2 1] = [0 3]0v1-3v2 = 0([1 2]) – 3([2 1]) = [6 3]


Linear Independence & Linear Dependence

• A set V of m-dimensional vectors is linearly independent if the only linear combination of vectors in V that equals 0 is the trivial linear combination.

• A set V of m-dimensional vectors is linearly dependent if there is a nontrivial linear combination of the vectors in V that adds up to 0.


Example 1

• V={[1,0], [0,1]} Try to find a linear combination of vectors in V that yields 0.

• c1([1 0]) + c2([0 1]) = [0 0]• In order to satisfy this, [c1 c2] = [0 0] c1=c2=0• The only linear combination of vectors in V that yields 0

is the trivial linear combination. Therefore, V is a linearly independent set of vectors.


Example 2• V={[1,2], [2,4]} Try to find a linear combination of

vectors in V that yields 0.• c1([1 2]) + c2([2 4]) = [0 0]

[c1 2c1] + [2c2 4c2] = [0 0]c1 + 2c2 = 0 c1 = -2c2

2c1 + 4c2 = 0 2c1 = -4c2• So, c1 = 2 c2 = -1 is one of the possible solutions.• There exists a nontrivial linear combination of vectors in

V that yields 0. Therefore, V is a linearly dependent set of vectors.


The Rank of a Matrix

• Let A be any mxn matrix, and denote the rows of A by r1, r2, …, rm. Also define R={ r1, r2, …, rm}.

• The rank of A is the number of vectors in the largest linearly independent subset of R.

• If for a matrix A with m rows, rank A=m; then the matrix is a collection of linearly independent set of vectors. If rank A<m; then the matrix contains a linearly dependent set of vectors.


Examples

3202

110001

320120001

A

1002

110001

100010001

Rank A = 3

011010001

rank B Rank B = 2


The Inverse of a Matrix• A single linear equation in a single variable can be solved

by multiplying both sides of the equation by multiplicative inverse of the variable coefficient.

• Example:4x=3 4-1(4x) = (4-1)3 x=3/4

• We can generalize this approach to square systems of linear equations (i.e., number of equations = number of unknowns).


Square and Identity Matrix

• A square matrix is any matrix that has an equal number of rows and columns.

• The diagonal elements of a square matrix are those elements aij such that i=j.

• A square matrix for which all diagonal elements are equal to 1 and all non-diagonal elements are equal to 0 is called an identity matrix.

100010001

, 1001

32 II


Inverse of A• For a given mxm matrix A, the mxm matrix B is

the inverse of A if BA = AB = Im

100010001

201715

101

101213102

AA-1=I


Finding the Inverse of a Matrix with the Gauss-Jordan Method

Step 1 Write down the mx2m matrix [AIm].

Step 2 Use ero’s to transform [AIm] into [ImB]. This will only be possible if rank A=m; in this case, B=A-1. If rank A<m, then A has no inverse.


Example

3152

A

1001

3152

2IA

1002

1

312

5112IA

12

102

1

210

251

22IA

2102

1

102

5132IA

2153

1001

42IA

21531A


Using Matrix Inverse to Solve Linear Systems of Equations

• Given a linear system of equations,Ax=b• Multiply both sides by A-1

AA-1x = A-1b x = A-1b


Example2x1 + 5x2 = 7

x1 + 3x2 = 4

47

3152

2

1xx

2153

3152 1-AA

11

47

2153

2

1

2

1xx

xx


Determinant of a Matrix• Any square matrix A has a number called the

determinant of A (shown by set A or A).

122122112221

1211 det aaaaAaaaa

A


Minor of A• If A is an mxm matrix; then for any values of i

and j, the ijth minor of A (Aij) is the (m-1)x(m-1) submatrix of A obtained by deleting row i and column j of A.

1211

, 2012

210112

101

3212 AAA


Determinant of a Matrix• Let A be any mxm matrix,

• det A = (-1)i+1ai1(det Ai1) + (-1)i+2ai2(det Ai2) + … +(-1)i+maim(det Aim)

mmmm

m

m

aaa

aaaaaa

A

...............

...

...

21

22221

11211


Example 1

210112

101A

1 1 1 2 1 31 1 2 1 2 1det ( 1) 1 det ( 1) 0 det ( 1) 1 det

1 2 0 2 0 1det 1x1x 3 (-1)x0x4 1x1x2det 5

A

AA

det A = (-1)1+1 1 det A11 + (-1)1+2 0 det A12 + (-1)i+3 1 det Ai3


Example 2

5121212211131201

A

det A = -48


indr 262 introduction to o ptimization methods

Documents

metin trkay

nxm matrix indr

matrix c

mxn matrix

given matrix

example indr

matrix multiplicationgiven

new matrix