ls 1 linear algebra basics

7/27/2019 LS 1 Linear Algebra Basics

1/42

Lec 02: Mathenatical EconomicsLinear Algebra Basics

Sugata Bag

Delhi School of Economics

6th August 2013

[SB] (Delhi School of Economics) Introductory Math Econ 6th A ugust 2013 1 / 42
http://find/


2/42

Vector Spaces Basic Setting

Basic Concepts

The objective here is to generalize the notion we see on a plane: a way toadd vectors, and to scale them up or down using scalars, s.t. vector

addition and multiplication with scalars follow certain rules. Our scalarswill mostly be real numbers; more generally, we want the set of scalars tocomprise a eld. About vectors, more later.

Lets start with some modern algebra, specically with Group Theory.

http://find/


3/42


Binary Composition / Operation

An operation (or ) which when applied to two elements of a set S

gives a unique element, also of the same set, is called a binary operation.Denition

Given a set S, any mapping of S S into S is called a binary compn on S.

Thus given any set S, let be a bin.comp. on S. If a, b 2 S so that(a, b) 2 S S and 9 a unique c 2 S such that c = ab = (a, b) 2 S .

This is the Closure Property and the set S is closed w.r.t. operation .

Algebraic Structure: A non-empty set together with one or more thanone operations is called algebraic structure. Ex: (N, +), (Z, +, )

Examples

Addition is a binary operation on the set of even numbers, but is noton the set of odd natural numbers. Why?

Subtraction is a binary operation the set of all integers, positive and

negative, but not, if be limited to only positive integers.[SB] (Delhi School of Economics) Introductory Math Econ 6th A ugust 2013 3 / 42
http://find/


4/42


Laws of Binary Operations

Associative law: A binary operation on the elements of the set S issaid to be associative, if and only if, for any three elements a, b, c 2 S,a (b c) = (a b) c.

Commutative law: A binary operation on the elements of the set S is

commutative, if and only if, for any two a, b 2 S, ab = baDistributive law : Two binary operations and , say, on elements of theset S are said to be distributive, i, for every element a, b, c 2 S, .. a(b c) = (ab) (ac) [left distributive]. (b c)a = (ba) (ca) [right distributive]

Identity element. A set S is said to possess an identity element withrespect the binary operation o, if there exists an element I 2 S withproperty that Ix = xI = x, for every x 2 S.


V S B i S i
http://goforward/http://find/http://goback/


5/42


Group

Denition

A group G is a nonempty set G along with one binary operation ? s.t.(i) for every pair g, h 2 G, we have g? h 2 G [ :: closure](ii) for every 3 elements g, h,j 2 G, we have. g? (h ?j) = (g? h) ?j [ :: associativity]

(iii) There exists a unique element e 2 G s.t. for all g 2 G,. g? e = e? g = g. (Here, e is the identity element)(iv) For every g 2 G, there exists a unique element h 2 G s.t.. g? h = h ? g = e. (Here, h is the inverse of g, written g1).

If, 8g, h 2 G, g? h = h ? g, then G is a commutative or Abelian group.

Examples

Examples: Real or complex numbers with addition. Multiplication of realsis not a group: 0 does not have an inverse under multiplication.


V t S B i S tti
http://find/http://goback/


6/42


Rings, Fields

DenitionA ring G is a set with at least 2 elements, along with 2 operations,addition (+) and multiplication ( ) , such that -(i) G with + is a commutative group with identity element called 0 (zero)(ii) G is closed under multiplication ( ) is associative, and there is amultiplicative identity element, called 1, s.t. 1 6= 0, where 0 is the identityelement under +.(iii) 8g, h,j 2 G, g (h +j) = (g h) + (g j), and. (g + h) j = (g j) + (h j).

Field: Further, if the non-zero elements of G form a commutative groupunder multiplication with 1 as identity element, then G is called a Field.Example: The rationals, the reals and the complex numbers with usualaddition and multiplication are elds.




7/42


Vector Spaces

Denition

A vector space V over a eld F of scalars, is a set with 2 operations -vector addition (called, an internal composition) and multiplication byscalar (called, an external composition) s.t. -

(i) V with + is a commutative group with identity element 0 2 V(ii) If v 2 V, cv 2 V, 8c 2 F; and for c1, c2 2 F, (c1c2)v = c1(c2v);

1 2 F, the multiplication identity element for eld F, satises1v = v, 8v 2 V.

(iii) For all c 2 F and v1 , v2 2 V, c(v1 + v2) = cv1 + cv2, and

for all c1 , c2 2 F and v 2 V, (c1 + c2)v = c1v + c2v.

Note that we use the same symbol + for both vector addition and scalaraddition but they are dierent things.


http://find/


8/42


Examples

Example

(1).


9/42

Vector Spaces Bases and Dimension

Span(nig) Set ; Subspace

Let v1 , ..., vm 2 A. A linear combination of these vectors is a vector of

the form c1v1 + ... + cm vm , where c1 , ..., cm 2 F eld of scalars.Denition

Span: Let V be a v.s. over the eld F and A = fv1 , ..., vm g be anynon-empty subset of V. The (linear) span of set A is dened as the set of

all linear combinations of nite sets of elements of A. It is denoted byL(A) or span(A).Thus, L(A) = fc1 .v1 + ...... + cn .vn : vi 2 A; ci 2 F, i = 1, 2, . . . , ng.L(A) is said to be generated or spanned by the set A; and A is said to bethe set of generators of L(A). Clearly, A L(A) .

Denition

Subspace: A non-empty set S V is a subspace of V, if S itself be avector space over eld F w.r.to the same compositions as dened in V andfor all v1, v2 2 S, span(fv1 , v2g) S.


http://find/


10/42


Subspace - Facts & Examples

Fact(i) A subset S of a vetor space V is a sub-space of V if S be closed w.r.tovector addition and multiplication by scalar. Therefore, a subspace is itselfa vector space,

(iii) every subspace contains 0, the zero vector (identity element in vectoraddition). The latter is true since 0v1 + 0v2 = 0 and LHS is a linearcombination of v1, v2 and hence lies in their span.

Example

Examples of subspaces of various dimensions in


11/42

p

Linear Dependence and Linear Independence

Denition

Linear Dependence: A set of vectors fv1, ..., vn g over a eld of scalarsfc1 , ..., cn g, not all zero, s.t. c1v1 + ....cnvn = 0 or,

ni=1 civi = 0.

. If there does not exist any such set of not-all-zero scalars, we sayfv1, ..., vn g are linearly independent (LI). Alternatively,.

fv

1, ..., v

n gis LI ifn

i=1c

iv

i =0

)c

i =0,

8i

2 f1, ..., n

g.

Note. (i) What is the geometry in


12/42

p

Bases

Denition

A set of vectors fv1 , ..., vn g is a basis for a vector space Vn if (i)it is LI,and (ii) span(fv1 , ..., vn g) = Vn.

Examples. f(1, 0), (0, 1)g and f(1, 0), (1, 1)g are 2 dierent bases for


13/42

Some Comments

A system consisting of a single non-zero vector is always linearlyindependent .

Every super-set of a linearly dependent set of vectors is linearlydependent.

Any sub-set of linearly independent set of vectors is linearly

independent.The sub-set containing the single element (1, 0, 0) of the space V3 inF generates the sub-space which is the totality of the elements of theform (a, 0, 0). The sub-set f(1, 0, 0), (0, 1, 0)g of V3 in F generates

sub-space which is the totality of elements of the form (a,

b,

0).Thus the sub-set f(1, 0, 0), (0, 1, 0)g of V3 in F can not generate anyvector (a, b, c) so they cannot form a basis in V3 . In fact we needanother vector, say (0, 0, 1), along with aforementioned two vectorsto span entire V3 and they become a basis.

[SB] (Delhi School of Economics) Introductory Math Econ 6th August 2013 13 / 42

http://find/


14/42

Unique Representation

TheoremSuppose fv1 , ..., vn g is a basis for a vector space V . Then every vectorv 2 V has a unique representation v = ni=1 civi in terms of the basis.

Proof.Suppose 2 sets of scalars (c1 , ..., cn ) and (c

0

1 , ..., c0

n ) represent v relative tothe basis. Then v = n1 civi =

n1 c

0

ivi.So,

n1 civi

n1 c

0

ivi = n1 (ci c

0

i)vi = 0.So by LI of the basis vectors, ci c

0

i = 0, or ci = c0

i, for all

i 2 f1, ..., ng.

Note that the basis vector vi has the representation (0, ..., 0, 1, 0, ..., 0)(with 1 in the ith place).


http://find/


15/42

Dimension

Theorem

If fv1 , ..., vn g and fw1 , ..., wm g are bases for a v.s. V , then m = n.

Proof.

Suppose WLOG m n. Since the vs form a basis, w1 = n1 civi, with at

least one ci 6= 0 (since w1 6= 0 by LI).WLOG let c1 6= 0; then v1 = c

11 w1

n2 c

11 civi. So fw1 , v2 , ..., vng

spans V. Its also LI, as fv2 , ..., vn g are LI and w1 is not a LC of them (ifit were, c1 would be 0). So fw1 , v2 , ..., vn g is a basis.

We can iterate this argument n times (see below), replacing v2 with w2and so on. After n replacements, we get that fw1 , ..., wn g is a basis. Butthen fw1 , ..., wm g cannot have any additional vectors: since this is a LIset, an additional vector, say wn+1 cannot be a LC of fw1 , ..., wn gcontradicting that this latter set is a basis. So, m = n.


http://find/


16/42

Notes

Note. To expand on the proof, we have a basisfw1 , ..., wk1 , vk, ..., vng after k 1 iterations.

So, wk = k11 aiwi +

nk civi, with not all cis being 0, because

otherwise we would imply that the wis were LD.

WLOG, let ck 6= 0 (by possibly renumbering the vectors vk, ..., vn),and so vk is a LC of fw1 , ..., wk, vk+1 , ..., vn g and we can argue thislatter set is a basis.

Note also that in the rst iteration, fw1 , v2, ..., vn g, you can show LImore painfully but clearly by showing

(i) fv2 , ..., vn g is LI since it is a subset of an LI set;(ii) w1 is not a LC of fv2 , ..., vn g;(iii) therefore, expanding the set fv2, ..., vn g with w1 preserves LI.


http://find/


17/42

Dimension (of V)

Denition

The number of elements in any basis of V is called the dimension of V,denoted dim(V). If this is nite, we say V is a nite dimensional vector

space.

Note. Our denitions of LD and LI, and things which followed, are all fornite sets of vectors. The denitions are restated slightly if the topic ofdiscussion is innite dimensional vector spaces. The function space C[a, b]

is one such space - function spaces are also important in economics butoutside the scope of the present discussion.


Linear Transformations The Fundamental Theorem
http://find/


18/42

Linear Transformations

Denition

Let V, W be two vector spaces. A linear transformation T : V ! W isa function satisfying T(c1v1 + c2v2) = c1T(v1) + c2T(v2), for all scalarsc1 , c2 and vectors v1 , v2 2 V.

This is equivalent to T satisfying - (i) T(v1 + v2) = T(v1) + T(v2) and

(ii) T(cv) = cT(v), 8c 2 K, v 2 V.

Example

Rotation of all points in


19/42

Range Space (of T)

Example

A matrix Amn is a linear transformation. T :


20/42

Range Space

In the context of linear transformations, the range is called range space

because it is a subspace. Indeed, let w1, w2 2 R(T) and c1 , c2 be scalars.So, w1 = T(v1), w2 = T(v2), for some vectors v1 , v2 2 V. Searching for apreimage for c1w1 + c2w2,

c1w1 + c2w2 = c1T(v1) + c2T(v2) = T(c1v1 + c2v2),

the last equality since T is linear. So, c1w1 + c2w2 has preimagec1v1 + c2v2, and is hence in R(T). So R(T) is a subspace.

Range spaces of matrices. Let Amn = (a1 a2 an) written in termsof its n columns. Since Ax = x1a1 + x2a2 + ... + xnan, R(A) or set of allimages like Ax is equal to span(fa1, ..., ang), the subspace spanned by the

columns of A.Draw range spaces of

1 21 2

1 23 4

0 00 0




21/42

Null Space (of T)

Since R(T) is a subspace, it has a dimension. The dimension of R(T)

(denoted dimR(T)) is called the Rank of T. A matrix Amn with k LIcolumns has the property that its columns span a k-dimensional subspace.This dimension therefore corresponds to the notion of Rank(A) beingequal to the number of LI columns of A.

DenitionThe Null Space or Kernel of an LT T : V ! W, denoted N(T), is thesubset of vectors in V with image 0 in W; i.e.N(T) = fv 2 VjT(v) = 0g.

What are the Null Spaces of the matrices in the previous slide? We cansee that they are all subspaces of the domain. This is true in general. Ifv1, v2 2 N(T), we have T(v1) = T(v2) = 0. So for all c1 , c2 2 K,T(c1v1 + c2v2) = c1T(v1) + c2T(v2) = 0 + 0. So c1v1 + c2v2 2 N(T).The dimension of N(T), dim(N(T), is called the Nullity of T.


http://find/


22/42

Some Remarks

Notice in the examples of 2 2 matrices that their Rank and Nullity add

up to 2, which is also the dimension of the domain


23/42

The Fundamental Theorem of Linear Algebra

Theorem

Let T : V ! W. Then dim(V) = Rank(T) + Nullity(T).

Proof.

Suppose that fv1 , ..., vkg be a basis for N(T). We extend this tofv

1, ..., v

k, v

k+1, ..., v

ng, a basis for V in the domain, so that dim(V) = n,

and Nullity(T) = k n. Then it suces to show that dim R(T) = n kor,fw1 , ..., wnkg fT(vk+1), ..., T(vn )g is a basis for R(T).Lets conjecture that fT(vk+1), ..., T(vn )g forms a basis of image T.Part (i). Show span(fT(vk+1), ..., T(vn )g) = R(T).

Let w 2 R(T), so w = T(v) for some v 2 V. In terms of the chosenbasis, let v = a1v1 + ... + akvk + ak+1vk+1 + ... + anvn, for some scalarsa1 , ..., an. So, w = T(v) = T(

k1 aivi) + T(

nk+1 aivi) =

nk+1 aiT(vi)

The 2nd equality is due to linearity of T; the 3rd because k1 aivi 2 N(T)

and T is linear. Since w is arbitrary in R(T) and is a LC of

fT(vk+1), ..., T(vn )g, this set spans R(T).[SB] (Delhi School of Economics) Introductory Math Econ 6th August 2013 23 / 42

http://find/


24/42

Fundamental Theorem proof contd...

We only now need to show that this list is not redundant; that is , that

fT(vk+1), ..., T(vn )g are linearly independent. We can do this by showingthat a linear combination of these vectors is zero if and only if thecoecient on each vector is zero.

Proof.

Part (ii). Show fT(vk+1), ..., T(vn )g is LI.Let ck+1T(vk+1) + ... + cnT(vn ) = 0, [for scalars ck+1 , ..., cn ]() T(ck+1vk+1 + ... + cnvn) = 0, [by linearity]) ck+1vk+1 + ... + cnvn 2 N(T), [by defn of nullity]But then, since fv1 , ..., vkg span N(T), there exists some scalars c1 , ..., ck

such that ck+1vk+1 + ... + cnvn = 0 = c1v1 + ... + ckvk) c1v1 + ... + ckvk ck+1vk+1 ... cnvn = 0.But, since fv1 , ..., vn g form a basis of V, they should be LI, so all the cisequal 0, so that fT(vk+1), ..., T(vn )g is LI and span the range.

For an m n matrix A we have n = Rank(A) + Nullity(A).[SB] (Delhi School of Economics) Introductory Math Econ 6th August 2013 24 / 42

Linear Transformations Linear Equations
http://find/


25/42

Application: Systems of Linear Equations

Given b = (b1 , ..., bm ) 2


26/42

General Solution Set of Ax=b

=[Particular Solution of Ax = b] + [General Solution of Ax = 0]

Lemma

Let A be an m n real matrix, b a vector in


27/42

Solution Set. contd ...

Note that if Nullity(A) 1, then the system of equations has multiplesolutions, if there are any solutions at all. As an example for the lemma,consider a system of equations with

A = 1 2

1 2 b =

5

5

Draw the solution set and the 1-dimensional subspace N(A).

Theorem

Let A be n n, and b2


28/42

Linear Equation - Theorems

Proof.

(i) Rank(A) = n ) R(A) =


29/42

Linear Equation - Theorems

Proof.

(i) Rank(A) = m ) R(A) =


30/42

Linear Implicit Function Theorem

Theorem

Consider the system

a11x1 + ... + a1kxk + a1k+1xk+1 + ... + a1nxn = b1 am1x1 + ... + amkxk + amk+1xk+1 + ... + amn xn = bm

Fix xk+1 , ..., xn at values xk+1 , ..., x

n respectively. For each such choice of

parameters", there exists a unique set of values x1 , ..., xk that solves the

equation system if and only if(i) k = m

(ii) Rank

0B@

a11 a1k...

. . ....

am1 amk

1CA

m=k

= k



Li I li i F i Th
http://find/


31/42

Linear Implicit Function Theorem

Proof.

For each equation i, carry aik+1xk+1 + ... + ain x

n to the right hand. Then

from the rst two theorems above, the only way for a solution to alwaysexist, and be unique, is if the underlying matrix is square and of fullrank.

See Simon and Blume, p 150. The implicit function theorem is ageneralization of this and in economics is used to solve for endogeneousvariables when one or more parameters change.Geometry of Ax = b another look:. Look at the ith row vector ri as a

1 n matrix. So rix = bi is itself a linear equation. Since Rank(ri) = 1,Nullity(ri) = n 1. So the solution set is a translate of an(n 1)-dimensional subspace. So, the solution set of Ax = b is theintersection of m such (n 1)-dimensional subspaces. E.g. if A is 3 3,the solution set is an intersection of 3 two-dimensional subspaces or planes.



S l i Li E ti E l
http://find/


32/42

Solving Linear Equations - Examples

Two examples from Treil (pages 40 - 45; section 2, chap 2). Draw

attention to writing out a solution set as a translate of the null space of A.This method also indicates a basis for N(A).

Note(1) Let A be an m n matrix, b 2


33/42

Solving Linear Equations

We use Row Reduction. Write down the augmented matrix (Ajb).

(1). Find leftmost non-zero column of the augmented matrix.(2). Make sure, by row interchange if necessary, that the rst entry of

this column is non-zero. This is the pivot.(3). Reduce all non-zero entries below the pivot by row operations of

type 3: Rj ! Rj + aR1.

(4). Ignore Row 1 and consider the matrix from Row 2 to Row m.Repeat steps (1) to (3). Iterate until the matrix is in echelon form.That is, the matrix has -

(i) all zero rows below non-zero rows, and

(ii) the rst non-zero entry of a non-zero row is to the right of therst non-zero entry of the row above it.(iii) Then use back substitution to solve the system.

Example: Consider the system x1 + 2x2 + 3x3 = 1, 3x1 + 2x2 + x3 = 7,2x1 + x2 + 2x3 = 1. The augmented matrix is:



E a le
http://find/


34/42

Example

0@ 1 2 3 13 2 1 72 1 2 1

1A

R2 3R1, R3 2R1 gives

0@ 1 2 3 10 4 8 4

0 3 4 1

1A

R3 (3/4)R2 gives

0@ 1 2 3 10 4 8 4

0 0 2 4

1A

Back substitution gives x3 = 2, x2 = 3, x1 = 1.



Multiple Solutions Example
http://find/


35/42

Multiple Solutions Example

Suppose the echelon form of a system is

0@ 1 2 0 0 0 10 0 1 5 0 20 0 0 0 1 3

1A

(i) Identify the pivot (rst nonzero entry) in each row (in bold above); (ii)Identify the columns without pivots (here, 2, 4), the correspondingvariables x2 , x4 are free - we can x them at any real numbers and solvefor x1 , x3, x5. (iii) Write out the system of equations with the free variableson the RHS. (x1 = 1 + 2x2, x2 free, x3 = 2 + 5x4, x4 free, x5 = 3.(iv)Rewrite the solution: Stack them with x 2


36/42

Multiple Solutions

The soln set is a translate of subspace spanf(2, 1, 0, 0, 0), (0, 0, 5, 1, 0)g .So this span must be the Null Space of the echelon matrix U below0

@ 1 2 0 0 00 0 1 5 00 0 0 0 1

1A

Note that N(U) = fx 2


37/42

Representing a LT wrt standard bases

Let T :


38/42

Matrix Representation of LT

Theorem

Let T : V ! W , A = fa1 , ..., an g, B = fb1 , ..., bm g bases in V, Wrespectively. Then there exists a matrix, denoted [T]BA s.t. w = T(v)implies [w]B = [Tv]B = [T]BA[v]A.

Proof.

Let v = n1 xjaj, i.e. coordinate representation [v]A = (x1 , ..., xn )T. So

we have, T(v) = T(n1 xjaj) = n1 xjT(aj), by linearity of T.

For j = 1, ..., n, writing in terms of basis B,

T(aj) = mi=1 aijbi, i.e. [T(aj)]B = (a1j, ..., amj)

T.

So, w = T(v) = nj=1 xj mi=1 aijbi =

mi=1 bi

nj=1 aijxj.

So if [T(v)]B = (y1 , ..., ym )T, we have, for i = 1, ..., m,

yi = n

j=1 aijxj, so that (y1 , ..., ym )T = A(x1 , ...., xn )T, where the columns

of A represent the T(aj)s w.r.t. B. This A [T]BA .


Linear Transformations Matrix Representation of Linear Transformations

Matrix Representation - Applications
http://find/


39/42

Matrix Representation - Applications

To summarize, if w = T(v) in coordinate-free language, then if we write

vectors in V and W in terms of bases A and B respectively, this mappingis achieved by the matrix [T]BA . If aj is the jth basis vector in A, thenthe jth column of [T]BA is [T(aj)]B, i.e. T(aj) written in terms of itscoordinates w.r.t. the bases B of W.

By the balance of indices in the formula [T(v)]B = [T]BA [v]A, Treilpresumably means that to go from basis A to B, the Bs appear in theformula rst, followed by the As.

Change of Basis Matrix. As an application, consider a vector space V.Consider 2 dierent bases A and B for V. What is the mapping[v]A 7! [v]B? That is, suppose I rst represent vectors v in terms of basisA, and then want to nd out their representation in terms of a dierentbasis B. Is there a matrix A s.t. [v]B = A[v]A? Replacing T in the aboveformula with I, we get [I(v)]B = [I]BA[v]A, and note that I(v) = v.


Linear Transformations Matrix Representation of Linear Transformations

Matrix Representation Applications
http://find/


40/42

Matrix Representation Applications

The jth column of [I]BA is [I(aj)]B = [aj]B, the jth basis vector in A

expressed in terms of the basis B. [I]BA is called the change of basismatrix. Premultiplying a vector written in terms of coordinates wrt basisA yields the same vector written in terms of coordinates wrt basis B.

Example

In

ls 1 linear algebra basics

Documents