linear algebra primer - artificial...
TRANSCRIPT
Linear Algebra ReviewStanford University
LinearAlgebraPrimer
JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab
10/2/171
Another,veryin-depthlinearalgebrareviewfromCS229isavailablehere:http://cs229.stanford.edu/section/cs229-linalg.pdfAndavideodiscussionoflinearalgebrafromEE263ishere(lectures3and4):https://see.stanford.edu/Course/EE263
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculus
10/2/172
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculus
10/2/173
Vectorsandmatricesarejustcollectionsoforderednumbersthatrepresentsomething:movementsinspace,scalingfactors,pixelbrightness,etc.We’lldefinesomecommonusesandstandardoperationsonthem.
Linear Algebra ReviewStanford University
Vector
• Acolumnvectorwhere
• Arowvectorwhere
denotesthetransposeoperation
10/2/174
Linear Algebra ReviewStanford University
Vector• We’lldefaulttocolumnvectorsinthisclass
• You’llwanttokeeptrackoftheorientationofyourvectorswhenprogramminginpython
• YoucantransposeavectorVinpythonbywritingV.t.(Butinclassmaterials,wewillalways useVT toindicatetranspose,andwewilluseV’tomean“Vprime”)
10/2/175
Linear Algebra ReviewStanford University
Vectorshavetwomainuses
• Vectorscanrepresentanoffsetin2Dor3Dspace
• Pointsarejustvectorsfromtheorigin
10/2/176
• Data(pixels,gradientsatanimagekeypoint,etc)canalsobetreatedasavector
• Suchvectorsdon’thaveageometricinterpretation,butcalculationslike“distance”canstillhavevalue
Linear Algebra ReviewStanford University
Matrix
• Amatrixisanarrayofnumberswithsizeby,i.e.mrowsandncolumns.
• If,wesaythatissquare.
10/2/177
Linear Algebra ReviewStanford University
Images
10/2/178
• Pythonrepresentsanimageasamatrixofpixelbrightnesses
• Notethattheupperleftcorneris[y,x]=(0,0)
=
Linear Algebra ReviewStanford University
ColorImages• Grayscaleimageshaveonenumberperpixel,andarestoredasanm× nmatrix.
• Colorimageshave3numbersperpixel– red,green,andbluebrightnesses (RGB)
• Storedasanm× n× 3matrix
10/2/179
=
Linear Algebra ReviewStanford University
BasicMatrixOperations• Wewilldiscuss:– Addition– Scaling– Dotproduct–Multiplication– Transpose– Inverse/pseudoinverse– Determinant/trace
10/2/1710
Linear Algebra ReviewStanford University
MatrixOperations• Addition
– Canonlyaddamatrixwithmatchingdimensions,orascalar.
• Scaling
10/2/1711
Linear Algebra ReviewStanford University
• Norm• Moreformally,anormisanyfunctionthatsatisfies4properties:
• Non-negativity: Forall• Definiteness: f(x)=0ifandonlyifx=0.• Homogeneity: Forall• Triangleinequality: Forall
10/2/1712
Vectors
Linear Algebra ReviewStanford University
• ExampleNorms
• Generalnorms:
10/2/1713
MatrixOperations
Linear Algebra ReviewStanford University
MatrixOperations• Innerproduct(dotproduct)ofvectors–Multiplycorrespondingentriesoftwovectorsandadduptheresult
– x·y isalso|x||y|Cos(theanglebetweenxandy)
10/2/1714
Linear Algebra ReviewStanford University
MatrixOperations
• Innerproduct(dotproduct)ofvectors– IfBisaunitvector,thenA·BgivesthelengthofAwhichliesinthedirectionofB
10/2/1715
Linear Algebra ReviewStanford University
MatrixOperations• Theproductoftwomatrices
10/2/1716
Linear Algebra ReviewStanford University
MatrixOperations• Multiplication
• TheproductABis:
• Eachentryintheresultis(thatrowofA)dotproductwith(thatcolumnofB)
• Manyuses,whichwillbecoveredlater
10/2/1717
Linear Algebra ReviewStanford University
MatrixOperations• Multiplicationexample:
10/2/1718
– Eachentryofthematrixproductismadebytakingthedotproductofthecorrespondingrowintheleftmatrix,withthecorrespondingcolumnintherightone.
Linear Algebra ReviewStanford University
MatrixOperations• Theproductoftwomatrices
10/2/1719
Linear Algebra ReviewStanford University
MatrixOperations
• Powers– Byconvention,wecanrefertothematrixproductAAasA2,andAAAasA3,etc.
– Obviouslyonlysquarematricescanbemultipliedthatway
10/2/1720
Linear Algebra ReviewStanford University
MatrixOperations• Transpose– flipmatrix,sorow1becomescolumn1
• Ausefulidentity:
10/2/1721
Linear Algebra ReviewStanford University
• Determinant– returnsascalar– Representsarea(orvolume)oftheparallelogramdescribedbythevectorsintherowsofthematrix
– For,– Properties:
10/2/1722
MatrixOperations
Linear Algebra ReviewStanford University
• Trace
– Invarianttoalotoftransformations,soit’susedsometimesinproofs.(Rarelyinthisclassthough.)
– Properties:
10/2/1723
MatrixOperations
Linear Algebra ReviewStanford University
• VectorNorms
• Matrixnorms:Normscanalsobedefinedformatrices,suchastheFrobenius norm:
10/2/1724
MatrixOperations
Linear Algebra ReviewStanford University
SpecialMatrices• IdentitymatrixI– Squarematrix,1’salongdiagonal,0’selsewhere
– I · [anothermatrix]=[thatmatrix]
• Diagonalmatrix– Squarematrixwithnumbersalongdiagonal,0’selsewhere
– Adiagonal· [anothermatrix]scalestherowsofthatmatrix
10/2/1725
Linear Algebra ReviewStanford University
SpecialMatrices
• Symmetricmatrix
• Skew-symmetricmatrix
10/2/1726
2
40 �2 �52 0 �75 7 0
3
5
Linear Algebra ReviewStanford University
LinearAlgebraPrimer
JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab
10/2/1727
Another,veryin-depthlinearalgebrareviewfromCS229isavailablehere:http://cs229.stanford.edu/section/cs229-linalg.pdfAndavideodiscussionoflinearalgebrafromEE263ishere(lectures3and4):https://see.stanford.edu/Course/EE263
Linear Algebra ReviewStanford University
Announcements– part1
• HW0submittedlastnight• HW1isduenextMonday• HW2willbereleasedtonight• ClassnotesfromlastThursdayduebeforeclassinexactly48hours
10/2/1728
Linear Algebra ReviewStanford University
Announcements– part2
• Futurehomeworkassignmentswillbereleasedviagithub–WillallowyoutokeeptrackofchangesIFtheyhappen.
• SubmissionsforHW1onwardswillbedoneallthroughgradescope.– NOMORECORNSUBMISSIONS– Youwillhaveseparatesubmissionsfortheipythonpdfandthepythoncode.
10/2/1729
Linear Algebra ReviewStanford University
Recap- Vector
• Acolumnvectorwhere
• Arowvectorwhere
denotesthetransposeoperation
10/2/1730
Linear Algebra ReviewStanford University
Recap- Matrix
• Amatrixisanarrayofnumberswithsizeby,i.e.mrowsandncolumns.
• If,wesaythatissquare.
10/2/1731
Linear Algebra ReviewStanford University
Recap- ColorImages• Grayscaleimageshaveonenumberperpixel,andarestoredasanm× nmatrix.
• Colorimageshave3numbersperpixel– red,green,andbluebrightnesses (RGB)
• Storedasanm× n× 3matrix
10/2/1732
=
Linear Algebra ReviewStanford University
• Norm• Moreformally,anormisanyfunctionthatsatisfies4properties:
• Non-negativity: Forall• Definiteness: f(x)=0ifandonlyifx=0.• Homogeneity: Forall• Triangleinequality: Forall
10/2/1733
Recap- Vectors
Linear Algebra ReviewStanford University
Recap– projection
• Innerproduct(dotproduct)ofvectors– IfBisaunitvector,thenA·BgivesthelengthofAwhichliesinthedirectionofB
10/2/1734
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculus
10/2/1735
Matrixmultiplicationcanbeusedtotransformvectors.Amatrixusedinthiswayiscalledatransformationmatrix.
Linear Algebra ReviewStanford University
Transformation
• Matricescanbeusedtotransformvectorsinusefulways,throughmultiplication:x’=Ax
• Simplestisscaling:
(Verifytoyourselfthatthematrixmultiplicationworksoutthisway)
10/2/1736
Linear Algebra ReviewStanford University
Rotation• Howcanyouconvertavectorrepresentedinframe“0”toanew,rotatedcoordinateframe“1”?
10/2/1737
Linear Algebra ReviewStanford University
Rotation• Howcanyouconvertavectorrepresentedinframe“0”toanew,rotatedcoordinateframe“1”?
• Rememberwhatavectoris:[componentindirectionoftheframe’sxaxis,componentindirectionofyaxis]
10/2/1738
Linear Algebra ReviewStanford University
Rotation• Sotorotateitwemustproducethisvector:[componentindirectionofnew xaxis,componentindirectionofnew yaxis]
• Wecandothiseasilywithdotproducts!• Newxcoordinateis[originalvector]dot [thenewxaxis]• Newycoordinateis[originalvector]dot [thenewyaxis]
10/2/1739
Linear Algebra ReviewStanford University
Rotation• Insight:thisiswhathappensinamatrix*vectormultiplication– Resultxcoordinateis:[originalvector]dot [matrixrow1]
– Somatrixmultiplicationcanrotateavectorp:
10/2/1740
Linear Algebra ReviewStanford University
Rotation• Supposeweexpressapointinthenewcoordinatesystemwhichisrotatedleft
• Ifweplottheresultintheoriginal coordinatesystem,wehaverotatedthepointright
10/2/1741
– Thus,rotationmatricescanbeusedtorotatevectors.We’llusuallythinkoftheminthatsense-- asoperatorstorotatevectors
Linear Algebra ReviewStanford University
2DRotationMatrixFormulaCounter-clockwise rotation by an angle q
úû
ùêë
éúû
ùêë
é -=ú
û
ùêë
éyx
yx
qqqq
cossinsincos
''
P
x
y’P’
q
x’
y
PRP'=
yθsinxθcos'x -=xθsinyθcos'y +=
10/2/1742
Linear Algebra ReviewStanford University
TransformationMatrices• Multipletransformationmatricescanbeusedtotransformapoint:p’=R2R1Sp
10/2/1743
Linear Algebra ReviewStanford University
TransformationMatrices• Multipletransformationmatricescanbeusedtotransformapoint:p’=R2R1Sp
• Theeffectofthisistoapplytheirtransformationsoneaftertheother,fromrighttoleft.
• Intheexampleabove,theresultis(R2(R1(Sp)))
10/2/1744
Linear Algebra ReviewStanford University
TransformationMatrices• Multipletransformationmatricescanbeusedtotransformapoint:p’=R2R1Sp
• Theeffectofthisistoapplytheirtransformationsoneaftertheother,fromrighttoleft.
• Intheexampleabove,theresultis(R2(R1(Sp)))
• Theresultisexactlythesameifwemultiplythematricesfirst,toformasingletransformationmatrix:p’=(R2R1S)p
10/2/1745
Linear Algebra ReviewStanford University
Homogeneoussystem
• Ingeneral,amatrixmultiplicationletsuslinearlycombinecomponentsofavector
– Thisissufficientforscale,rotate,skewtransformations.
– Butnotice,wecan’taddaconstant!L
10/2/1746
Linear Algebra ReviewStanford University
Homogeneoussystem
– The(somewhathacky)solution?Sticka“1”attheendofeveryvector:
– Nowwecanrotate,scale,andskewlikebefore,ANDtranslate (notehowthemultiplicationworksout,above)
– Thisiscalled“homogeneouscoordinates”
10/2/1747
Linear Algebra ReviewStanford University
Homogeneoussystem– Inhomogeneouscoordinates,themultiplicationworksoutsotherightmostcolumnofthematrixisavectorthatgetsadded.
– Generally,ahomogeneoustransformationmatrixwillhaveabottomrowof[001],sothattheresulthasa“1”atthebottomtoo.
10/2/1748
Linear Algebra ReviewStanford University
Homogeneoussystem• Onemorethingwemightwant:todividetheresultbysomething– Forexample,wemaywanttodividebyacoordinate,tomakethingsscaledownastheygetfartherawayinacameraimage
– Matrixmultiplicationcan’tactuallydivide– So,byconvention,inhomogeneouscoordinates,we’lldividetheresultbyitslastcoordinateafterdoingamatrixmultiplication
10/2/1749
Linear Algebra ReviewStanford University
2DTranslation
t
P
P’
10/2/1750
Linear Algebra ReviewStanford University 10/2/1751
2DTranslationusingHomogeneousCoordinates
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
Linear Algebra ReviewStanford University 10/2/1752
2DTranslationusingHomogeneousCoordinates
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
Linear Algebra ReviewStanford University 10/2/1753
2DTranslationusingHomogeneousCoordinates
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
Linear Algebra ReviewStanford University 10/2/1754
2DTranslationusingHomogeneousCoordinates
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
Linear Algebra ReviewStanford University 10/2/1755
2DTranslationusingHomogeneousCoordinates
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
tP
PTPtI
×=×úû
ùêë
é=
10
Linear Algebra ReviewStanford University
Scaling
P
P’
10/2/1756
Linear Algebra ReviewStanford University
ScalingEquation
P
x
y
sx x
P’sy y
)1,,(),(')1,,(),(
ysxsysxsyxyx
yxyx ®=®=
PP
)ys,xs(')y,x( yx=®= PP
10/2/1757
Linear Algebra ReviewStanford University
ScalingEquation
P
x
y
sx x
P’sy y
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é®
11000000
1' y
xs
sysxs
y
x
y
x
P
)1,,(),(')1,,(),(
ysxsysxsyxyx
yxyx ®=®=
PP
)ys,xs(')y,x( yx=®= PP
10/2/1758
Linear Algebra ReviewStanford University
ScalingEquation
P
x
y
sx x
P’sy y
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é®
11000000
1' y
xs
sysxs
y
x
y
x
P
)1,,(),(')1,,(),(
ysxsysxsyxyx
yxyx ®=®=
PP
S
PSP100S
×=×úû
ùêë
é=
'
)ys,xs(')y,x( yx=®= PP
10/2/1759
Linear Algebra ReviewStanford University
P
P’=S∙PP’’=T∙P’
P’’=T·P’=T·(S·P)=T·S·P
Scaling&Translating
P’’
10/2/1760
Linear Algebra ReviewStanford University
Scaling&Translating
P '' = T ⋅S ⋅P =1 0 tx0 1 ty0 0 1
"
#
$$$$
%
&
''''
⋅
sx 0 00 sy 0
0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
=
sx 0 tx0 sy ty0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
sxx + txsyy+ ty1
"
#
$$$$
%
&
''''
= S t0 1
"
#$
%
&'
xy1
"
#
$$$
%
&
'''
A10/2/1761
Linear Algebra ReviewStanford University
Scaling&Translating
P '' = T ⋅S ⋅P =1 0 tx0 1 ty0 0 1
"
#
$$$$
%
&
''''
⋅
sx 0 00 sy 0
0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
=
sx 0 tx0 sy ty0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
sxx + txsyy+ ty1
"
#
$$$$
%
&
''''
= S t0 1
"
#$
%
&'
xy1
"
#
$$$
%
&
'''
10/2/1762
Linear Algebra ReviewStanford University
Translating&Scalingversus Scaling&Translating
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tystxs
1yx
100ts0t0s
1yx
1000s000s
100t10t01
''' yy
xx
yy
xx
y
x
y
x
PSTP
10/2/1763
Linear Algebra ReviewStanford University
Translating&Scaling!=Scaling&Translating
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tsystsxs
1yx
100tss0ts0s
1yx
100t10t01
1000s000s
'''
yyy
xxx
yyy
xxx
y
x
y
x
PTSP
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tystxs
1yx
100ts0t0s
1yx
1000s000s
100t10t01
''' yy
xx
yy
xx
y
x
y
x
PSTP
10/2/1764
Linear Algebra ReviewStanford University
Translating&Scaling!=Scaling&Translating
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tsystsxs
1yx
100tss0ts0s
1yx
100t10t01
1000s000s
'''
yyy
xxx
yyy
xxx
y
x
y
x
PTSP
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tystxs
1yx
100ts0t0s
1yx
1000s000s
100t10t01
''' yy
xx
yy
xx
y
x
y
x
PSTP
10/2/1765
Linear Algebra ReviewStanford University
Rotation
P
P’
10/2/1766
Linear Algebra ReviewStanford University
RotationEquationsCounter-clockwise rotation by an angle q
úû
ùêë
éúû
ùêë
é -=ú
û
ùêë
éyx
yx
qqqq
cossinsincos
''
P
x
y’P’
q
x’
y
PRP'=
yθsinxθcos'x -=xθsinyθcos'y +=
10/2/1767
Linear Algebra ReviewStanford University
RotationMatrixProperties
A2Drotationmatrixis2x2
1)det( ==×=×
RIRRRR TT
úû
ùêë
éúû
ùêë
é -=ú
û
ùêë
éyx
yx
qqqq
cossinsincos
''
Note: R belongs to the category of normal matrices and satisfies many interesting properties:
10/2/1768
Linear Algebra ReviewStanford University
RotationMatrixProperties• Transposeofarotationmatrixproducesarotationintheoppositedirection
• Therowsofarotationmatrixarealwaysmutuallyperpendicular(a.k.a.orthogonal)unitvectors– (andsoareitscolumns)
10/2/1769
1)det( ==×=×
RIRRRR TT
Linear Algebra ReviewStanford University
Scaling+Rotation+TranslationP’=(TRS)P
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é -
úúú
û
ù
êêê
ë
é=×××=
1yx
1000s000s
1000θcossinθ0inθsθcos
100t10t01
R' y
x
y
x
PSTP
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é -=
1yx
1000s000s
100tθcossinθtinθsθcos
y
x
y
x
úúú
û
ù
êêê
ë
é
úû
ùêë
é=
úúú
û
ù
êêê
ë
é
úû
ùêë
éúû
ùêë
é=
110
1100
10yx
tSRyx
StR
10/2/1770
Thisistheformofthegeneral-purposetransformationmatrix
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculate
10/2/1771
Theinverseofatransformationmatrixreversesitseffect
Linear Algebra ReviewStanford University
• GivenamatrixA,itsinverseA-1 isamatrixsuchthatAA-1=A-1A=I
• E.g.
• Inversedoesnotalwaysexist.IfA-1 exists,A isinvertible ornon-singular.Otherwise,it’ssingular.
• Usefulidentities,formatricesthatareinvertible:
10/2/1772
Inverse
Linear Algebra ReviewStanford University
• Pseudoinverse– SayyouhavethematrixequationAX=B,whereAandBareknown,andyouwanttosolveforX
10/2/1773
MatrixOperations
Linear Algebra ReviewStanford University
• Pseudoinverse– SayyouhavethematrixequationAX=B,whereAandBareknown,andyouwanttosolveforX
– Youcouldcalculatetheinverseandpre-multiplybyit:A-1AX=A-1B→X=A-1B
10/2/1774
MatrixOperations
Linear Algebra ReviewStanford University
• Pseudoinverse– SayyouhavethematrixequationAX=B,whereAandBareknown,andyouwanttosolveforX
– Youcouldcalculatetheinverseandpre-multiplybyit:A-1AX=A-1B→X=A-1B
– Pythoncommandwouldbenp.linalg.inv(A)*B– Butcalculatingtheinverseforlargematricesoftenbringsproblemswithcomputerfloating-pointresolution(becauseitinvolvesworkingwithverysmallandverylargenumberstogether).
– Or,yourmatrixmightnotevenhaveaninverse.
10/2/1775
MatrixOperations
Linear Algebra ReviewStanford University
• Pseudoinverse– Fortunately,thereareworkaroundstosolveAX=Binthesesituations.Andpythoncandothem!
– Insteadoftakinganinverse,directlyaskpythontosolveforXinAX=B,bytypingnp.linalg.solve(A,B)
– Pythonwilltryseveralappropriatenumericalmethods(includingthepseudoinverseiftheinversedoesn’texist)
– PythonwillreturnthevalueofXwhichsolvestheequation• Ifthereisnoexactsolution,itwillreturntheclosestone• Iftherearemanysolutions,itwillreturnthesmallestone
10/2/1776
MatrixOperations
Linear Algebra ReviewStanford University
• Pythonexample:
10/2/1777
MatrixOperations
>> import numpy as np>> x = np.linalg.solve(A,B)x =
1.0000-0.5000
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculate
10/2/1778
Therankofatransformationmatrixtellsyouhowmanydimensionsittransformsavectorto.
Linear Algebra ReviewStanford University
Linearindependence• Supposewehaveasetofvectorsv1, …, vn• Ifwecanexpressv1 asalinearcombinationoftheothervectorsv2…vn,thenv1 islinearlydependent ontheothervectors.– Thedirectionv1 canbeexpressedasacombinationofthedirectionsv2…vn. (E.g.v1 =.7 v2-.7 v4)
10/2/1779
Linear Algebra ReviewStanford University
Linearindependence• Supposewehaveasetofvectorsv1, …, vn• Ifwecanexpressv1 asalinearcombinationoftheothervectorsv2…vn,thenv1 islinearlydependent ontheothervectors.– Thedirectionv1 canbeexpressedasacombinationofthedirectionsv2…vn.(E.g.v1 =.7 v2-.7 v4)
• Ifnovectorislinearlydependentontherestoftheset,thesetislinearlyindependent.– Commoncase:asetofvectorsv1, …, vn isalwayslinearlyindependentifeachvectorisperpendiculartoeveryothervector(andnon-zero)
10/2/1780
Linear Algebra ReviewStanford University
LinearindependenceNotlinearlyindependent
10/2/1781
Linearlyindependentset
Linear Algebra ReviewStanford University
Matrixrank
• Column/rowrank
– Columnrankalwaysequalsrowrank
• Matrixrank
10/2/1782
Linear Algebra ReviewStanford University
Matrixrank• Fortransformationmatrices,theranktellsyouthedimensionsoftheoutput
• E.g.ifrankofA is1,thenthetransformation
p’=Apmapspointsontoaline.
• Here’samatrixwithrank1:
10/2/1783
Allpointsgetmappedtotheliney=2x
Linear Algebra ReviewStanford University
Matrixrank• Ifanm xmmatrixisrankm,wesayit’s“fullrank”– Mapsanm x1vectoruniquelytoanotherm x1vector– Aninversematrixcanbefound
• Ifrank<m,wesayit’s“singular”– Atleastonedimensionisgettingcollapsed.Nowaytolookattheresultandtellwhattheinputwas
– Inversedoesnotexist
• Inversealsodoesn’texistfornon-squarematrices
10/2/1784
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors(SVD)• MatrixCalculus
10/2/1785
Linear Algebra ReviewStanford University
EigenvectorandEigenvalue
• Aneigenvector x ofalineartransformation A isanon-zerovectorthat,when A isappliedtoit,doesnotchangedirection.
10/2/1786
Linear Algebra ReviewStanford University
EigenvectorandEigenvalue
• Aneigenvector x ofalineartransformation A isanon-zerovectorthat,when A isappliedtoit,doesnotchangedirection.
• Applying A totheeigenvectoronlyscalestheeigenvectorbythescalarvalue λ,calledaneigenvalue.
10/2/1787
Linear Algebra ReviewStanford University
EigenvectorandEigenvalue
• WewanttofindalltheeigenvaluesofA:
• Whichcanwewrittenas:
• Therefore:
10/2/1788
Linear Algebra ReviewStanford University
EigenvectorandEigenvalue
• Wecansolveforeigenvaluesbysolving:
• Sincewearelookingfornon-zerox,wecaninsteadsolvetheaboveequationas:
10/2/1789
Linear Algebra ReviewStanford University
Properties
• ThetraceofaAisequaltothesumofitseigenvalues:
10/2/1790
Linear Algebra ReviewStanford University
Properties• ThetraceofaAisequaltothesumofitseigenvalues:
• ThedeterminantofAisequaltotheproductofitseigenvalues
10/2/1791
Linear Algebra ReviewStanford University
Properties• ThetraceofaAisequaltothesumofitseigenvalues:
• ThedeterminantofAisequaltotheproductofitseigenvalues
• TherankofAisequaltothenumberofnon-zeroeigenvaluesofA.
10/2/1792
Linear Algebra ReviewStanford University
Properties• ThetraceofaAisequaltothesumofitseigenvalues:
• ThedeterminantofAisequaltotheproductofitseigenvalues
• TherankofAisequaltothenumberofnon-zeroeigenvaluesofA.
• TheeigenvaluesofadiagonalmatrixD=diag(d1,...dn)arejustthediagonalentriesd1,...dn
10/2/1793
Linear Algebra ReviewStanford University
Spectraltheory
• Wecallaneigenvalueλ andanassociatedeigenvectoraneigenpair.
• Thespaceofvectorswhere(A−λI)=0isoftencalledtheeigenspace ofAassociatedwiththeeigenvalueλ.
• ThesetofalleigenvaluesofAiscalleditsspectrum:
10/2/1794
Linear Algebra ReviewStanford University
Spectraltheory
• Themagnitudeofthelargesteigenvalue(inmagnitude)iscalledthespectralradius
–WhereCisthespaceofalleigenvaluesofA
10/2/1795
Linear Algebra ReviewStanford University
Spectraltheory
• Thespectralradiusisboundedbyinfinitynormofamatrix:
• Proof:Turntoapartnerandprovethis!
10/2/1796
Linear Algebra ReviewStanford University
Spectraltheory
• Thespectralradiusisboundedbyinfinitynormofamatrix:
• Proof:Letλ andvbeaneigenpair ofA:
10/2/1797
Linear Algebra ReviewStanford University
Diagonalization
• Ann× nmatrixAisdiagonalizableifithasnlinearlyindependenteigenvectors.
• Mostsquarematrices(inasensethatcanbemademathematicallyrigorous)arediagonalizable:– Normalmatricesarediagonalizable– Matriceswithndistincteigenvaluesarediagonalizable
Lemma:Eigenvectorsassociatedwithdistincteigenvaluesarelinearlyindependent.
10/2/1798
Linear Algebra ReviewStanford University
Diagonalization
• Ann× nmatrixAisdiagonalizableifithasnlinearlyindependenteigenvectors.
• Mostsquarematricesarediagonalizable:– Normalmatricesarediagonalizable–Matriceswithndistincteigenvaluesarediagonalizable
Lemma:Eigenvectorsassociatedwithdistincteigenvaluesarelinearlyindependent.
10/2/1799
Linear Algebra ReviewStanford University
Diagonalization
• Eigenvalueequation:
–WhereDisadiagonalmatrixoftheeigenvalues
10/2/17100
Linear Algebra ReviewStanford University
Diagonalization
• Eigenvalueequation:
• Assumingallλi’s areunique:
• Rememberthattheinverseofanorthogonalmatrixisjustitstransposeandtheeigenvectorsareorthogonal
10/2/17101
Linear Algebra ReviewStanford University
Symmetricmatrices
• Properties:– ForasymmetricmatrixA,alltheeigenvaluesarereal.
– TheeigenvectorsofAareorthonormal.
10/2/17102
Linear Algebra ReviewStanford University
Symmetricmatrices
• Therefore:
– where• So,ifwewantedtofindthevectorxthat:
10/2/17103
Linear Algebra ReviewStanford University
Symmetricmatrices
• Therefore:
– where• So,ifwewantedtofindthevectorxthat:
– Isthesameasfindingtheeigenvectorthatcorrespondstothelargesteigenvalue.
10/2/17104
Linear Algebra ReviewStanford University
SomeapplicationsofEigenvalues
• PageRank• Schrodinger’sequation• PCA
10/2/17105
Linear Algebra ReviewStanford University
Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors(SVD)• MatrixCalculus
10/2/17106
Linear Algebra ReviewStanford University
MatrixCalculus– TheGradient
• LetafunctiontakeasinputamatrixAofsizem× nandreturnsarealvalue.
• Thenthegradient off:
10/2/17107
Linear Algebra ReviewStanford University
MatrixCalculus – TheGradient
• Everyentryinthematrixis:• thesizeof∇Af(A)isalwaysthesameasthesizeofA.SoifAisjustavectorx:
10/2/17108
Linear Algebra ReviewStanford University
Exercise
• Example:
• Find:
10/2/17109
Linear Algebra ReviewStanford University
Exercise
• Example:
• Fromthiswecanconcludethat:
10/2/17110
Linear Algebra ReviewStanford University
MatrixCalculus – TheGradient
• Properties
10/2/17111
Linear Algebra ReviewStanford University
MatrixCalculus – TheHessian
• TheHessianmatrixwithrespecttox,writtenorsimplyasHisthen× nmatrixof
partialderivatives
10/2/17112
Linear Algebra ReviewStanford University
MatrixCalculus – TheHessian
• Eachentrycanbewrittenas:
• Exercise:WhyistheHessianalwayssymmetric?
10/2/17113
Linear Algebra ReviewStanford University
MatrixCalculus – TheHessian
• Eachentrycanbewrittenas:
• TheHessianisalwayssymmetric,because
• ThisisknownasSchwarz'stheorem:Theorderofpartialderivativesdon’tmatteraslongasthesecondderivativeexistsandiscontinuous.
10/2/17114
Linear Algebra ReviewStanford University
MatrixCalculus– TheHessian
• Notethatthehessianisnotthegradientofwholegradientofavector(thisisnotdefined).Itisactuallythegradientofeveryentry ofthegradientofthevector.
10/2/17115
Linear Algebra ReviewStanford University
MatrixCalculus– TheHessian
• Eg,thefirstcolumnisthegradientof
10/2/17116
Linear Algebra ReviewStanford University
Exercise
• Example:
10/2/17117
Linear Algebra ReviewStanford University
Exercise
10/2/17118
Linear Algebra ReviewStanford University
Exercise
10/2/17119
Dividethesummationinto3partsdependingonwhether:• i ==kor• j==k
Linear Algebra ReviewStanford University
Exercise
10/2/17120
Linear Algebra ReviewStanford University
Exercise
10/2/17121
Linear Algebra ReviewStanford University
Exercise
10/2/17122
Linear Algebra ReviewStanford University
Exercise
10/2/17123
Linear Algebra ReviewStanford University
Exercise
10/2/17124
Linear Algebra ReviewStanford University
Exercise
10/2/17125
Linear Algebra ReviewStanford University
Exercise
10/2/17126
Linear Algebra ReviewStanford University
Exercise
10/2/17127
Linear Algebra ReviewStanford University
Exercise
10/2/17128
Linear Algebra ReviewStanford University
Whatwehavelearned• Vectorsandmatrices– BasicMatrixOperations– SpecialMatrices
• TransformationMatrices– Homogeneouscoordinates– Translation
• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculate
10/2/17129