6

1

Linear Transformations

6

2

Hopfield Network Questions

6

3

Hopfield Network Questions

• The network output is repeatedly multiplied by the weight matrix W.

• What is the effect of this repeated operation?• Will the output converge, go to infinity, oscillate?• In this chapter we want to investigate matrix multiplication,

which represents a general linear transformation.

6

4

Linear Transformations

A transformation consists of three parts:1. A set of elements X = {x i}, called the domain,

2. A set of elements Y = {y i}, called the range, and

3. A rule relating each x i Î X to an element y i Î Y.

A transformation is linear if:1. For all x 1, x 2 Î X, A (x 1 + x 2 ) = A (x 1) + A (x 2 ),

2. For all x Î X, a Î Â , A (a x ) = a A (x ) .

6

5

Example - Rotation

Is rotation linear?

θ

6

6

Example - Rotation

Is rotation linear?

1.

2.

6

7

Example - Translation

Is translation linear?

𝒜 (𝒳)=𝒳+𝒯𝒳𝒳+𝒯

𝒯𝒜 (𝒳1+𝒳2 )=𝒳1+𝒳2+𝒯+

𝒜 (𝒳1 )+𝒜 (𝒳2 )=𝒳1+𝒳2+2𝒯

6

8

Matrix Representation - (1)

Any linear transformation between two finite-dimensional vector spaces can be represented by matrix multiplication.

Let {v1, v2, ..., vn} be a basis for X, and let {u1, u2, ..., um} be a

basis for Y.

Let :X®Y

𝜒=∑𝑖=1

𝑛

𝑥𝑖𝑣 𝑖

𝒜 (𝒳)=𝒴𝒴=∑

𝑖=1

𝑚

𝑦 𝑖𝑢𝑖

𝒜(∑𝑗=1

𝑛

𝑥 𝑗 𝑣 𝑗)=∑𝑖=1

𝑚

𝑦𝑖𝑢𝑖

6

9

Matrix Representation - (2)

Since is a linear operator,

Since the ui are a basis for Y,

(The coefficients a i j will make up the matrix representation of the transformation.)

∑𝑗=1

𝑛

𝑥 𝑗𝒜(𝑣𝑗¿)=∑𝑖=1

𝑚

𝑦 𝑖𝑢𝑖¿

𝒜 (𝑣 𝑗 )=∑𝑖=1

𝑚

𝑎𝑖𝑗 𝑢𝑖

∑𝑗=1

𝑛

𝑥 𝑗∑𝑖=1

𝑚

𝑎 𝑖𝑗𝑢𝑖=∑𝑖=1

𝑚

𝑦 𝑖𝑢𝑖 ∑𝑖=1

𝑚

𝑢𝑖∑𝑗=1

𝑛

𝑎𝑖𝑗 𝑥 𝑗=∑𝑖=1

𝑚

𝑦 𝑖𝑢𝑖

6

10

Matrix Representation - (3)

Because the u i are independent,

a 11 a 12 ¼ a 1 n

a 21 a 22 ¼ a 2 n

a m 1 a m 2 ¼ a m n

x 1

x 2

x n

y 1

y 2

y m

=This is equivalent to matrix multiplication.

∑𝑖=1

𝑚

𝑢𝑖∑𝑗=1

𝑛

𝑎𝑖𝑗 𝑥 𝑗=∑𝑖=1

𝑚

𝑦 𝑖𝑢𝑖 ∑𝑖=1

𝑚

𝑢𝑖 (∑𝑗=1

𝑛

𝑎𝑖𝑗 𝑥 𝑗− 𝑦𝑖)=0

∑𝑗=1

𝑛

𝑎𝑖𝑗 𝑥 𝑗=𝑦 𝑖

6

11

Summary

• A linear transformation can be represented by matrix multiplication.

• To find the matrix which represents the transformation we must transform each basis vector for the domain and then expand the result in terms of the basis vectors of the range.

Each of these equations gives us one column of the matrix.

𝒜 (𝑣 𝑗 )=∑𝑖=1

𝑚

𝑎𝑖𝑗 𝑢𝑖

6

12

Example - (1)

To find the matrix we need to transform each of the basis vectors.

We will use the standard basis vectors for both the domain and the range.

𝒜 (𝑣 𝑗 )=∑𝑖=1

𝑚

𝑎𝑖𝑗 𝑢𝑖

𝒜 (𝑠 𝑗 )=∑𝑖=1

2

𝑎𝑖𝑗 𝑠𝑖=𝑎 1 𝑗 𝑠1+𝑎 2 𝑗 𝑠2

6

13

Example - (1)

We begin with s1:

This gives us the first column of the matrix.

If we rotate s1 counterclockwise by the angle we obtain

𝒜 (𝑠1 )=𝑐𝑜𝑠𝜃 𝑠1+𝑠𝑖𝑛𝜃𝑠2=∑𝑖=1

2

𝑎𝑖1 𝑠𝑖=𝑎11 𝑠1+𝑎21 𝑠2

6

14

Example - (1)

Next, we skew s2:

This gives us the second column of the matrix.

If we rotate s2 counterclockwise by the angle we obtain

𝒜 (𝑠1 )=−𝑠𝑖𝑛𝜃𝑠1+𝑐𝑜𝑠𝜃𝑠 2=∑𝑖=1

2

𝑎𝑖1𝑠𝑖=𝑎12 𝑠1+𝑎22 𝑠2

6

15

Example - (1)

The matrix of the transformation is:

𝐀=[c osθ − sinθsin θ cosθ ]

=

==Az

6

16

Change of Basis

Consider the linear transformation A:X®Y. Let {v1, v2, ..., vn}

be a basis for X, and let {u1, u2, ..., um} be a basis for Y.

The matrix representation is:

¼¼

a11

a12

¼ a1 n

a21

a22

¼ a2 n

am 1

am 2

¼ am n

x1

x2

xn

y1

y2

ym

=

¼ ¼ ¼

𝜒=∑𝑖=1

𝑛

𝑥𝑖𝑣 𝑖 𝒴=∑𝑖=1

𝑚

𝑦 𝑖𝑢𝑖

𝒜 (𝒳)=𝒴

𝐀𝐱=𝐲

6

17

New Basis Sets

Now let’s consider different basis sets. Let {t1, t2, ..., tn} be a

basis for X, and let {w1, w2, ..., wm} be a basis for Y.

The new matrix representation is:

¼¼¼

a '11

a '12

¼ a '1 n

a '21

a '22

¼ a '2 n

a 'm 1

a 'm 2

¼ a 'm n

x '1

x '2

x 'n

y '1

y '2

y 'm

=

¼ ¼

𝜒=∑𝑖=1

𝑛

𝑥 ′ 𝑖 𝑡 𝑖 𝒴=∑𝑖=1

𝑚

𝑦 ′ 𝑖𝑤 𝑖

𝐀 ′ 𝐱 ′=𝐲 ′

𝒜 (𝒳)=𝒴

6

18

How are A and A' related?

Expand ti in terms of the original basis vectors for X.

Expand wi in terms of the original basis vectors for Y.

𝑡 𝑖=∑𝑗=1

𝑛

𝑡 𝑗𝑖𝑣 𝑗

𝑤 𝑖=∑𝑗=1

𝑚

𝑤 𝑗𝑖𝑢 𝑗

𝐭 𝑖=[𝑡 1 𝑖𝑡 2 𝑖⋮𝑡 𝑛𝑖

]

𝐰 𝑖=[ 𝑤1 𝑖𝑤2 𝑖⋮

𝑤𝑚𝑖]

6

19

How are A and A' related?

SimilarityTransform

𝐁 t=[𝐭 1 𝐭 2 ⋯ 𝐭 n ]𝐁w=[𝐰1 𝐰 2 ⋯ 𝐰m ]

𝐱=𝑥′ 1𝐭 1+𝑥 ′ 2𝐭 2+⋯𝑥 ′ n𝐭 n=𝐁 t 𝐱 ′

𝐲=𝑦 ′ 1𝐰1+𝑦 ′ 2𝐰 2+⋯ y ′ m𝐰m=𝐁w 𝐲 ′

𝐀𝐱=𝐲 𝐀𝐁 t 𝐱 ′=𝐁w 𝐲 ′

[𝐁w −1𝐀𝐁t ]𝐱′=𝐲 ′

𝐀 ′ 𝐱′=𝐲 ′𝐀′=[𝐁w− 1𝐀𝐁 t ]

6

20

Example(2)

Consider a transformation A :Ｒ 3→Ｒ 2 whose matrix representation relative to the standard basis sets is

Find the matrix for this transformation relative to the basis sets :

6

21

Example(2)

Step 1 : form the matrices

Step 2 : Use

6

22

Eigenvalues and Eigenvectors

Let A:X®X be a linear transformation. Those vectorsz Î X, which are not equal to zero, and those scalarsl which satisfy

A(z) = l z

are called eigenvectors and eigenvalues, respectively.

6

23

Computing the Eigenvalues

𝐀𝐳=𝜆𝐳 0 0

𝐴=[𝑐𝑜𝑠 𝜃 −𝑠𝑖𝑛𝜃𝑠𝑖𝑛𝜃 𝑐𝑜𝑠𝜃 ] =0

𝜆2 −2𝜆𝑐𝑜𝑠 𝜃+cos 2θ+sin 2θ=𝜆2−2𝜆𝑐𝑜𝑠𝜃+1=0

𝜆1=𝑐𝑜𝑠𝜃+ 𝑖𝑠𝑖𝑛𝜃

𝜆2=𝑐𝑜𝑠𝜃− 𝑖𝑠𝑖𝑛𝜃

[−𝑖𝑠𝑖𝑛𝜃 −𝑠𝑖𝑛𝜃𝑠𝑖𝑛𝜃 −𝑖𝑠𝑖𝑛𝜃 ] [𝑧 11

𝑧 21]=[00]𝐳1=[𝑧 11

𝑧 21]=[1𝑖 ][𝑖𝑠𝑖𝑛𝜃 −𝑠𝑖𝑛𝜃𝑠𝑖𝑛𝜃 𝑖𝑠 𝑖𝑛𝜃 ] [𝑧 11

𝑧 21]=[00]𝐳 2=[𝑧 12

𝑧 2 2]=[ 𝑖1]

6

24

Diagonalization

Perform a change of basis (similarity transformation) usingthe eigenvectors as the basis vectors. If the eigenvalues aredistinct, the new matrix will be diagonal.

B z1 z2 zn=

z1 z2 zn{ , } Eigenvectors1 2 n{ , } Eigenvalues

n

¼¼

B1–AB[ ]

l 1 0 ¼ 0

0 l 2 ¼ 0

0 0 ¼ l

=¼

6

25

Example(3)

Diagonal Form:

𝐀=[1 11 1] |[1− 𝜆 1

1 1− 𝜆]|=0

=(-2)=0 𝜆1=0𝜆2=2

[1− 𝜆 11 1− 𝜆]𝐳=[00 ]

𝜆1=0

𝜆2=2

[1 11 1] [𝑧 11

𝑧 21]=[00]

[−1 11 −1] [𝑧 12

𝑧 22 ]=[00 ]

𝐳1=[𝑧 11𝑧 21]=[ 1

−1]𝐳 2=[𝑧 12

𝑧 22]=[11]

𝐀′=[𝐁−1𝐀𝐁 ]=[ 12

−12

12

12

] [1 11 1][ 1 1

− 1 1]=[0 00 2]

6

26

Example - (4)

Take the skewing problem described previously, and find thenew matrix representation using the basis set {s1, s2}.

t 1 0.5s1 s2+=

t2 s– 1 s2+=

Bt t1 t20.5 1–

1 1= = Bw Bt

0.5 1–

1 1= =

t10.5

1=

t21–

1=

(Same basis fordomain and range.)

6

27

Example - (4)

A' Bw1– AB t 2 3 2 3

2– 3 1 31 tan

0 1

0.5 1–

1 1= =

A' 2 3 tan 1+ 2 3 tan2– 3 tan 2– 3 tan 1+

=

A' 5 3 2 32– 3 1 3

=

For q = 45°:

A 1 1

0 1=

6

28

Example - (4)

Try a test vector: x 0.5

1= x' 1

0=

y' A'x' 5 3 2 32– 3 1 3

1

0

5 32– 3

= = =y Ax 1 1

0 1

0.5

1

1.5

1= = =

y' B 1– y 0.5 1–

1 1

1–1.5

1

2 3 2 32– 3 1 3

1.5

1

5 32 3–

= = = =

Check using reciprocal basis vectors: