session 9 linear algebra and rec

7/27/2019 Session 9 linear algebra and rec

1/18

Linear algebra and regression


2/18

Solving linear equations

The simplest system of linear equations hastwo equations and two variables, for example:

This system can be represented using matrices

and vectors in the form Ax = b


3/18


Solving this system in Matlab is straightforward:

octave:9> A = [1 -1 ; 3 1]

A =

1 -1

3 1octave:10> b = [-1 ; 9]

b =

-1

9

octave:11> x = A\bx =

2

3


4/18


We can now verify that x1 = 2, x2 = 3 is a solutionby calculating Ax

octave:12> A*x

ans =-1

9

octave:13> A*x-b

ans =0

0


5/18

Exercise

Solve the following linear system:

2x 3y = 3

4x 5y + z = 72x -y -3z = 5


6/18

Determinants


7/18

Determinants

octave:15> det(A)

ans = 4

octave:16> A(1,1)*A(2,2)-A(1,2)*A(2,1)ans = 4


8/18

Linear independence

Consider the following system:

The determinant of this matrix is zero.

(check this in octave)


9/18

Linear independence and rank

The rank of a matrix is simply the number of rows

which are not linearly dependent, or linearly

independent rows.

It can be shown that the rank with respect to

rows is equal to the rank with respect to columns,

i.e. the rank of a matrix is also equal to the

number of linearly independent columns. In Matlab we can use the rank() function to

compute the rank.


10/18

Underdetermined vs overdetermined

systems

The matrix A can have dimension mn with mn.

If m < n then there are more variables than

equations. Here it will usually be impossible to

find an unique exact solution. This is an

underdeterminedsystem.

If m > n then there are more equations than

variables. It may be impossible to satisfy allequations simultaneously. This is an

overdeterminedsystem.


11/18

Least squares and Linear regression

When you use the Matlab backslash operator in thecontext of an overdetermined system it automaticallyreturns a least squares solution.

This feature can be used for linear regression. Forexample, if we want to find the best fit line through aset of points (x1, y1), (x2, y2), ..., (xm, ym) then one of thesimplest forms of regression we can come up withinvolves a single estimate () for the slope of a line.

We redefine the problem as finding the beta valuesthat minimize the sum of square differences:


12/18

Exercise

Download the incomplete script

simpleReg.m and finish it by implementing

the function S(x,y,beta). The output should be:


13/18

Linear regression


14/18

Linear regression

We can define this in matrix form too. Here,

whereXis a matrix withXi1 = 1 andXi2 =xi , i.e.

a column of ones followed by the columnx.


15/18

Linear regression

Within Matlab, we simply write this as anoverdetermined system

X = y

to obtain our parameter estimates by leastsquares.

Matlab performs the minimization itself. This

is the simple linear regression.


16/18

Exercise

Start by loading the file agevbp.txt from the

course site. These are data comparing age (1st

column) vs systolic blood pressure.

1. Plot age (x-axis) vs blood pressure (y-axis)

2. Build the two-column matrix X as defined

above.

3. SolveX = y

4. Add the regression line to the plot


17/18

Quadratic regression

We then solveX = y as before


18/18

Exercises

1. Download the file qdata.txt of x,y pairs and

perform a quadratic regression. Plot the data

and the regression curve.

2. For the following data:

Perform an exponential regression using the

model:

Hours 1 2 3 4 5 6 7

Bacteria 25 38 58 89 135 206 315

session 9 linear algebra and rec

Documents