ECE  5424:  Introduction  to  Machine  Learning

Stefan  LeeVirginia  Tech

Topics:  – Regression

Readings:  Barber  17.1,  17.2

Administrativia• HW1

– 39  Submissions– 58  Students  Enrolled– 19  MIA    (?????)

• Project  Proposal– Due:  09/21,  11:55  pm       (LESS  THAN  A  WEEK!)– <=  2pages,  NIPS  format

• HW2  – Out  today– Due  on  Wednesday  09/28,  11:55pm– Please  please  please  please  please  start  early– Implement  Linear  Regression,  Naïve  Bayes,  Logistic  Regression

(C)  Dhruv  Batra   2

Recap  of  last  time

(C)  Dhruv  Batra   3

Learning  a  Gaussian• Collect  a  bunch  of  data

– Hopefully,  i.i.d.  samples– e.g.,  exam  scores

• Learn  parameters– Mean– Variance

(C)  Dhruv  Batra   4

MLE  for  Gaussian• Prob.  of  i.i.d.  samples  D={x1,…,xN}:

• Log-­likelihood   of  data:

Slide  Credit:  Carlos  Guestrin(C)  Dhruv  Batra   5

Your  second  learning  algorithm:MLE  for  mean  of  a  Gaussian

• What’s  MLE  for  mean?

Slide  Credit:  Carlos  Guestrin(C)  Dhruv  Batra   6

Learning  Gaussian  parameters

• MLE:

(C)  Dhruv  Batra   7

• Conjugate  priors– Mean:  Gaussian  prior– Variance:  Inverse  Gamma  or  Wishart Distribution

• Prior  for  mean:

Slide  Credit:  Carlos  Guestrin(C)  Dhruv  Batra   8

Bayesian  learning  of  Gaussian  parameters

MAP  for  mean  of  Gaussian

Slide  Credit:  Carlos  Guestrin(C)  Dhruv  Batra   9

New  Topic:  Regression

(C)  Dhruv  Batra   10

1-­NN  for  Regression• Often  bumpy  (overfits)

(C)  Dhruv  Batra   11Figure  Credit:  Andrew  Moore

(C)  Dhruv  Batra   12Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   13Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   14Slide  Credit:  Greg  Shakhnarovich

Linear  Regression• Demo

– http://hspm.sph.sc.edu/courses/J716/demos/LeastSquares/LeastSquaresDemo.html

(C)  Dhruv  Batra   15

(C)  Dhruv  Batra   16Slide  Credit:  Greg  Shakhnarovich

Plan  for  Today• Regression

– Linear  Regression  Recap• Some  matrix  calculus  review

– Connections  with  Gaussians• The  outlier  problem  L

– Robust  Least  Squares

(C)  Dhruv  Batra   17

(C)  Dhruv  Batra   18Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   19Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   20Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   21Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   22Slide  Credit:  Greg  Shakhnarovich

(C)  Dhruv  Batra   23Slide  Credit:  Greg  Shakhnarovich

• Why  sum  squared  error???• Gaussians,  Watson,  Gaussians…

But,  why?

24(C)  Dhruv  Batra  

(C)  Dhruv  Batra   25Slide  Credit:  Greg  Shakhnarovich

MLE  Under  Gaussian  Model• On  board

(C)  Dhruv  Batra   26

Is  OLS  Robust?• Demo

– http://www.calpoly.edu/~srein/StatDemo/All.html

• Bad  things happen when the  data  does not  comefrom  your model!

• How  do  we  fix this?

(C)  Dhruv  Batra   27

Robust  Linear  Regression• y ~  Lap(w’x,  b)

• On  board

(C)  Dhruv  Batra   28

0 0.2 0.4 0.6 0.8 1−6

−5

−4

−3

−2

−1

0

1

2

3

4Linear data with noise and outliers

least squareslaplace

−3 −2 −1 0 1 2 3−0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

L2

L1

huber