logistic regressionbrenocon/inlp2015/04-logreg.pdf · logistic regression. classification: logreg...
TRANSCRIPT
![Page 1: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/1.jpg)
Logistic RegressionSeptember 17, 2015
![Page 2: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/2.jpg)
Questions?
● From previous lecture?
● From HW?
![Page 3: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/3.jpg)
Naive Bayes Recap● What do you remember about classification
with Naive Bayes?
![Page 4: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/4.jpg)
Naive Bayes Recap● What do you remember about classification
with Naive Bayes?
● What statistics do you need to make a classification?
![Page 5: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/5.jpg)
Naive Bayes: Bag of Words
● BoW - Order independent
● Can we add more features to the model?
![Page 6: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/6.jpg)
Naive Bayes: Bag of Words
● Features statistically independent given class
● Examples of non-independent features?
![Page 7: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/7.jpg)
Independence Assumption
● Correlated features -> double counting
● Can hurt classifier accuracy & calibration
![Page 8: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/8.jpg)
● (Log) Linear Model - similar to Naive Bayes
● Doesn’t assume features are independent
● Correlated features don’t “double count”
Logistic Regression
![Page 9: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/9.jpg)
![Page 10: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/10.jpg)
Classification: LogReg (I)First, we’ll discuss how LogReg works.
![Page 11: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/11.jpg)
First, we’ll discuss how LogReg works.
Then, why it’s set up the way that it is.
Application: spam filtering
Classification: LogReg (I)
![Page 12: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/12.jpg)
● compute features (xs)
Classification: LogReg (I)
![Page 13: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/13.jpg)
● compute features (xs)
Classification: LogReg (I)
= (count “nigerian”, count “prince”, count “nigerian prince”)
![Page 14: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/14.jpg)
● compute features (xs)
● given weights (betas)
Classification: LogReg (I)
= (count “nigerian”, count “prince”, count “nigerian prince”)
![Page 15: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/15.jpg)
● compute features (xs)
● given weights (betas)
Classification: LogReg (I)
= (count “nigerian”, count “prince”, count “nigerian prince”)
= (-1.0, -1.0, 4.0)
![Page 16: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/16.jpg)
● compute features (x’s)● given weights (betas)● compute the dot product
Classification: LogReg (I)
![Page 17: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/17.jpg)
● compute features (x’s)● given weights (betas)● compute the dot product
Classification: LogReg (I)
![Page 18: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/18.jpg)
● compute the dot product
Classification: LogReg (II)
![Page 19: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/19.jpg)
● compute the dot product
● compute the logistic function
Classification: LogReg (II)
![Page 20: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/20.jpg)
LogReg Exercise
= (-1.0, -1.0, 4.0)
features: (count “nigerian”, count “prince”, count “nigerian prince”)
= (1, 1, 1)
![Page 21: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/21.jpg)
LogReg Exercise
= (-1.0, -1.0, 4.0)
= (1, 1, 1)
![Page 22: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/22.jpg)
OK, let’s take this step by step...
● Why dot product?
Classification: LogReg
![Page 23: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/23.jpg)
OK, let’s take this step by step...
● Why dot product?
● Why would we use the logistic function?
Classification: LogReg
![Page 24: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/24.jpg)
Intuition: weighted sum of features
All linear models have this form!
Classification: Dot Product
![Page 25: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/25.jpg)
NB as Log-Linear Model
Recall that Naive Bayes is also a linear model...
![Page 26: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/26.jpg)
NB as Log-Linear Model● What are the features in Naive Bayes?
● What are the weights in Naive Bayes?
![Page 27: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/27.jpg)
NB as Log-Linear Model
![Page 28: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/28.jpg)
NB as Log-Linear Model
![Page 29: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/29.jpg)
NB as Log-Linear Model
![Page 30: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/30.jpg)
NB as Log-Linear Model
In both NB and LogReg
we compute the dot product!
![Page 31: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/31.jpg)
What does this function look like?
What properties does it have?
Logistic Function
![Page 32: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/32.jpg)
Logistic Function
![Page 33: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/33.jpg)
● logistic function
Logistic Function
![Page 34: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/34.jpg)
● logistic function
● decision boundary is dot product = 0 (2 class)
Logistic Function
![Page 35: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/35.jpg)
● logistic function
● decision boundary is dot product = 0 (2 class)
● comes from linear log odds
Logistic Function
![Page 36: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/36.jpg)
● Both compute the dot product
● NB: sum of log probs; LogReg: logistic fun.
NB vs. LogReg
![Page 37: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/37.jpg)
● NB: learn conditional probabilities separately via counting
● LogReg: learn weights jointly
Learning Weights
![Page 38: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/38.jpg)
Learning Weights● given: a set of feature vectors and labels
● goal: learn the weights.
![Page 39: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/39.jpg)
Learning Weights
n examples; xs - features; ys - class
![Page 40: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/40.jpg)
Learning WeightsWe know:
So let’s try to maximize probability of the entire dataset - maximum likelihood estimation
![Page 41: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/41.jpg)
Learning WeightsSo let’s try to maximize probability of the entire
dataset - maximum likelihood estimation
![Page 42: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/42.jpg)
Learning WeightsSo let’s try to maximize probability of the entire
dataset - maximum likelihood estimation
![Page 43: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/43.jpg)
![Page 44: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/44.jpg)
![Page 45: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/45.jpg)
LogReg Exercise
= (1.0, -3.0, 2.0)
features: (count “nigerian”, count “prince”, count “nigerian prince”)
63% accuracy
![Page 46: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/46.jpg)
LogReg Exercise
= (1.0, -3.0, 2.0)
features: (count “nigerian”, count “prince”, count “nigerian prince”)
63% accuracy
= (0.5, -1.0, 3.0) 75% accuracy
![Page 47: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/47.jpg)
LogReg Exercise
= (1.0, -3.0, 2.0)
features: (count “nigerian”, count “prince”, count “nigerian prince”)
63% accuracy
= (0.5, -1.0, 3.0)
= (-1.0, -1.0, 4.0)
75% accuracy
81% accuracy
![Page 48: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/48.jpg)
Pros & Cons● LogReg doesn’t assume independence
○ better calibrated probabilities
● NB is faster to train; less likely to overfit
![Page 49: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/49.jpg)
NB & Log Reg● Both are linear models:
● Training is different:○ NB: weights trained independently○ LogReg: weights trained jointly
![Page 50: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/50.jpg)
LogReg: Important Details!● Overfitting / regularization● Visualizing decision boundary / bias term● Multiclass LogReg
You can use scikit-learn (python) to test it out!
![Page 51: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/51.jpg)
![Page 52: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/52.jpg)
Bias Term
![Page 53: Logistic Regressionbrenocon/inlp2015/04-logreg.pdf · Logistic Regression. Classification: LogReg (I) First, ... compute the logistic function Classification: LogReg (II) ... via](https://reader036.vdocument.in/reader036/viewer/2022071101/5fda30260410551f2e7f70bb/html5/thumbnails/53.jpg)