csc321 lecture 4: learning a classifier · 2018-01-13 · de ne a model and a cost function...

CSC321 Lecture 4: Learning a Classifier

Roger Grosse

Roger Grosse CSC321 Lecture 4: Learning a Classifier 1 / 31

Overview

Last time: binary classification, perceptron algorithm

Limitations of the perceptron

no guarantees if data aren’t linearly separablehow to generalize to multiple classes?linear model — no obvious generalization to multilayer neural networks

This lecture: apply the strategy we used for linear regression

define a model and a cost functionoptimize it using gradient descent


Overview

Design choices so far

Task: regression, binary classification, multiway classification

Model/Architecture: linear, log-linear

Loss function: squared error, 0–1 loss, cross-entropy, hinge loss

Optimization algorithm: direct solution, gradient descent,perceptron


Overview

Recall: binary linear classifiers. Targets t ∈ {0, 1}

z = wTx + b

y =

{1 if z ≥ 00 if z < 0

Goal from last lecture: classify all training examples correctly

But what if we can’t, or don’t want to?

Seemingly obvious loss function: 0-1 loss

L0−1(y , t) =

{0 if y = t1 if y 6= t

= 1y 6=t .


Attempt 1: 0-1 loss

As always, the cost E is the average loss over training examples; for0-1 loss, this is the error rate:

E =1

N

N∑i=1

1y (i) 6=t(i)


Attempt 1: 0-1 loss

Problem: how to optimize?

Chain rule:∂L0−1∂wj

=∂L0−1∂z

∂z

∂wj

But ∂L0−1/∂z is zero everywhere it’s defined!

∂L0−1/∂wj = 0 means that changing the weights by a very smallamount probably has no effect on the loss.The gradient descent update is a no-op.


Attempt 2: Linear Regression

Sometimes we can replace the loss function we care about with onewhich is easier to optimize. This is known as a surrogate loss function.

We already know how to fit a linear regression model. Can we usethis instead?

y = w>x + b

LSE(y , t) =1

2(y − t)2

Doesn’t matter that the targets are actually binary.

Threshold predictions at y = 1/2.


Attempt 2: Linear Regression

The problem:

The loss function hates when you make correct predictions with highconfidence!

If t = 1, it’s more unhappy about y = 10 than y = 0.


Attempt 3: Logistic Activation Function

There’s obviously no reason to predict values outside [0, 1]. Let’ssquash y into this interval.

The logistic function is a kind of sigmoidal, orS-shaped, function:

σ(z) =1

1 + e−z

A linear model with a logistic nonlinearity is known as log-linear:

z = w>x + b

y = σ(z)

LSE(y , t) =1

2(y − t)2.

Used in this way, σ is called an activation function, and z is called thelogit.


Attempt 3: Logistic Activation Function

The problem:(plot of LSE as a function of z)

∂L∂wj

=∂L∂z

∂z

∂wj

wj ← wj − α∂L∂wj

In gradient descent, a small gradient (in magnitude) implies a smallstep.

If the prediction is really wrong, shouldn’t you take a large step?


Logistic Regression

Because y ∈ [0, 1], we can interpret it as the estimated probabilitythat t = 1.

The pundits who were 99% confident Clinton would win were muchmore wrong than the ones who were only 90% confident.

Cross-entropy loss captures this intuition:

LCE(y , t) =

{− log y if t = 1− log(1− y) if t = 0

= −t log y − (1− t) log(1− y)


Logistic Regression

Logistic Regression:

z = w>x + b

y = σ(z)

=1

1 + e−z

LCE = −t log y − (1− t) log(1− y)

[[gradient derivation in the notes]]


Logistic Regression

Problem: what if t = 1 but you’re really confident it’s a negativeexample (z � 0)?

If y is small enough, it may be numerically zero. This can cause verysubtle and hard-to-find bugs.

y = σ(z) ⇒ y ≈ 0

LCE = −t log y − (1− t) log(1− y) ⇒ computes log 0

Instead, we combine the activation function and the loss into a singlelogistic-cross-entropy function.

LLCE(z , t) = LCE(σ(z), t) = t log(1 + e−z) + (1− t) log(1 + ez)

Numerically stable computation:

E = t * np.logaddexp(0, -z) + (1-t) * np.logaddexp(0, z)


Logistic Regression

Comparison of loss functions:


Logistic Regression

Comparison of gradient descent updates:

Linear regression:

w← w − α

N

N∑i=1

(y (i) − t(i)) x(i)

Logistic regression:

w← w − α

N

N∑i=1

(y (i) − t(i)) x(i)

Not a coincidence! These are both examples of matching lossfunctions, but that’s beyond the scope of this course.


Hinge Loss

Another loss function you might encounter is hinge loss. Here, we taket ∈ {−1, 1} rather than {0, 1}.

LH(y , t) = max(0, 1− ty)

This is an upper bound on 0-1 loss (auseful property for a surrogate lossfunction).

A linear model with hinge loss is calleda support vector machine. You alreadyknow enough to derive the gradientdescent update rules!

Very different motivations from logisticregression, but similar behavior inpractice.


Logistic Regression

Comparison of loss functions:


Multiclass Classification

What about classification tasks with more than two categories?It is very hard to say what makes a 2 Some examples from an earlier version of the net



Targets form a discrete set {1, . . . ,K}.It’s often more convenient to represent them as one-hot vectors, or aone-of-K encoding:

t = (0, . . . , 0, 1, 0, . . . , 0)︸︷︷︸entry k is 1



Now there are D input dimensions and K output dimensions, so weneed K × D weights, which we arrange as a weight matrix W.

Also, we have a K -dimensional vector b of biases.

Linear predictions:

zk =∑j

wkjxj + bk

Vectorized:z = Wx + b



A natural activation function to use is the softmax function, amultivariable generalization of the logistic function:

yk = softmax(z1, . . . , zK )k =ezk∑k ′ ezk′

The inputs zk are called the logits.

Properties:

Outputs are positive and sum to 1 (so they can be interpreted asprobabilities)If one of the zk ’s is much larger than the others, softmax(z) isapproximately the argmax. (So really it’s more like “soft-argmax”.)Exercise: how does the case of K = 2 relate to the logistic function?

Note: sometimes σ(z) is used to denote the softmax function; in thisclass, it will denote the logistic function applied elementwise.



If a model outputs a vector of class probabilities, we can usecross-entropy as the loss function:

LCE(y, t) = −K∑

k=1

tk log yk

= −t>(log y),

where the log is applied elementwise.

Just like with logistic regression, we typically combine the softmaxand cross-entropy into a softmax-cross-entropy function.



Multiclass logistic regression:

z = Wx + b

y = softmax(z)

LCE = −t>(log y)

Tutorial: deriving the gradient descent updates

∂LCE

∂z= y − t


Convex Functions

Recall: a set S is convex if for any x0, x1 ∈ S,

(1− λ)x0 + λx1 ∈ S for 0 ≤ λ ≤ 1.

A function f is convex if for any x0, x1 in the domain of f ,

f ((1− λ)x0 + λx1) ≤ (1− λ)f (x0) + λf (x1)

Equivalently, the set ofpoints lying above thegraph of f is convex.

Intuitively: the functionis bowl-shaped.


Convex Functions

We just saw that theleast-squares lossfunction 1

2 (y − t)2 isconvex as a function of y

For a linear model,z = w>x + b is a linearfunction of w and b. Ifthe loss function isconvex as a function ofz , then it is convex as afunction of w and b.


Convex Functions

Which loss functions are convex?


Convex Functions

Why we care about convexity

All critical points are minima

Gradient descent finds the optimal solution (more on this in a laterlecture)


Gradient Checking

We’ve derived a lot of gradients so far. How do we know if they’recorrect?

Recall the definition of the partial derivative:

∂

∂xif (x1, . . . , xN) = lim

h→0

f (x1, . . . , xi + h, . . . , xN)− f (x1, . . . , xi , . . . , xN)

h

Check your derivatives numerically by plugging in a small value of h,e.g. 10−10. This is known as finite differences.


Gradient Checking

Even better: the two-sided definition

∂

∂xif (x1, . . . , xN) = lim

h→0

f (x1, . . . , xi + h, . . . , xN)− f (x1, . . . , xi − h, . . . , xN)

2h


Gradient Checking

Run gradient checks on small, randomly chosen inputs

Use double precision floats (not the default for most deep learningframeworks!)

Compute the relative error:

|a− b||a|+ |b|

The relative error should be very small, e.g. 10−6


Gradient Checking

Gradient checking is really important!

Learning algorithms often appear to work even if the math is wrong.

But:They might work much better if the derivatives are correct.Wrong derivatives might lead you on a wild goose chase.

If you implement derivatives by hand, gradient checking is the singlemost important thing you need to do to get your algorithm to workwell.


csc321 lecture 4: learning a classifier · 2018-01-13 · de ne a model and a cost function...

Documents