pattern recognition and machine learning chapter 6: kernel methods
Post on 19-Dec-2015
234 views
TRANSCRIPT
PATTERN RECOGNITION AND MACHINE LEARNINGCHAPTER 6: KERNEL METHODS
Kernel methods (1)
In chapters 3 and 4 we dealed with linear parametric models of this form:
where Áj(x) are known as basis functions and Á(x) is a fixed nonlinear feature space mapping
This models can be re-cast into an equivalent ‘dual representation’ in which the prediction is based on linear combination of a kernel functions
Dual representation(1)
0wwxwfwJ TN
i
Tn
,22
1
1
2
Let’s consider a linear regression model whose parameters are determines by minimizing a regularized sum-of-squares error function
If we set the gradient of J(w) with respect to w equal to zero, the solution for w takes the following form
If we substitute it into the model we obtain the following prediction
tIw TT 1
xItxwwxy TTT 1
,
Dual representation(2)
Let’s define the Gramm matrix , which is an NxN symmetric matrix with elements
In terms of Gram matrix the prediction can be written as
So now the prediction is expressed entirely in terms of the kernel function
tKIxkxItxwwxy TTT 11)(,
TK
mnmT
nnm xxkxxK ,
Kernel trick
If we have an algorithm formulated in such a way that the input vector x enters only in the form of scalar products, then we can replace the scalar product with some other choice of kernel.
Advantages of dual representation
1. We have to invert an NxN matrix, whereas in the original parameter space formulation we had to invert MxM matrix in order to determine w. I could be very important if N is significantly smaller than M.
2. As a dual representation is expressed entirely in terms of kernel function k(x,x’), we can work directly in terms of kernels, which allows to use feature spaces of high, even infinite, dimensionality.
3. Kernel functions can be defined not only over simply vectors of real numbers but also over objects as diverse as graphs, sets, string, and text documents
Constructing kernels (1)
Kernel is valid if it corresponds to a scalar product in some (perhaps infinite dimensional) feature space
Three approaches to construct kernels:1. To choose a feature space mapping Á(x)
2. To construct kernel function directly
M
iii
T zxzxzxk1
,
zx
zzzzxxxx
zxzxzxzx
zxzxzxzxk
T
T
T
2221
21
2221
21
22
222211
21
21
22211
2
,2,,2,
2
,
Constructing kernels (2)3.
Constructing kernels (3)
A necessary and sufficient condition for function k(x,x’) to be a valid kernel, is that Gram matrix K should be positive semidefinite for all possible choices of the set {x}
Some worth mentioning kernels (1)
• Linear kernel
• Gaussian kernel
• Kernel for sets
Some worth mentioning kernels (2)
• Kernel for probabilistic generative models
• Hidden Markov models
• Fisher kernel
Radial Basis Function Networks
To specify a regression model based on linear combination of fixed basis function we should choose the particular form of such functions.
On possible choice is to use radial basis functions, where each basis function depends only on the radial distance form a certain centre so that
We want to find the regression function y(x), using a Parzen density estimator to model the joint distribution
It can be shown that
Where and
Nadaraya-Watson model (1)
Nadaraya-Watson model (2)
Gaussian processes (1)
Let’s apply kernels to probabilistic discriminative models
Instead of defining prior on parameters vector w we define a prior probability over functions directly
?
Gaussian processes (2)
y is a linear combination of Gaussian distributed variables and hence is itself Gaussian
Where K is the Gram matrix with elements
So the marginal distribution p(y) is defined by a Gram matrix so that
Gaussian processes (3)
A Gaussian process is defined as a probability distribution over functions y(x) such that the set of values of y(x) evaluated at an arbitrary set of points {x} jointly has a Gaussian distribution.
This distribution is specified completely by the second-order statistics, the mean and the covariance.
The mean by symmetry is taken to be zero. The covariance is given by the kernel function
Gaussian processes (4)
Gaussian kernel Exponential kernel
Gaussian processes for regression(1)
Let’s consider t to be a target value
Where is a hyperparameter representing the precision of the noise.
Mentioning that we can find the marginal distribution
where
Gaussian processes for regression(2)
Our goal is to find the conditional distributionThe joint distribution is given by
Where
Using the results from Chapter 2 we see that is a Gaussian distribution with mean and covariance given by
Gaussian processes for regression(3)
Gaussian processes for regression(4)
Automatic relevance determination
Gaussian processes for classification (1)
Our goal is to model the posterior probabilities of the target variable for a new input vector, given a set of training data.
These probabilities must lie in the interval (0, 1), whereas a Gaussian process model makes predictions that lie on the entire real axis.
To adapt Gaussian processes we should transform the output of the Gaussian processes using an appropriate nonlinear activation function.
Gaussian processes for classification (2)
Let’s define a Gaussian process over a function a(x) and a logistic sigmoid transformation of the output
Gaussian processes for classification (3)
We need to determine the predictive distribution So we introduce a Gaussian process prior over the
vector a which in turn defines a non-Gaussian process over t
For two-class problem the required prediction distribution is given by
where
Gaussian processes for classification (4)
Approaches to Gaussian approximation:1. Variational inference2. Expectation propagation 3. Laplace approximation