Minimal Neural Networks
Support vector machines and Bayesian learning for neural
networks
Peter Andras
Bayesian neural networks I.The Bayes rule: Let’s consider a model of a system and an
observation of the system, an event. The a posteriori probability of correctness of the model, after the observation of the event, is proportional to the product of the a priori correctness of the model and the probability of the event conditioned by the correctness of the model.
Mathematically:)(
)()|()|(
DP
HPHDPDHP
where is the parameter of the model H and D is the observed event
Bayesian neural networks II.
Best model: model with highest a posteriori probability of correctness
Model selection by optimizing the formula:
)(ln()|(ln(min HPHDP
Bayesian neural networks III.
Application to neural networks:
g is the function represented by the neural network,
niyxD ii ,1|),( is the observed event
where is the vector of all parameters of the network
we suppose normal distribution for the data conditioned by the validity of a model, i.e., the observed values yi are normally distributed around g(xi), if is the correct parameter vector
Bayesian neural networks IV.
By making the calculations we get:
n
i
ii
n
i
r
xgy
xgyr
eHDP
ii
1
22
1
2
))((
))((2
11ln)|(ln(
2
2
and the new formula for optimization is:
n
i
ii Prxgy
1
22 ))(ln())((2
1min
Bayesian neural networks V.
The equivalence of the regularization and Bayesian model selection
Regularization formula:
n
i
ii Tgxgy
1
22 ||||))((2
1min
Bayesian optimization formula:
n
i
ii Prxgy
1
22 ))(ln())((2
1min
Equivalence: 22 ||||))(ln( TgPr
Both represents a priori information about the correct solution
Bayesian neural networks VI.
Bayesian pruning by regularization
Gauss pruning:
Laplace pruning:
Cauchy pruning:
n
i
N
kk
ii
rxgy
1 1
22
2
2))((
2
1min
n
i
N
kk
ii rxgy
1 1
22 ||))((2
1min
n
i
N
kk
ii
rxgy
1 1
22
2 )1ln())((2
1min
N is the number of components of the vectors
Support vector machines - SVM I.
Linear separable classes:
- many separators
- there is an optimal separator
Support vector machines - SVM II.
How to find the optimal separator ?
- support vectors
- overspecification
Property: one less support vector new optimal separator
Support vector machines - SVM III.
We look for minimal and robust separators.
These are minimal and robust models of the data.
The full data set is equivalent with the set of the support vectors with respect to the specification of the minimal robust model.
Support vector machines - SVM IV.Mathematical problem formulation I.
1,1,,1),,( iii yniyx
we represent the separator as a pair (w,b), where w is vector and b is a scalar
we look w and b such that they satisfy:
11
11
iiT
iiT
yifbxw
yifbxw
The support vectors are those xi-s for which this inequality is in fact equality.
Support vector machines - SVM V.
Mathematical problem formulation II.
The distances form the origo of the hyper-planes of the support vectors are:
2
2
||||
|1|
||||
|1|
w
bd
w
bd
The distance between the two planes is: 2||||
2
wd
Support vector machines - SVM VI.
Mathematical problem formulation III.
Optimal separator: the distance between the two hyper-planes is maximal
Optimization: 2||||2
1w
with the restrictions that
or in other form
SViiT Iiifybxw
SViT
i Iiifbxwy 01)(
Support vector machines - SVM VII.
Mathematical problem formulation IV.
Complete optimization formula, using Lagrange multipliers
0
)(||||2
1
11
2
i
n
ii
n
i
iTiiP bxwywL
Support vector machines - SVM VIII.Mathematical problem formulation V.
Writing the optimality conditions for w and b we get:
n
iii
n
i
iii
y
xyw
1
1
0
The dual problem is:
n
j
jijiji
n
i
n
iiD xxyyL
111
,2
1
The support vectors are those xi-s for which i is strictly positive
Support vector machines - SVM IX.
Graphical interpretation
We search for the tangent point of a hyper-ellipsoid with the positive space quadrant
Support vector machines - SVM X.
How to solve the support vector problem ?
Optimization with respect to the -s
- gradient method
- Newton and quasi-Newton methods
We get as result:
- the support vectors
- the optimal linear separator
Support vector machines - SVM XI.
Implications for artificial neural networks:
- robust perceptron (low sensitivity to noise)
- minimal linear classificatory neural network
Support vector machines - SVM XII.What can we do if the boundary is nonlinear ?
Idea: transform the data vectors to a space where the separator is linear
Support vector machines - SVM XIII.
The transformation many times is made to an infinite dimensional space, usually a function space.
Example: x cos(uTx)
Support vector machines - SVM XIV.
The new optimization formulas are:
)(,0
))(,(||||2
1
11
2
iii
n
ii
n
i
iiiP
xx
bxwywL
n
j
jijiji
n
i
n
iiD xxyyL
111
)(),(2
1
Support vector machines - SVM XIV.
How to handle the products of the transformed vectors ?
Idea: use a transformation that fits the Mercer theorem
Mercer theorem: Let RRRK nn : then K has a decomposition
)(),(),( yxyxK
where HRn : and H is a function space
if and only if
nn RR
dxdyygyxKxg )(),()(0
for each )(2nRLg
Support vector machines - SVM XV.
Optimization formula with transformation that fits the Mercer theorem:
n
j
jijiji
n
i
n
iiD xxKyyL
111
),(2
1
The form of the solution:
n
i
iii bxxKybxw
1
0),()(,
the b is determined from an equation valid for a support vector
Support vector machines - SVM XVI.
Examples of transformations and kernels
22221
21
32 ,),(,),2,()(;: vuvuKxxxxxRR Ta.
b.
c.
2222121
21
42 ,),(,),,,()(;: vuvuKxxxxxxxRR T
2sin2
))(21
(sin),(
,))sin(,),2sin(),sin(),cos(,),2cos(),cos(,2
1()(;:
vu
vuNvuK
NxxxNxxxxHR T
Support vector machines - SVM XVII.
Other typical kernels
),tanh(),(.
),(.
)1,(),(.
),(),(.
2
2
2
||||
vuavuKd
evuKc
vuvuKb
vuvuKa
r
vu
p
p
Support vector machines - SVM XVIII.
Summary of main ideas
• look for minimal complexity classification
• transform the data to another space where the class boundaries are linear
• use Mercer kernels
Support vector machines - SVM XIX.
Practical issues
• the global optimization doesn’t work with large amount of data sequential optimization with chunks of the data
• the resulted models are minimal complexity models, they are insensitive to noise and keep the generalization ability of the more complex models
• applications: character recognition, economic forecasting
Regularization neural networks
General optimization vs. optimization over the grid
The regularization operator specifies the grid:
- we look for functions that satisfy ||Tg||2=0
- in the relaxed case the regularization operator is incorporated as a constraint in the error function:
2||||TgEE usualT