nuno vasconcelos ucsdsvcl.ucsd.edu/courses/ece271a/handouts/ml.pdf · nuno vasconcelos ucsd....
TRANSCRIPT
-
Maximum likelihood estimationMaximum likelihood estimation
Nuno VasconcelosUCSD
-
Bayesian decision theory• recall that we have
– Y – state of the world– X – observations– g(x) – decision function– L[g(x),y] – loss of predicting y with g(x)
• Bayes decision rule is the rule that minimizes the risk
[ ]),( YXLERisk YX=• for the “0-1” loss
[ ]),(,YX
⎧ ≠)(1
• optimal decision rule is the maximum a posteriori⎩⎨⎧
=≠
=yxgyxg
yxgL)(,0)(,1
]),([
2
• optimal decision rule is the maximum a-posteriori probability rule
-
MAP ruleh h th t it b i l t d i f th• we have shown that it can be implemented in any of the
three following ways– 1) )|(maxarg)( |
* xiPxi XY=)
– 2)
)|(maxarg)( | xiPxi XYi
[ ])()|(maxarg)( |* iPixPxi YYX=)
– 3)
[ ]|i
[ ])(log)|(logmaxarg)( |* iPixPxi YYX +=)• by introducing a “model” for the class-conditional
distributions we can express this as a simple equation
[ ])(g)|(gg)( | YYXi
distributions we can express this as a simple equation– e.g. for the multivariate Gaussian
⎬⎫
⎨⎧ Σ )()(11)|( 1TP
3⎭⎬⎫
⎩⎨⎧ −Σ−−
Σ= − )()(
21exp
||)2(1)|( 1| ii
Ti
idYX
xxixP µµπ
-
The Gaussian classifierdi i i t
• the solution is
[ ]iii xdxi αµ += ),(minarg)(*discriminant:PY|X(1|x ) = 0.5
with
[ ]iiii
xdxi αµ +),(minarg)(
)()(),( 1 yxyxyxd iT
i −Σ−=−
)(log2)2log( iPYid
i −Σ= πα
• the optimal rule is to assign x to the closest class• closest is measured with the Mahalanobis distance di(x,y)• can be further simplified in special cases
4
• can be further simplified in special cases
-
Geometric interpretation• for Gaussian classes, equal covariance σ2I
µµ −
( )Yji
ji
iP
w
σµµσµµ
+
=
)(l2
2
( )jiY
Y
ji
ji
jPx µµ
µµ
µµ−
−−=
)()(log
2 20
σ
w
µi
µjσ
x0 )(1 iP
5)()(log1
2
2 jPiP
Y
Y
ji
σµµ −
-
Geometric interpretation• for Gaussian classes, equal but arbitrary covariance
( )w µµΣ−1( )
( ) ( ) ( )jiY
Tji
ji
jPiPx
w
µµµµµµ
µµ
µµ
−Σ
−+
=
−Σ=
− )()(log1
2 10 ( ) ( ) Yjiji jPµµµµ −Σ− )(2
w
µi
µjx0
6
-
Bayesian decision theory• advantages:
– BDR is optimal and cannot be beaten– Bayes keeps you honest– models reflect causal interpretation of the problem, this is how
we think– natural decomposition into “what we knew already” (prior) and
“what data tells us” (CCD)– no need for heuristics to combine these two sources of infono need for heuristics to combine these two sources of info– BDR is, almost invariably, intuitive– Bayes rule, chain rule, and marginalization enable modularity,
and scalability to very complicated models and problemsand scalability to very complicated models and problems
• problems:– BDR is optimal only insofar the models are correct.
7
-
Implementation• we do have an optimal solution
( )jiw µµ −Σ= −1
f( ) ( ) ( )jiY
Y
jiT
ji
ji
jPiPx µµ
µµµµµµ
−−Σ−
−+
=− )(
)(log12 10
• but in practice we do not know the values of the parameters µ, Σ, PY(1)– we have to somehow estimate these values– this is OK, we can come up with an estimate from a training set– e.g. use the average value as an estimate for the mean
( )( )
( )Yjiji
iPx
w
µµµµ
µµ
ˆˆ)(ˆ
log1ˆˆ
ˆˆˆ 1
+=
−Σ= −
8( ) ( ) ( )jiYjiTji jP
x µµµµµµ )(ˆ
logˆˆˆˆ2 10
−−Σ−
−=−
-
Important • warning: at this point all optimality claims for the BDR
cease to be valid!!• the BDR is guaranteed
to achieve the minimumloss when we use theloss when we use thetrue probabilities
• when we “plug in” the b bilit ti tprobability estimates,
we could be implementing p ga classifier that is quite distant from the optimal– e.g. if the PX|Y(x|i) look like the example above
I could never approximate them well by parametric models (e g
9
– I could never approximate them well by parametric models (e.g. Gaussian)
-
Maximum likelihood• this seems pretty serious
– how should I get these probabilities then?
• we rely on the maximum likelihood (ML) principle• this has three steps:
1) we choose a parametric model for all probabilities– 1) we choose a parametric model for all probabilities– to make this clear we denote the vector of parameters by Θ and
the class-conditional distributions by
note that this means that Θ is NOT a random variable (otherwise
);|(| ΘixP YX– note that this means that Θ is NOT a random variable (otherwise
it would have to show up as subscript)– it is simply a parameter, and the probabilities are a function of this
parameter
10
parameter
-
Maximum likelihood• three steps:
– 2) we assemble a collection of datasetsD(i) = {x (i) x (i)} set of examples drawn independently fromD(i) = {x1(i) , ..., xn(i)} set of examples drawn independently from class i
3) we select the parameters of class i to be the ones that– 3) we select the parameters of class i to be the ones that maximize the probability of the data from that class
( )ΘΘ ;|maxarg )( iDP i( )( )Θ=
Θ=ΘΘ
;|logmaxarg
;|maxarg
)(|
)(|
iDP
iDP
iYX
YXi
– like before, it does not really make any difference to maximize
( )Θ
;|gg |YX
11
, y yprobabilities or their logs
-
Maximum likelihood• since
– each sample D(i) is considered independentlyt ti t d l f l (i)– parameter Θi estimated only from sample D(i)
• we simply have to repeat the procedure for all classes• so from now on we omit the class variable• so, from now on we omit the class variable
( )Θ=ΘΘ
;maxarg* DPX
( )Θ=Θ
Θ
;logmaxarg DPX
• the function PX(D;Θ) is called the likelihood of the parameter Θ with respect to the data
i l h lik lih d f i12
• or simply the likelihood function
-
Maximum likelihood• note that the likelihood
function is a function of the parameters Θof the parameters Θ
• it does not have the same shape as the
fdensity itself• e.g. the likelihood
function of a Gaussianfunction of a Gaussian is not bell-shaped
• the likelihood isdefined only after we have a sample
⎫⎧ 2)(1 µd
13⎭⎬⎫
⎩⎨⎧ −−=Θ 22 2
)(exp)2(
1);(σµ
σπddPX
-
Maximum likelihood• given a sample, to obtain ML estimate we need to solve
( )Θ=Θ ;maxarg* DP
• when Θ is a scalar this is high-school calculus
( )Θ=ΘΘ
;maxarg DPX
when Θ is a scalar this is high school calculus
• we have a maximum whenfirst derivative is zero
14
– first derivative is zero– second derivative is negative
-
The gradient• in higher dimensions, the generalization of the derivative
is the gradient
∇f
• the gradient of a function f(w) at z is
Tff ⎞⎜⎛ ∂∂ ∇f
• the gradient has a nice geometrici t t ti
n
zwfz
wfzf ⎟
⎠
⎞⎜⎜⎝
⎛∂∂
∂∂
=∇−
)(,),()(10
L
interpretation– it points in the direction of maximum
growth of the functionf(x,y)
– which makes it perpendicular to the contours where the function is constant
),( 00 yxf∇
15
),( 00 yf
),( 11 yxf∇
-
maxThe gradient
max
• note that if ∇f = 0– there is no direction of growthg– also -∇f = 0, and there is no direction of
decrease– we are either at a local minimum or maximum
“ ddl ” i tor “saddle” point• conversely, at local min or max or saddle
pointdi ti f th d min– no direction of growth or decrease
– ∇f = 0• this shows that we have a critical point if
d l if ∇f 0saddle
and only if ∇f = 0• to determine which type we need second
order conditions
16
-
The Hessian• the extension of the second-order derivative is the
Hessian matrix⎤⎡
⎥⎥⎥⎤
⎢⎢⎢⎡
∂∂∂
∂∂
=∇−
)()(
)(10
2
20
2
2
xxxfx
xf
xfn
M
L
⎥⎥⎥
⎦⎢⎢⎢
⎣ ∂∂
∂∂∂
=∇
−−
)()(
)(
21
2
01
2
xx
fxxx
fxf
nn
L
M
– at each point x, gives us the quadratic function
⎦⎣
xxfx t )(2∇
that best approximates f(x)
xxfx )(∇
17
-
The Hessian• this means that, when gradient is
zero at x, we have max– a maximum when function can be
approximated by an “upwards-facing” quadratic
– a minimum when function can be approximated by a “downwards-facing” quadratic
– a saddle point otherwise
saddlemin
18
-
The Hessian max• for any matrix M, the function
Mxx t
• is– upwards facing quadratic when M
Mxx
upwards facing quadratic when Mis negative definite
– downwards facing quadratic when Mis positive definite
min
saddle
is positive definite– saddle otherwise
• hence, all that matters is the positived fi it f th H idefiniteness of the Hessian
• we have a maximum when Hessianis negative definite
19
is negative definite
-
Maximum likelihood• in summary, given a sample, we need to solve
( )ΘΘ* DPmax
• the solutions are the parameters
( )Θ=ΘΘ
;maxarg DPX
the solutions are the parameters such that
0);( =Θ∇ DP 0);( =Θ∇Θ DPXn
Xt DP ℜ∈∀≤∇Θ θθθθ ,0);(
2
• note that you always have to check the second-order condition!
XΘ ,);(
20
condition!
-
Maximum likelihood• let’s consider the Gaussian example
• given a sample {T1, …, TN} of independent pointsgiven a sample {T1, …, TN} of independent points• the likelihood is
21
-
Maximum likelihood• and the log-likelihood is
• the derivative with respect to the mean is zero when
• or
• note that this is just the sample mean22
• note that this is just the sample mean
-
Maximum likelihood• and the log-likelihood is
• the derivative with respect to the variance is zero when
• or
• note that this is just the sample variance23
• note that this is just the sample variance
-
Maximum likelihood• example:
– if sample is {10,20,30,40,50}
24
-
Homework• show that the Hessian is negative definite
nt 2
• show that these formulas can be generalized to the vector
nX
t DP ℜ∈∀≤∇Θ θθθθ ,0);(2
show that these formulas can be generalized to the vector case– D(i) = {x1(i) , ..., xn(i)} set of examples from class i
th ML ti t– the ML estimates are
∑=Σ Tii xx ))((1 )()( µµ∑ ix )(1µ
• note that the ML solution is usually intuitive
∑ −−=Σj
ijiji xxn))(( )()( µµ∑=
jji xn
)(µ
25
• note that the ML solution is usually intuitive
-
Estimators• when we talk about estimators, it is important to keep in
mind that– an estimate is a number– an estimator is a random variable
)(ˆ XXfθ
• an estimate is the value of the estimator for a given
),,( 1 nXXf K=θ
gsample.
• if D = {x1 , ..., xn}, when we say ∑=j
jxn1µ̂
what we mean is with j
nn xXxXnXXf
===
,,1 11),,(ˆ
KKµ
∑ XXXf 1)(26
the Xi are random variables∑=j
jn XnXXf ),,( 1 K
-
Bias and variance• we know how to produce estimators (by ML)• how do we evaluate an estimator?• Q1: is the expected value equal to the true value?• this is measured by the bias
if– if
then
),,(ˆ 1 nXXf K=θthen
an estimator that has bias will usually not converge to the perfect
( ) [ ]θθ −= ),,(ˆ 1,,1 nXX XXfEBias n KK– an estimator that has bias will usually not converge to the perfect
estimate θ, no matter how large the sample is– e.g. if θ is negative and the estimator is
the bias is clearly non zero∑=
jjn Xn
XXf 211),,( K
27
the bias is clearly non-zero j
-
Bias and variance• the estimators is said to be biased
– this means that it is not expressive enough to approximate the tr e al e arbitraril elltrue value arbitrarily well
– this will be clearer when we talk about density estimation
• Q2: assuming that the estimator converges to the true 2 g gvalue, how many sample points do we need?– this can be measured by the variance
( )[ ]( ){ }2)()(
ˆ
XXfEXXfE
Var =θ
– the variance usually decreases as one collects more training
[ ]( ){ }1,,1,, ),,(),,( 11 nXXnXX XXfEXXfE nn KK KK −
28
y gexamples
-
Example• ML estimator for the mean of a Gaussian N(µ,σ2)
( ) [ ] [ ]ˆˆˆ −=−= µµµµµ XXXX EEBias ( ) [ ] [ ]1
,,,,
1
11
−⎥⎦
⎤⎢⎣
⎡=
==
∑ µ
µµµµµ
iXX
XXXX
XE
EEBias
n
nn KK
[ ]1,,1
−=
⎥⎦
⎢⎣
∑
∑
µ
µ
iXX
iiXX
XE
nnK
[ ]
[ ]1,,1
−= ∑
∑
µ
µ
iX
iiXX
XE
n nK
[ ]
0=−=
∑µµ
µ
i
iX XEn i
29• the estimator is unbiased
-
38