maximum likelihood estimates and the em algorithms i
DESCRIPTION
Maximum Likelihood Estimates and the EM Algorithms I. Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University [email protected]. Part 1 Computation Tools. Computation Tools. R ( http://www.r-project.org/ ): good for statistical computing - PowerPoint PPT PresentationTRANSCRIPT
Maximum Likelihood Estimates and the EM
Algorithms I
Henry Horng-Shing LuInstitute of Statistics
National Chiao Tung [email protected]
1
Part 1Computation Tools
2
Computation Tools R (http://www.r-project.org/): good for
statistical computing C/C++: good for fast computation and large
data sets More:
http://www.stat.nctu.edu.tw/subhtml/source/teachers/hslu/course/statcomp/links.htm
3
The R Project R is a free software environment for
statistical computing and graphics. It compiles and runs on a wide variety of UNIX platforms, Windows and MacOS.
Similar to the commercial software of Splus.
C/C++, Fortran and other codes can be linked and called at run time.
More: http://www.r-project.org/
4
Download R from http://www.r-project.org/
5
Choose one Mirror Site of R
6
Choose the OS System
7
Select the Base of R
8
Download the Setup Program
9
Install R
Double click R-icon to install R
10
Execute R
Interactive command window
11
Download Add-on Packages
12
Choose a Mirror Site
Choose a mirror site close to you
1.
2.
13
Select One Package to Download
Choose one package to download, like rgl.
1.
2. 14
Load Packages There are two methods to load packages:
Method 1:
Click from the menu bar
Method 2:
Type “library(rgl)” in the command window
15
Help in R (1) What is the loaded library?
help(rgl)
16
Help in R (2) How to search functions for key words?
help.search(“key words”)It will show all functions has the key words.
help.search(“3D plot”)
Function name (belong to which package) description
17
Help in R (3) How to find the illustration of function?
?function nameIt will show the usage, arguments, author, reference, related functions, and examples.
?plot3d
18
R Operators (1)Mathematic operators:
+, -, *, /, ^ Mod: %% Sqrt, exp, log, log10, sin, cos, tan, …
19
R Operators (2)Other operators:
: sequence operator %*% matrix algebra <, >, <=, >= inequality ==, != comparison &, &&, |, || and, or ~ formulas <-, = assignment
20
Algebra, Operators and Functions
> 1+2[1] 3> 1>2[1] FALSE> 1>2|2>1[1] TRUE> A=1:3> A[1] 1 2 3> A*6[1] 6 12 18> A/10[1] 0.1 0.2 0.3> A%%2[1] 1 0 1
> B=4:6> A*B[1] 4 10 18> t(A)%*%B
[1][1] 32> A%*%t(B)
[1] [2] [3][1] 4 5 6 [2] 8 10 12[3] 12 15 18> sqrt(A)[1] 1.000 1.1414 1.7320> log(A)[1] 0.000 0.6931 1.0986
> round(sqrt(A),2)[1] 1.00 1.14 1.73> ceiling(sqrt(A))[1] 1 2 2> floor(sqrt(A))[1] 1 1 1> eigen(A%*%t(B))$values[1] 3.20e+01 5.83e-16
2.48e-16$vectors
[1] [2] [3][1] 0.2672 0.3273 -0.8890[2] 0.5345 -0.5217 0.2530[3] 0.8017 0.4665 0.3810
21
Variable TypesItem Descriptions
VectorX=c(10.4,5.6,3.1,6.4) or
Z=array(data_vector, dim_vector)
MatricesX=matrix(1:8,2,4) or
Z=matrix(rnorm(30),5,6)
Factors Statef=factor(state)
Lists pts = list(x=cars[,1], y=cars[,2])
Data Frames
data.frame(cbind(x=1, y=1:10), fac=sample(LETTERS[1:3], 10, repl=TRUE))
Functions name=function(arg_1,arg_2,…) expression
Missing Values
NA or NAN22
Define Your Own Function (1) Use “fix(myfunction)” # a window will show up function (parameter){
statements;return (object);# if you want to return some values
} Save the document Use “myfunction(parameter)” in R
23
Define Your Own Function (2) Example: Find all the factors of an integer
1.
2.3.
24
Define Your Own Function (3)
When you leave the program, remember to save the work space for the next use, or the function you defined will disappear after you close R project.
25
Read and Write Files
Write Data to a CSV File
Write Data to a TXT File
Read TXT and CSV Files
Demo
26
Write Data to a TXT File Usage:
write(x,file,…)> X=matrix(1:6,2,3)> X
[,1] [,2] [,3][1,] 1 3 5[2,] 2 4 6> write(t(X),file=“d:/out2.txt”,ncolumns=3)> write(X,file=“d:/out3.txt”,ncolumns=3)
d:/out2.txt1 3 52 4 6
d:/out3.txt1 2 34 5 6
27
Write Data to a CSV File
d:/out4.txt1,23,45,6
d:/out5.txt1,3,52,4,6
Usage: write.table(x,file=“foo.csv”,sep=“,”,…)> X=matrix(1:6,2,3)> X [,1] [,2] [,3][1,] 1 3 5[2,] 2 4 6>write.table(t(X),file=“d:/out4.txt”,sep=“,”,col.names=FALSE,row.names=FALSE)>write.table(X,file=“d:/out5.txt”,sep=“,”,col.names=FALSE,row.names=FALSE)
28
Read TXT and CSV Files Usage:read.table(file,...)> X=read.table(file="d:/out2.txt")> X v1 v2 v31 1 3 52 2 4 6> Y=read.table(file="d:/out5.txt",sep=",",header=FALSE)> Y V1 V21 1 22 3 43 5 6
29
Demo> Data=read.table(file="d:/01.csv",header=TRUE,sep=",")>Data Y X1 X21 2.651680 13.808990 26.758962 1.875039 17.734520 37.898573 1.523964 19.891030 26.036244 2.984314 15.574260 30.217545 10.423090 9.293612 28.914596 0.840065 8.830160 30.385787 8.126936 9.615875 32.69579>mean(Data$Y)[1] 4.060727>boxplot(Data$Y)
01.csv
30
Part 2Motivation Examples
31
Example 1 in Genetics (1) Two linked loci with alleles A and a, and B
and b A, B: dominant a, b: recessive
A double heterozygote AaBb will produce gametes of four types: AB, Ab, aB, ab
F ( Female) 1- r’ r’ (female recombination fraction)
M (Male) 1-r r (male recombination fraction)
A
B b
a B
A
b
a a
B
b
A
A
B b
a
32
Example 1 in Genetics (2) r and r’ are the recombination rates for male
and female Suppose the parental origin of these
heterozygote is from the mating of . The problem is to estimate r and r’ from the offspring of selfed heterozygotes.
Fisher, R. A. and Balmukand, B. (1928). The estimation of linkage from the offspring of selfed heterozygotes. Journal of Genetics, 20, 79–92.
http://en.wikipedia.org/wiki/Genetics http://www2.isye.gatech.edu/~brani/isyebayes/bank/handout12.pdf
AABB aabb
33
Example 1 in Genetics (3)
b
a
B
A
A
B b
a
a
b b
aA
B B
A
A
B
A
B b
a
b
a
1/2 1/2
a
B b
A
A
B b
a
1/2 1/2
AB ab aB Ab
Male (1-r)/2 (1-r)/2 r/2 r/2
Female (1-r’)/2 (1-r’)/2 r’/2 r’/2
34
Example 1 in Genetics (4)MALE
AB (1-r)/2
ab(1-r)/2
aBr/2
Abr/2
FEMALE
AB (1-r’)/2
AABB (1-r) (1-r’)/4
aABb(1-r) (1-r’)/4
aABBr (1-r’)/4
AABbr (1-r’)/4
ab(1-r’)/2
AaBb(1-r) (1-r’)/4
aabb(1-r) (1-r’)/4
aaBbr (1-r’)/4
Aabbr (1-r’)/4
aB r’/2
AaBB(1-r) r’/4
aabB(1-r) r’/4
aaBBr r’/4
AabBr r’/4
Ab r’/2
AABb(1-r) r’/4
aAbb(1-r) r’/4
aABbr r’/4
AAbb r r’/4
35
Example 1 in Genetics (5) Four distinct phenotypes: A*B*, A*b*, a*B* and
a*b*. A*: the dominant phenotype from (Aa, AA, aA). a*: the recessive phenotype from aa. B*: the dominant phenotype from (Bb, BB, bB). b* : the recessive phenotype from bb. A*B*: 9 gametic combinations. A*b*: 3 gametic combinations. a*B*: 3 gametic combinations. a*b*: 1 gametic combination. Total: 16 combinations.
36
Example 1 in Genetics (6)
Let (1 )(1 '), then
2( * *) ,
41
( * *) ( * *) ,4
( * *) .4
r r
P A B
P A b P a B
P a b
37
Example 1 in Genetics (7)
2 1 1; , , , .
4 4 4 4Multinomial n
We know that (1 )(1 '), 0 1/ 2, and 0 ' 1/ 2.
So, 1 1/ 4.
r r r r
Hence, the random sample of n from the offspring of selfed heterozygotes will follow a multinomial distribution:
38
Example 1 in Genetics (8)
2 1 1; , , , .
4 4 4 4Multinomial n
2 31 4
1 2 3 4
! 2 1( , ) ( ) ( ) ( ) .
! ! ! ! 4 4 4y yy yn
g yy y y y
Suppose that we observe the data of y = (y1, y2, y3, y4) = (125, 18, 20, 24), which is a random sample from
Then the probability mass function is
39
Estimation Methods Frequentist Approaches:
http://en.wikipedia.org/wiki/Frequency_probability
Method of Moments Estimate (MME)http://en.wikipedia.org/wiki/Method_of_moments_%28statistics%29
Maximum Likelihood Estimate (MLE)http://en.wikipedia.org/wiki/Maximum_likelihood
Bayesian Approaches:http://en.wikipedia.org/wiki/Bayesian_probability
40
Method of Moments Estimate (MME) Solve the equations when population means are
equal to sample means:
for k = 1, 2, …, t, where t is the number of parameters to be estimated.
MME is simple. Under regular conditions, the MME is consistent! More:
http://en.wikipedia.org/wiki/Method_of_moments_%28statistics%29
' 'k km
41
MME for Example 1
Note: MME can’t assure
11 1 1
22 2 2
1 2 3 4
33 3 3
44 4 4
2 1ˆ( ) 4( )
4 21
ˆ( ) 1 4ˆ ˆ ˆ ˆ4 ˆ
1 4ˆ( ) 1 4
44
ˆ( ) 4
MME
yE Y n y
ny
E Y n yny
E Y n yn
yE Y n y
n
ˆ [1/ 4,1]!MME
42
MME by R
43
MME by C/C++
44
Maximum Likelihood Estimate (MLE) Likelihood: Maximize likelihood: Solve the score
equations, which are setting the first derivates of likelihood to be zeros.
Under regular conditions, the MLE is consistent, asymptotic efficient and normal!
More: http://en.wikipedia.org/wiki/Maximum_likelihood
45
Example 2 (1)
# of tossing head ( ) probability
0 (0,0,0) (1-p)3
1 (1,0,0) (0,1,0) (0,0,1) p(1-p)2
2 (0,1,1) (1,0,1) (1,1,0) p2(1-p)
3 (1,1,1) p3
1 2 3, ,x x x
1, if the ith trial is head;
0, if the ith trial is tail.iX
1 with probability ;
0 with probability 1- .i
pX
p
We toss an unfair coin 3 times and the random variable is
If p is the probability of tossing head, then
46
Example 2 (2)
21 2 3
3( | , , ) (1 ) , where 0 p 1.
2L p x x x p p
2 2
2
3(1 ) 3(1 ) 6 (1 )
2
9 12 3 = 0.
p p p p pp
p p
Suppose we observe the toss of 1 heads and 2 tails, the likelihood function becomes
One way to maximize this likelihood function is by solving the score equation, which sets the first derivative to be zero:
47
Example 2 (3) The solution of p for the score equation is
1/3 or 1.
One can check that p=1/3 is the maximum point. (How?)
Hence, the MLE of p is 1/3 for this example.
48
MLE for Example 1 (1) Likelihood MLE:
2 31 4
1 2 3 4
! 2 1( ) ( ) ( ) ( )
! ! ! ! 4 4 4y yy yn
Ly y y y
ˆ ˆmax ( ) max log ( ) MLE MLEL L
2 3 41 2 3 4
( ) ( )
! 2 1 log( ) log( ) ( ) log( ) log( )
! ! ! ! 4 4 4
logL
ny y y y
y y y y
2 31 4log ( ) 02 1
y yy ydL
d
21 2 3 4 1 2 3 4 4( ) ( 2 2 ) 2 0y y y y y y y y y A B C
49
MLE for Example 1 (2)
Checking:
(1)
(2)
(3)
2 4
2MLE
B B AC
A
50
2
2ˆ
( )0?
MLE
d
d
ˆ1/ 4 1?MLE
ˆCompare log ( )?MLEL
Use R to find MLE (1)
51
Use R to find MLE (2)
52
Use C/C++ to find MLE (1)
53
Use C/C++ to find MLE (2)
54
Exercises Write your own programs for those
examples presented in this talk. Write programs for those examples
mentioned at the following web page:http://en.wikipedia.org/wiki/Maximum_likelihood
Write programs for the other examples that you know.
55
More Exercises (1) Example 3 in genetics: The observed data
are (nO, nA, nB, nAB) = (176, 182, 60, 17) ~ Multinomial(r^2, p^2+2pr, q^2+2qr, 2pq), where p, q, and r fall in [0,1] such that p+q+r = 1. Find the likelihood function and score equations for p, q, and r.
56
More Exercises (2) Example 4 in the positron emission tomography
(PET): The observed data are n*(d) ~Poisson(λ*(d)), d = 1, 2, …, D, and
The values of p(b,d) are known and the unknown parameters are λ(b), b = 1, 2, …, B.
Find the likelihood function and score equations for λ(b), b = 1, 2, …, B.
*
1
( ) ( , ) ( ).B
b
d p b d b
.
57
More Exercises (3) Example 5 in the normal mixture: The observed
data xi, i = 1, 2, …, n, are random samples from the following probability density function:
Find the likelihood function and score equations for the following parameters:
2
1
1
( ) ( , ),
1, and 0 1 for all .
K
i k k kk
K
k kk
f x Normal
k
1 1 1( ,..., , ,..., , ,..., ).K K K 58