entropy of hidden markov processes
DESCRIPTION
Entropy of Hidden Markov Processes. Or Zuk 1 Ido Kanter 2 Eytan Domany 1 Weizmann Inst. 1 Bar-Ilan Univ. 2. Overview. Introduction Problem Definition Statistical Mechanics approach Cover&Thomas Upper-Bounds Radius of Convergence Related subjects Future Directions. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/1.jpg)
.
Entropy of Hidden Markov Processes
Or Zuk1 Ido Kanter2 Eytan Domany1
Weizmann Inst.1 Bar-Ilan Univ.2
![Page 2: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/2.jpg)
2
Overview
Introduction Problem Definition Statistical Mechanics approach Cover&Thomas Upper-Bounds Radius of Convergence Related subjects Future Directions
![Page 3: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/3.jpg)
3
HMP - Definitions
Markov Process:
X – Markov Process
M – Transition Matrix Mij = Pr(Xn+1 = j| Xn = i)
Hidden Markov Process :Y – Noisy Observation of XN – Noise/Emission Matrix Nij = Pr(Yn = j| Xn = i)
M
NN
Xn Xn+1
Yn+1Yn
![Page 4: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/4.jpg)
4
Example: Binary HMP
0 1
p(1|0)
p(0|1)
p(1|1)
p(0|0)
)1|1()1|0(
)0|1()0|0(
pp
pp
0 1
q(0|0) q(1|0)q(0|1)
q(1|1)
)1|1()1|0(
)0|1()0|0(
Transition Emission
![Page 5: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/5.jpg)
5
Example: Binary HMP (Cont.) For simplicity, we will concentrate on
Symmetric Binary HMP :
M = N =
So all properties of the process depend on two parameters, p and . Assume (w.l.o.g.) p, < ½
pp
pp
1
1
1
1
![Page 6: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/6.jpg)
6
HMP Entropy Rate
Definition :
H is difficult to compute, given as a Lyaponov Exponent (which is hard to compute generally.) [Jacquet et al 04]
What to do ? Calculate H in different Regimes.
![Page 7: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/7.jpg)
7
Different Regimes
p -> 0 , p -> ½ ( fixed)
-> 0 , -> ½ (p fixed)
[Ordentlich&Weissman 04] study several regimes.
We concentrate on the ‘small noise regime’ -> 0.
Solution can be given as a power-series in :
![Page 8: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/8.jpg)
8
Statistical Mechanics
First, observe the Markovian Property :
Perform Change of Variables :
![Page 9: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/9.jpg)
9
Statistical Mechanics (cont.)
Ising Model :
, {-1,1} Spin Glasses
+ + + + - + - -
+ + - - - + + -
1
1
2
2
K
J
K
J
n
n
![Page 10: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/10.jpg)
10
Statistical Mechanics (cont.)
Summing, we get :
![Page 11: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/11.jpg)
11
Statistical Mechanics (cont.) Computing the Entropy (low-temperature/high-field
expansion) :
![Page 12: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/12.jpg)
12
Cover&Thomas BoundsIt is known (Cover & Thomas 1991) :
We will use the upper-bounds C(n), and derive their orders :
Qu : Do the orders ‘saturate’ ?
![Page 13: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/13.jpg)
13
Cover&Thomas Bounds (cont.)
0 0.1 0.2 0.3 0.4 0.50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
eps
p
Upperbound / Lowerbound Average
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
eps
p
Upperbound Minus Lowerbound
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0 0.1 0.2 0.3 0.4 0.50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
eps
p
Relative Error Upperbound Minus Lowerbound / Average
0.02
0.04
0.06
0.08
0.1
0.12
0 0.1 0.2 0.3 0.4 0.50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
eps
p
Relative Error Upperbound Minus Lowerbound / (1-Average)
0
0.5
1
1.5
2
2.5
3
n=4
![Page 14: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/14.jpg)
14
Cover&Thomas Bounds (cont.)
Ans : Yes. In fact they ‘saturate’ sooner than would have
been expected ! For n (K+3)/2 they become constant.
We therefore have : Conjecture 1 : (proven for k=1)
How do the orders look ? Their expression is simpler when expressed using = 1-2p, which is the 2nd eigenvalue of P.
Conjecture 2 :
![Page 15: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/15.jpg)
15
First Few Orders :
Note : H0-H2 proven. The rest are conjectures from the upper-bounds.
![Page 16: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/16.jpg)
16
First Few Orders (Cont.) :
![Page 17: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/17.jpg)
17
First Few Orders (Cont.) :
![Page 18: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/18.jpg)
18
Radius of Convergence :
When is our approximation good ? Instructive : Compare to the I.I.D. model
For HMP, the limit is unknown. We used the fit :
![Page 19: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/19.jpg)
19
Radius of Convergence (cont.) :
![Page 20: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/20.jpg)
20
Radius of Convergence (cont.) :
![Page 21: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/21.jpg)
21
Relative Entropy Rate
Relative entropy rate :
We get :
![Page 22: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/22.jpg)
22
Index of Coincidence Take two realizations Y,Y’ (of length n) of the same HMP. What is the probability that
they are equal ?
Exponentially decaying with n.
We get :
Similarly, we can solve for three and four (but not five) realizations. Can give bounds on the entropy rate.
![Page 23: Entropy of Hidden Markov Processes](https://reader036.vdocument.in/reader036/viewer/2022062322/568146e3550346895db41bdd/html5/thumbnails/23.jpg)
23
Future Directions
Proving conjectures Generalizations (e.g. any alphabets, continuous case) Other regimes Relative Entropy of two HMPs
Thank You