CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNINGTemporal sequences: Hidden Markov Models and Dynamic Bayesian Networks

MOTIVATION Observing a stream of data

Monitoring (of people, computer systems, etc)

Surveillance, tracking Finance & economics Science

Questions: Modeling & forecasting Unobserved variables

TIME SERIES MODELING Time occurs in steps t=0,1,2,…

Time step can be seconds, days, years, etc State variable Xt, t=0,1,2,… For partially observed problems, we see

observations Ot, t=1,2,… and do not see the X’s X’s are hidden variables (aka latent variables)

MODELING TIME Arrow of time

Causality => Bayesian networks are natural models of time series

Causes Effects

PROBABILISTIC MODELING For now, assume fully observable case

What parents?

X0 X1 X2 X3

X0 X1 X2 X3

MARKOV ASSUMPTION Assume Xt+k is independent of all Xi for i<t

P(Xt+k | X0,…,Xt+k-1) = P(Xt+k | Xt,…,Xt+k-1) K-th order Markov Chain

X0 X1 X2 X3

X0 X1 X2 X3

X0 X1 X2 X3

X0 X1 X2 X3

Order 0

Order 1

Order 2

Order 3

1ST ORDER MARKOV CHAIN MC’s of order k>1 can be converted into a

1st order MC on the variable Yt = {Xt,…,Xt+k-

1} So w.o.l.o.g., “MC” refers to a 1st order MC

Y0 Y1 Y2 Y3

X0 X1 X2 X3

X0 X1’ X2’ X3’

X1 X2 X3 X4

INFERENCE IN MC What independence relationships can we

read from the BN?

X0 X1 X2 X3

Observe X1

X0 independent of X2, X3, …

P(Xt|Xt-1) known as transition model

INFERENCE IN MC Prediction: the probability of future state?

P(Xt) = Sx0,…,xt-1P (X0,…,Xt) = Sx0,…,xt-1P (X0) Px1,…,xt P(Xi|Xi-1)= Sxt-1P(Xt|Xt-1) P(Xt-1)

Approach: maintain a belief state bt(X)=P(Xt), use above equation to advance to bt+1(X) Equivalent to VE algorithm in sequential order

[Recursive approach]

BELIEF STATE EVOLUTION P(Xt) = Sxt-1P(Xt|Xt-1) P(Xt-1) “Blurs” over time, and (typically) approaches

a stationary distribution as t grows Limited prediction power Rate of blurring known as mixing time

STATIONARY DISTRIBUTIONS For discrete variables Val(X)={1,…,n}:

Transition matrix Tij = P(Xt=i|Xt-1=j) Belief bt(X) is just a vector bt,i=P(Xt=i) Belief update equation: bt+1 = T*bt

A stationary distribution b is one in which b = Tb => b is an eigenvector of T with eigenvalue 1 => b is in the null space of (T-I)

HISTORY DEPENDENCE In Markov models, the state must be chosen

so that the future is independent of history given the current state

Often this requires adding variables that cannot be directly observed

Are these people walking toward you or away from you?

What comes next?

“the bare”

minimum

essentials

market

wipes himselfwith the rabbit

PARTIAL OBSERVABILITY Hidden Markov Model (HMM)

X0 X1 X2 X3

O1 O2 O3

Hidden state variables

Observed variables

P(Ot|Xt) called the observation model (or sensor model)

INFERENCE IN HMMS Filtering Prediction Smoothing, aka hindsight Most likely explanation

X0 X1 X2 X3

O1 O2 O3

INFERENCE IN HMMS Filtering Prediction Smoothing, aka hindsight Most likely explanation

X0 X1 X2

O1 O2

Query variable

FILTERING Name comes from signal processing P(Xt|o1:t) = Sxt-1 P(xt-1|o1:t-1) P(Xt|xt-1,ot) P(Xt|Xt-1,ot) = P(ot|Xt-1,Xt)P(Xt|Xt-1)/P(ot|Xt-1)

= a P(ot|Xt)P(Xt|Xt-1)

X0 X1 X2

O1 O2

Query variable

FILTERING P(Xt|o1:t) = a Sxt-1P(xt-1|o1:t-1) P(ot|Xt)P(Xt|xt-1) Forward recursion If we keep track of belief state bt(X) = P(Xt|o1:t)

=> O(|Val(X)|2) updates for each t!

X0 X1 X2

O1 O2

Query variable

PREDICT-UPDATE INTERPRETATION Given old belief state bt-1(X) Predict: First compute MC update

bt’(Xt)=P(Xt|o1:t-1) = a Sxbt-1(x) P(Xt|Xt-1=x) Update: Re-weight to account for observation

probabilities: bt(x) = bt’(x)P(ot|Xt=x)

X0 X1 X2

O1 O2

Query variable

INFERENCE IN HMMS Filtering Prediction Smoothing, aka hindsight Most likely explanation

X0 X1 X2 X3

O1 O2 O3

Query

PREDICTION P(Xt+k|o1:t) 2 steps: P(Xt|o1:t), then P(Xt+k|Xt) Filter to time t, then predict as with standard

MC

X0 X1 X2 X3

O1 O2 O3

Query

INFERENCE IN HMMS Filtering Prediction Smoothing, aka hindsight Most likely explanation

X0 X1 X2 X3

O1 O2 O3

Query

SMOOTHING P(Xk|o1:t) for k < t P(Xk|o1:k,ok+1:t)

= P(ok+1:t|Xk,o1:k)P(Xk|o1:k)/P(ok+1:t|o1:k)= a P(ok+1:t|Xk)P(Xk|o1:k)

X0 X1 X2 X3

O1 O2 O3

Query

Standard filtering to time k

SMOOTHING Computing P(ok+1:t|Xk) P(ok+1:t|Xk) = Sxk+1P(ok+1:t|Xk,xk+1) P(xk+1|Xk)

= Sxk+1P(ok+1:t|xk+1) P(xk+1|Xk)= Sxk+1P(ok+2:t|xk+1)P(ok+1|xk+1)P(xk+1|Xk)

X0 X1 X2 X3

O1 O2 O3

Given prior states

What’s the probability of this sequence?

Backward recursion

INTERPRETATION Filtering/prediction:

Equivalent to forward variable elimination / belief propagation

Smoothing: Equivalent to forward VE/BP up to query variable,

then backward VE/BP from last observation back to query variable

Running BP to completion gives the smoothed estimates for all variables (forward-backward algorithm)

INFERENCE IN HMMS Filtering Prediction Smoothing, aka hindsight Most likely explanation

Subject of next lecture

X0 X1 X2 X3

O1 O2 O3

Query returns a path through state space x0,…,x3

APPLICATIONS OF HMMS IN NLP Speech recognition Hidden phones

(e.g., ah eh ee th r) Observed, noisy acoustic

features (produced by signal processing)

PHONE OBSERVATION MODELS

Phonet

Signal processing

Features(24,13,3,59)

Featurest

Model defined to be robust over variations in accent, speed, pitch, noise

PHONE TRANSITION MODELS

Phonet

Featurest

Good models will capture (among other things):Pronunciation of wordsSubphone structureCoarticulation effects Triphone models = order 3 Markov chain

Phonet+1

WORD SEGMENTATION Words run together when

pronounced Unigrams P(wi) Bigrams P(wi|wi-1) Trigrams P(wi|wi-1,wi-2)

Logical are as confusion a may right tries agent goal the was diesel more object then information-gathering search is

Planning purely diagnostic expert systems are very similar computational approach would be represented compactly using tic tac toe a predicate

Planning and scheduling are integrated the success of naïve bayes model is just a possible prior source by that time

Random 20 word samples from R&N using N-gram models

WHAT ABOUT MODELS WITH MANY VARIABLES? Say X has n binary variables, O has m binary variables Naively, a distribution over Xt may be intractable to

represent (2n entries) Transition models P(Xt |Xt-1) require 22n entries Observation models P(Ot |Xt) require 2n+m entries

Is there a better way?

EXAMPLE: FAILURE DETECTION Consider a battery meter sensor

Battery = true level of battery BMeter = sensor reading

Transient failures: send garbage at time t Persistent failures: send garbage forever

EXAMPLE: FAILURE DETECTION Consider a battery meter sensor

Battery = true level of battery BMeter = sensor reading

Transient failures: send garbage at time t 5555500555…

Persistent failures: sensor is broken 5555500000…

DYNAMIC BAYESIAN NETWORK Template model relates variables on prior

time step to the next time step (2-TBN) “Unrolling” the template for all t gives the

ground Bayesian network

BMetert

BatterytBatteryt-1

BMetert ~ N(Batteryt,s)

DYNAMIC BAYESIAN NETWORK

BMetert

BatterytBatteryt-1

BMetert ~ N(Batteryt,s)

P(BMetert=0 | Batteryt=5) = 0.03Transient failure model

RESULTS ON TRANSIENT FAILUREE

(Bat

tery

t)

Transient failure occurs

Without model

With model

Meter reads 55555005555…

RESULTS ON PERSISTENT FAILUREE

(Bat

tery

t)

Persistent failure occurs

With transient model

Meter reads 5555500000…

PERSISTENT FAILURE MODEL

BMetert

BatterytBatteryt-1

BMetert ~ N(Batteryt,s)

P(BMetert=0 | Batteryt=5) = 0.03

Brokent-1 Brokent

P(BMetert=0 | Brokent) = 1

RESULTS ON PERSISTENT FAILUREE

(Bat

tery

t)

Persistent failure occurs

With transient model

Meter reads 5555500000…

With persistent failure model

HOW TO PERFORM INFERENCE ON DBN? Exact inference on “unrolled” BN

E.g. Variable Elimination Typical order: eliminate sequential time steps so

that the network isn’t actually constructed Unrolling is done only implicitly

BM1

Ba1Ba0

Br0 Br1

BM2

Ba2

Br2

BM3

Ba3

Br3

BM4

Ba4

Br4

ENTANGLEMENT PROBLEM After n time steps, all n variables in the belief

state become dependent! Unless 2-TBN can be partitioned into disjoint

subsets (rare) Lost sparsity structure

APPROXIMATE INFERENCE IN DBNS Limited history updates Assumed factorization of belief state Particle filtering

INDEPENDENT FACTORIZATION Idea: assume belief state P(Xt) factors across

individual attributes P(Xt) = P(X1,t)*…*P(Xn,t) Filtering: only maintain factored distributions

P(X1,t|O1:t),…,P(Xn,t|O1:t) Filtering update: P(Xk,t|O1:t) = Sxt-1P(Xk,t|Ot,Xt-1)

P(Xt-1|O1:t-1) = marginal probability query over 2-TBN

X1,t-1

Xn,t-1

X1,t

Xn,t

O1,t

Om,t

NEXT TIME Viterbi algorithm

Read K&F 13.2 for some context Kalman and particle filtering

Read K&F 15.3-4