John K.S. Lau & Assoc Prof C.K. Tham , ECE, NUSE-mail: {elelksj, eletck}@nus.edu.sg

Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams

IEEE ICPADS’12 International Workshop on Scalable Computing for Big Data Analytics (SC-BDA)

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 2

Motivation

How to leverage computationally expensive model in detecting abnormal events in real-time?

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 3

Contributions

•Application of Hidden Markov Models (HMM) in identifying not only hidden meaningful events but also abnormal events. The focus in this work is on abnormal events.

•Event-Driven Architecture (EDA) for decoupling of HMM computations from real-time event processing.

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 4

Event-Driven Architecture (EDA)

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 5

Abnormal Event Processing (AEP)

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 6

Abnormal Event Processing (AEP)

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 7

Abnormal Event Processing (AEP)

•SELECT tmin, tmax FROM [table name] WHERE [unique identifier]–tmin and tmax pre-computed via HMM based on most

recent data. Normal event is within tmin <= t <= tmax.

•SELECT * FROM [stream name] HAVING duration < tmin OR duration > tmax

–Any incoming event is flagged abnormal in real-time.

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 8

Experimental Results

HMM

Find highest probability

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 9

Experimental Results

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 10

Experimental Results

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 11

Experimental Results

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 12

Experimental Results

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 13

Future Works

•Ability to detect more meaningful hidden states such as accident, heavy rain, flood, etc. besides just general anomaly.

•Application of Map-Reduce in HMM computations.

Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 14

THANK YOUQuestions?