anomaly detection - university of edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... ·...

71
1 Anomaly Detection Josef Kittler Dept. Electronic Engineering, University of Surrey [email protected] Support by EPSRC and dstl is gratefully acknowledged

Upload: others

Post on 14-Oct-2020

8 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

1

Anomaly Detection

Josef Kittler

Dept. Electronic Engineering, University of Surrey

[email protected]

Support by EPSRC and dstl is gratefully acknowledged

Page 2: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

2

Aims

n  To introduce the subject of anomaly detection, its content and relevance

n  To introduce the terminology of anomaly detection

n  To riview/introduce the mathematical background required

Page 3: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Outline

n  Introduction to anomaly detection n  Problem formulation n  Statistical hypothesis testing n  One class classification (SVM) n  Critique of classical anomaly detection n  Complementary mechanisms for anomaly

detection n  Anomaly detection system architecture n  Incongruence detection n  Dempster Shaffer reasoning (Prof David Parish)

3

Page 4: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Introduction to anomaly

n  Anomaly – n  an important notion in human

understanding of the environment n  deviation from normal order or rule n  failure to relate sensor data to a meaning n  manifest in weak or no support for domain

specific hypotheses

n  Many synonyms signifying different nuances n  rarity, irregularity, incongruence,

abnormality, unexpected event, novelty, innovation, outlier

4

Page 5: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

n  In science/engineering n  prove disprove hypothesis n  fault detection n  outdated model requires adaptation

5

Page 6: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Diverse applications

n  Many applications formulated as anomaly detection problems n  surveillance n  novel object detection n  abnormal communication network activity n  medical diagnostics n  video segmentation n  suspicious behaviour

6

Page 7: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Anomaly detection problem formulations

n Classification problem n Abnormality types known

n Detection problem n Samples of normal class and negative

examples available n Hypothesis testing problem

n Only samples of normal class available n One class classification problem

7

Page 8: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Prior art in anomaly detection

n  Edgeworth (1888) n  Hundreds of papers n  Many approaches

n  statistical, NN, classification, clustering, information theoretic, spectral

n  Excellent surveys n  Markou&Singh (SP 2003, statistical, neural) n  Hodge&Austin (AI Review 2004) n  Agyemang&Barker&Alhajj (Int Data Anal 2006) n  Chandola&Banerjee&Kumar (ACM Surveys 2010) n  Saligrama&Konrad&Vodoin (SPM2010, video)

8

Page 9: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Anomaly detection as a problem in classification

n  Microcalcification detection: anomaly in tissue texture

n  Anomaly class known n  Anomaly detection solved as a

classification problem

9

Page 10: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Two Class Problem

n  Many decision boundaries can separate these two classes.

Class 1

Class 2

Page 11: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Classification

Class 1

Class 2

Simple Perceptron learning Algorithm

Page 12: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Support Vector Machine (SVM)

SVM finds the best separating boundary

Class 1

Class 2

Support Vectors

Optimal Separating Hyperplane

Page 13: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

If not Linearly Separable

n Slack variable ξi we allow “error” in classification

Class 1

Class 2

Page 14: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Extension to Non-linear Decision Boundary

n SVM solves this using kernel function

n Kernel tricks for efficient computation n Minimizing ||w||2 produces a “good” classifier

Feature space Input space

Φ: x → φ(x)

Page 15: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Classical anomaly model

n  Conventional mathematical model n  outlier of a distribution n  empirical distribution deviates from

the model distribution

15

x

Page 16: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Hypothesis testing

n  This typically involves some proposition, referred to as a null hypothesis and a test statistics.

n  If the outcome of the test statistics is consistent with its known distribution model , then the null hypothesis is accepted.

n  An outlier of that distribution would lead to the hypothesis rejection.

n  Example: pdf is uniform over support domain S

n  For any x outside S the hypothesis would be rejected

16

Page 17: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Normal (Gaussian) distribution

n  Gaussian distribution

n  where is its mean vector and is the covariance matrix

n  Gaussian extends to infinity, hence technically no observation is an outlier

n  An observation is considered an outlier at a given level of significance, i.e. if the test statistics value is beyond a boundary corresponding to some vestigial probability outside it, such as 5% or 1%.

17

Page 18: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

18

Examples of gaussians

Page 19: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

19

Examples of gaussians

Page 20: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

20

Examples of gaussians

Page 21: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Anomaly detection as one class classification

21

Page 22: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Data support domain estimation

n  Fundamentally different from the two class formulation

n  Possible approaches n  Parametric/nonparametric density estimation n  Quantile function estimation n  Convex hull enclosure n  One class SVM 22

Page 23: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

One class SVM

23

n  The aim of one class SVM is to enclose the available one class training set

n  Solution should generalise well

Page 24: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Kernel space

n  The kernel function, e.g. Gaussian, defines high dimensional feature space implicitly

n  The solution defined in terms of a linear boundary in the feature space

24

Page 25: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Objective function

n  It delineates the training set at a specified level of confidence

25

Page 26: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Slack variables

26

Page 27: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Constrained optimisation

n  The use of slack variables and the regularisation term control the trade-off between empirical risk and overfitting

n  The optimisation problems can be stated as

n  Solve by method of Lagrange multipliers

27

Page 28: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Dual optimisation problem

n aa

28

Page 29: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Relationship to sphere fitting

29

n  Hence, all points lie on a hypersphere

n  Finding the smallest hypersphere is equivalent to maximising the marging between data and the origin

Page 30: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Relationship with the Parzen estimator

n  When

30

Page 31: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Example

n  25 samples from a Gaussian

n  Parameter v=0.5

31

Page 32: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Example of coefficients

32

Page 33: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Example: A linear case

n  Example: Two Gaussians

33

Page 34: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

1 class SVM vs 2 class SVM

34

Page 35: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Example: A linear case

35

Page 36: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

36

Page 37: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Comments

n Selection of meta parameters n Kernel bandwidth n  Parameters

n Applicable to high dimensional problems

n Nonlinear boundary facilitated by the kernel trick

37

Page 38: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

One class SVM summary

n  Quantile estimation formulated as kernel machine learning

n  High probability regions are estimated subject to regularisation

n  One class SVM solution compared with two class SVM

38

Page 39: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Classical model and its critique

n  Multiple models n  Discriminative classifiers n  Ambiguity of interpretation n  Contextual reasoning n  Hierarchical representation n  Data quality n  Model pruning

39

������������������

� �������������

�� �������������

������� �� �����

����

� ���������

Page 40: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Different aspects of anomaly

40

Page 41: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Different aspects of anomaly

41

-Distribution drift -Novelty detection

Page 42: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Data quality

42

Page 43: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Data quality

43

Page 44: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Incongruence/unexpected event

44

n  Magritte’s La duree poignard n  Model base pruning

n  Computational efficiency n  Hierarchical representation

o i

o j

ω α ω β

Scene Graph Model Graph

Page 45: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Data quality/ decision confidence

45

n  Data quality n  effect of noise on the notion of

normality n  need to measure data quality n  notion of data quality and its

dependence on context n  Confidence in classifier output

Page 46: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Challenges

n Challenges of a more comprehensive approach n Meaning of data quality

• Quality is relative, not absolute • Different levels of representation

n Data quality measures • Multiple aspects of quality • Measures of quality • Overall quality/fusion

46

Page 47: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Distribution drift

47

n  Transfer of learning

Page 48: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Anomaly detection system architecture

n  Classical model deficient n  Outlier detection not enough

n  Other mechanisms required n  Data quality detection n  Incongruence detection n  Decision confidence estimation n  Drift detection n  As well as outlier detection n  Reasoning (fusion)

48

Page 49: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Nuances of anomaly

n  No anomaly n  Noisy measurement n  Unknown object n  Corrupted

measurement n  Congruent labelling n  Unknown structure n  Spurious

measurement errors

49

n  Unexpected structural component

n  Unexpected structural component & structure

n  Measurement model drift

Page 50: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Context of anomaly detection

n Designing an operational system with anomaly detection capability n Data collection n System architecture n Representation n Machine learning n Context modelling n High level reasoning n Validation

50

Page 51: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Incongruence detection

n  Detecting differences between observations and expectations (anomaly, rare event, incongruence)

n  Basic principle – comparison of outputs of weak and strong classifiers (Ketabdar et al 2007)

n  Dirac Project (Burget et al 2008, Weinshall et all [2009-2012])

n  Exemplified by out-of-vocabulary word detection n  Phoneme recognizer (weak classifier) n  HMM speech recognizer (strong, contextual classifier)

51

Page 52: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Classifier incongruence

n  Testing for incongruence n  need an incongruence measure n  understand its properties n  sensitivity to noise

n  classifier 1 output n  classifier 2 output

52

Page 53: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Incongruence measures

n  Kulback-Leibler divergence: measures mutual information between the two distributions

n  Known as Bayesian surprise

n  Chi-square measure

n  Assumption: estimation errors Gaussian n  Variance proportional to the sum of probabilities

53

Page 54: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Properties of Chi-square

n  Errors for non dominant classes are magnified (scaled by small variance)

n  Joint zero entries are ignored n  Even when the probabilities of the dominant

hypotheses agree, the sum over all the other hypotheses could be high

n  The test statistics based on the assumption that the sampling distribution of errors is a product of Gaussian with zero mean and different variance for each class posterior

54

Page 55: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Bhattacharyya distance

n  Bhattacharyya (geometric) distance

n  Properties: n  Distance different for different distributions, even if the

two classifier outputs are identical for all hypotheses n  Works as a matched filter n  Measure can be affected by disagreements in the

probabilities of minor hypotheses n  Using as a reference the classifier output with the

lowest entropy, the measure would yield much higher value than the posterior distribution with the highest entropy 55

Page 56: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Bhattacharyya distance

n  Properties (cont)

56

Page 57: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Kolmogorov-Smirnov test

57

Page 58: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Properties of K-S test

n  Resilience to estimation noise

58

Page 59: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Cramer von Mises measure

n  Defined as

n  Measures cumulative sum differences weighted by sum of probabilities (variance)

n  All terms contribute, not only the max term n  This may impact on error robustness

59

Page 60: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Bayesian surprise measure

n  Properties n  ~

n  Not symmetric n  Divergence difficult to calibrate n  Classifier decision agnostic

60

Page 61: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Delta measure

n  Defined as

n  Dominant hypotheses taken into account, non dominant ignored

61

Page 62: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Distribution of

n  Distribution of noise free values

62

Page 63: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Estimation errors

n Class probabilities corrupted by noise

n  satisfying

63

Page 64: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Error sensitivity

n  Probabilities estimates affected by errors

n  Constraints

64

Page 65: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Estimation error distribution

n  Gaussian

with folded tails

65

Page 66: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

n  Pdf curves of for classifier output similarity with estimation error noise N(0,0.1)

66

Page 67: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

n  Label agreement

67

Page 68: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

n  Label disageement

68

Page 69: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

n  Results of simulation studies to determine decision threshold

69

Page 70: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Error sensitivity of incongruence measures

70

Scenario •  Identical class probabilities •  Estimation error st.dev 0.05

Page 71: Anomaly Detection - University of Edinburghudrc.eng.ed.ac.uk/sites/udrc.eng.ed.ac.uk/files... · Anomaly detection system architecture ! Incongruence detection ! Dempster Shaffer

Thresholding

n  One of the classifier incongruence measures can be used as a test statistics to detect incongruence

n  An error sensitivity analysis would need to be carried for the chosen measure to estimate the test statistics distribution

n  An appropriate decision threshold could then be determined to achieve a specified level of significance

71