![Page 1: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/1.jpg)
Five Things I Learned While Building Anomaly Detection Tools
(Or: 5 things that bit me in the …)
Toufic Boubez, Ph.D.
Founder, CTO
Metafor Software
![Page 2: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/2.jpg)
2
Preamble
• IANA Data Scientist! I’m just an engineer that needed to get stuff done!
• I learned (!) many more things, but cannnot be mentioned!– Because lawyers – But ask me later
• I usually beat up on parametric, Gaussian, supervised techniques– This talk is not an exception,– But more of a “lessons learned” message
• Note: all data real• Note: no y-axis labels on charts – on purpose!!• Note to self: remember to SLOW DOWN!• Note to self: mention the cats!! Everybody loves cats!!
![Page 3: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/3.jpg)
3
• Co-Founder/CTO Metafor Software• Co-Founder/CTO Layer 7 Technologies
– Acquired by Computer Associates in 2013– I escaped
• CTO Saffron Technology• IBM Chief Architect for SOA• Co-Author, Co-Editor: WS-Trust, WS-
SecureConversation, WS-Federation, WS-Policy• Building large scale software systems for >20
years (I’m older than I look, I know!)
Toufic intro – who I am
![Page 4: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/4.jpg)
4
Why Anomaly Detection?
• Watching screens on the “Wall of Charts” cannot scale!– Leads to alert fatigue
• Need to automate detection of anomalous behaviors
• Anomaly detection is the search for items or events which do not conform to an expected pattern. [Chandola, V.; Banerjee, A.; Kumar, V. (2009). "Anomaly detection: A survey". ACM Computing Surveys 41 (3): 1]
![Page 5: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/5.jpg)
Thing 1:Your data is NOT Gaussian
1
![Page 6: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/6.jpg)
6
Gaussian or Normal distribution
• Bell-shaped distribution
– Has a mean and a standard deviation
![Page 7: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/7.jpg)
7
This is Normally distributed data
![Page 8: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/8.jpg)
8
Quick check: Histogram
![Page 9: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/9.jpg)
9
Normal distributions are really useful
• I can make powerful predictions because of the statistical properties of the data
• I can easily compare different metrics since they have similar statistical properties
• There is a HUGE body of statistical work on parametric techniques for normally distributed data
![Page 10: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/10.jpg)
- Confidential - 10
Normal distributions
• Most naturally occurring processes
• Population height, IQ distributions (present company excepted of course)
• Widget sizes, weights in manufacturing
• …
Not
• Your metrics!
Normally distributed vs Not
![Page 11: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/11.jpg)
11
Why is that important?
• Most analytics tools are based on two assumptions:
1. Parametric techniques: Data is normally distributed with a useful and usable mean and standard deviation
2. Supervised Learning techniques: Data is probabilistically “stationary”
![Page 12: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/12.jpg)
12
Example: Three-Sigma Rule
• Three-sigma rule
– ~68% of the values lie within 1 std deviation of the mean
– ~95% of the values lie within 2 std deviations
– 99.73% of the values lie within 3 std deviations: anything else is considered an outlier
![Page 13: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/13.jpg)
13
Aaahhhh
• The mysterious red lines explained
mean
3s
3s
![Page 14: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/14.jpg)
14
Doesn’t work because THIS
![Page 15: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/15.jpg)
15
Histogram – probability distribution
![Page 16: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/16.jpg)
16
3-sigma rule alerts
![Page 17: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/17.jpg)
17
Holt-Winters predictions
![Page 18: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/18.jpg)
18
Or worse, THIS!
![Page 19: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/19.jpg)
19
Histogram – probability distribution
![Page 20: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/20.jpg)
20
3-sigma rule alerts
![Page 21: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/21.jpg)
Thing 2:Yesterday’s anomaly is today’s normal
2
![Page 22: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/22.jpg)
22
Why is that important?
• Most analytics tools are based on two assumptions:
1. Parametric techniques: Data is normally distributed with a useful and usable mean and standard deviation
2. Supervised Learning techniques: Data is probabilistically “stationary”
![Page 23: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/23.jpg)
23
Remember this data?
![Page 24: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/24.jpg)
24
No matter where you look
![Page 25: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/25.jpg)
25
Its characteristics are stationary
![Page 26: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/26.jpg)
26
Meanwhile, in our real world
• Stationarity is not a realistic assumption in the large complex systems with which we’re dealing
• “Concept Drift” is very common
– http://en.wikipedia.org/wiki/Concept_drift
“ … the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.”
![Page 27: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/27.jpg)
27
Meanwhile, in our real world
• Stationarity is not a realistic assumption in the large complex systems with which we’re dealing
• “Concept Drift” is very common
– http://en.wikipedia.org/wiki/Concept_drift
“ … the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.”
![Page 28: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/28.jpg)
28
Supervised learning
• In ML, Supervised Learning is the general set of techniques for inferring a model from a set of observations:– Observations in a Training Set are labelled with the
desired outcomes (e.g. “normal vs. anomalous”, “normal vs. fraudulent”, “red/green/yellow”, etc)
– As observations are fed into the learning system, it learns to differentiate by inferring a model based on these labels
– Once sufficiently “trained”, the system is used in production on “real” unlabelled data and can label the new data based on the inferred model
![Page 29: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/29.jpg)
29
What happens when something changes in your fundamentals?
![Page 30: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/30.jpg)
30
This is your new normal: all red all the time
![Page 31: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/31.jpg)
31
Mean Shift and Breakout Detection
• https://blog.twitter.com/2014/breakout-detection-in-the-wild
![Page 32: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/32.jpg)
Thing 3:Saying Kolmogorov-Smirnov is a great way to
impress everyone
3
![Page 33: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/33.jpg)
33
Why is that important?
• Seriously!?
• Ok, actually non-parametric techniques that make no assumptions about normality or any other probability distribution are crucial in your effort to understand what’s going on in your systems
![Page 34: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/34.jpg)
34
The Kolmogorov-Smirnov test
• Non-parametric test– Compare two probability
distributions– Makes no assumptions (e.g.
Gaussian) about the distributions of the samples
– Measures maximum distance between cumulative distributions
– Can be used to compare periodic/seasonal metric periods (e.g. day-to-day or week-to-week)
http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
![Page 35: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/35.jpg)
35
KS with windowing
![Page 36: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/36.jpg)
36
Data from similar windows
![Page 37: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/37.jpg)
37
Cumulative distribution for those windows
![Page 38: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/38.jpg)
38
Data from dissimilar windows
![Page 39: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/39.jpg)
39
Cumulative distribution for those windows
![Page 40: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/40.jpg)
40
Sliding window of KS scores
![Page 41: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/41.jpg)
41
KS anomaly results
![Page 42: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/42.jpg)
Thing 4:Take Scope and Context into account!
4
![Page 43: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/43.jpg)
43
Some data – is that normal?
![Page 44: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/44.jpg)
44
Wider scope
![Page 45: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/45.jpg)
45
Is this an anomlay?
![Page 46: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/46.jpg)
46
Even wider scope
![Page 47: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/47.jpg)
47
Is every weekend an anomaly?
![Page 48: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/48.jpg)
48
Would this be more accurate?
![Page 49: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/49.jpg)
49
Use domain knowledge!
• Domain knowledge is NOT a bad thing!– There is no algorithm that will work on everything
– Know your data and it general patterns• Periodicity/Seasonality
• Known events (maintenance, backups, etc)
– Apply the appropriate algorithms, taking into account enough scope for any inherent periodicity to appear
– Customize your alerts to take into accounts known events
![Page 50: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/50.jpg)
Thing 5:No data != No information
![Page 51: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/51.jpg)
51
Why is that important?
• Some data channels are inherently non-chatty:
– We don’t have the luxury of always generating non-zero values
– There is a lot of useful information in the fact that nothing is happening on a particular channel
• A lot of time series analytics techniques fail on time series with too few values (e.g. RF, adjusted box plot, etc)
![Page 52: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/52.jpg)
52
Communication channel
![Page 53: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/53.jpg)
53
Box plot results
![Page 54: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/54.jpg)
54
![Page 55: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/55.jpg)
55
Simple lookup table with priors
![Page 56: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/56.jpg)
56
Don’t be an analytics snob
• Sparse data is VERY hard to analyze using typical analytics techniques
• Sparse data conveys VERY important information
• Sometimes the simplest rules, thresholds, lookup tables will work
![Page 57: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/57.jpg)
57
Recap
1. Your data is NOT Gaussian
2. Yesterday’s anomaly is today’s normal
3. Kolmogorov-Smirnov is really cool
4. Scope and Context are important
5. No data != No information
![Page 58: Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez - Metafor Software - LISA 2014](https://reader030.vdocument.in/reader030/viewer/2022032616/55a3f6f51a28abf6718b4854/html5/thumbnails/58.jpg)
58
Questions?
• Shout out to the Metafor Data Science team!
– Fred Zhang
– Iman Makaremi